Deploying StoreGrid
Online Backup on Amazon EC2 / S3
A 'How To' guide that helps you run your online backup service on Amazon's Cloud
Section 1 - Introduction
Vembu StoreGrid Cloud AMI which facilitates deploying StoreGrid on Amazon's Cloud Computing Infrastructure is available to partners for production use. The StoreGrid Cloud AMI is like a virtual appliance which can be instantiated as a backup or a replication server and run on Amazon Elastic Cloud Computing (EC2) platform. The StoreGrid Cloud AMI will use Amazon Simple Storage Service (S3) to store the backup data from StoreGrid Clients.
So far, we have primarily worked with partners who are willing to host StoreGrid in their own data centers and offer online backup services to their customers. With the StoreGrid Cloud AMI Virtual Appliance, any IT solution provider can now start an online backup service using Amazon Web Services without any upfront capital investment on servers and storage infrastructure. Besides the current practice of deploying the StoreGrid backup server and replication server in their own data centers, service providers now have the added options of:
Running their StoreGrid backup server and replication servers in the Amazon Web Services cloud computing infrastructure. The (Windows or Linux) servers will run on Amazon EC2 instances with Amazon S3 as the primary data storage backend. The Amazon EBS volumes are used for the MySQL database storage and as a temporary local cache for the backup data.
Deploying a StoreGrid backup server in their own data center and having the backup server replicate the backup data to a StoreGrid replication server running on Amazon EC2.
Running StoreGrid backup servers 'on-premise' at customer locations so that there is local copy of the backup data for quick restores. The StoreGrid Cloud AMI can function as a replication server to replicate the various 'on-premise backup servers' to the Amazon Cloud - in effect, providing an additional disaster recovery option.
The StoreGrid Cloud AMI virtual appliance is currently available for Windows Server and CentOS Linux server. Other requisite modules like the MySQL 5.1 back-end database are bundled together in the StoreGrid Cloud AMI to facilitate ease of deployment for partners. However, as detailed below, the Amazon deployment will currently require some work on your part.
If you have questions, please email us at storegrid-cloud@vembu.com
Section 2 - Get your Amazon AWS account
-
Point your browser to http://aws.amazon.com and click on the button ‘Sign Up Now’
-
Enter your Amazon account details, if you have one (or select ‘I am a new user’)
-
Enter your name and email address to register for an Amazon Web Services Account and click ‘Continue’
-
Please enter your contact information and accept the AWS Customer Agreement
-
You will see this screen below if you have successfully created your AWS account.
-
Login into Amazon Web Services and select Your Account > Access Identifiers from the top menu. Note down your Amazon AWS account number from the top-right corner.
-
You will see the ‘Access Key ID’ and your ‘Secret Access Key’ listed below. Please note them down for future use.
To proceed further, you can use AWS Management Console, ElasticFox or Amazon Command Line tools. In this guide, we will proceed with the AWS Management Console to manage your Amazon Cloud instances.
Section 3 - Launching StoreGrid Cloud AMI Instance
-
Login to AWS Management Console from the URL, https://console.aws.amazon.com
-
Select the region from the combo box under 'Navigation' panel. By default, US East region will be selected.
Section 3.1 - Generating a Secure Key Pair for Authentication
In order to remotely login to your StoreGrid Cloud AMI instance running in Amazon EC2 through SSH or Remote Desktop (RDP) you need to generate a Secure Key-Pair for secure communication. Please note, launching public AMIs (such as StoreGrid Cloud AMI) without a key pair ID will leave them inaccessible.
The Secure Key-Pair is a 2048 bit RSA key pair generated with a specified name. The generated key-pair should be specified while launching an Amazon EC2 instance and will be used for authenticating TCP connections such as SSH & RDP.
In Linux, the key pair content is used as an identity [SSH Authorization Key] to authenticate the root user connecting to the EC2 Instance.
In Windows, the key pair is used to generate and encrypt the initial random password for the EC2 Instance, such that the password is not preset in a public image. To fetch and decrypt the initial random password which is set in the EC2 instance, you have to use the Key-Pair. After fetching the password, you would be able to access the EC2 instance with UserName: Administrator and Password: <as fetched>
Create a key pair by following the below steps in AWS Management Console (If you have already created the key pair, you can ignore these steps)
In AWS Management Console, click 'Key Pairs' link under 'Navigation' panel.
-
Click 'Create Key Pair' button and enter the Key Pair Name and click 'Create'.
-
After clicking 'Create' button, you will be asked to save the key pair file. Save the key-pair file in a location in your machine.
NOTE: This key-pair file will be used for fetching password in a Windows instance & authenticating root user in Linux installation. Hence, this key-pair file needs to be preserved.
Section 3.2 - Launch StoreGrid EC2 instance
The AMI ID of Vembu StoreGrid Cloud AMI are as follows:
- StoreGrid Cloud AMIs for the US East Region
- Vembu StoreGrid Cloud AMI ID for Windows [32 bit]: ami-8c5589e5
- Vembu StoreGrid Cloud AMI ID for Windows [64 bit]: ami-ba518dd3
- Vembu StoreGrid Cloud AMI ID for Cent OS [32 bit]: ami-682df101
- StoreGrid Cloud AMIs for the US West North California Region
- Vembu StoreGrid Cloud AMI ID for Windows [32 bit]: ami-ab4f17ee
- Vembu StoreGrid Cloud AMI ID for Windows [64 bit]: ami-b74f17f2
- Vembu StoreGrid Cloud AMI ID for Cent OS [32 bit]: ami-33461e76
- StoreGrid Cloud AMIs for the EU West Region
- Vembu StoreGrid Cloud AMI ID for Windows [32 bit]: ami-dd5a62a9
- Vembu StoreGrid Cloud AMI ID for Windows [64 bit]: ami-395a624d
- Vembu StoreGrid Cloud AMI ID for Cent OS [32 bit]: ami-bd5c64c9
- StoreGrid Cloud AMIs for the Asia Pacific Region
- Vembu StoreGrid Cloud AMI ID for Windows [32 bit]: ami-d81f588a
- Vembu StoreGrid Cloud AMI ID for Windows [64 bit]: ami-2a2e6978
- Vembu StoreGrid Cloud AMI ID for Cent OS [32 bit]: ami-661b5c34
- StoreGrid Cloud AMIs for the Asia Pacific(Sydney) Region
- Vembu StoreGrid Cloud AMI ID for Windows [64 bit]: ami-6040d65a
- StoreGrid Cloud AMIs for the US West Oregon Region
- Vembu StoreGrid Cloud AMI ID for Cent OS [32 bit]: ami-ce0986fe
Launch a StoreGrid EC2 instance by following the below steps :
Click on 'AMIs' link under 'Navigation' panel in AWS Management Console. The StoreGrid public AMIs will be listed under 'Amazon Machine Images'.
-
Right click on the StoreGrid public AMI listed and select 'Launch Instance'.
-
Enter the number of instance(s) to start and the instance type and click 'Continue' button.
-
Leave the 'Kernel ID' and 'AKI ID' as 'Use Default' and click 'Continue' button.
-
Select the option 'Choose from your existing Key Pairs' and select the key pair name which we have created previously. Click 'Continue' button.
-
Choose the 'Security Group' and click 'Continue' button.
-
Review your settings and click 'Launch' button.
-
You will get a window that your instance is running.
Make sure the instance has entered running mode, as the Public DNS Name will be available only after the instance has entered running mode. The Public DNS Name will enable you to remotely access the EC2 instance.
Section 3.3 - Firewall Configuration
To allow the remote access to your EC2 instance, you need to grant permission to the 'default' security group which determines the ports opened up for communication in your EC2 instance.
Enable the firewall configurations by following the below steps:
- Click on 'Security Groups' link under 'Navigation' panel.
- Click on default security group.
- Add the necessary ports under 'Allowed Connections' in default group.
The TCP port for which the permission should be granted is,
- Port 22 for opening up SSH access in Linux
-
Port 3389 for opening up Remote Desktop (RDP) access in Windows
Section 3.4 - Logging to the instance
-
For Linux instance, login to the Public DNS of the instance through SSH by executing the below command :
ssh -i <key-pair file> root@<Public DNS Name of the instance>
where <key-pair file> is the key-pair file to authenticate the SSH Connection with the EC2 Instance.
-
For Windows instance, to get the instance password, please follow the below steps:
Click on 'Instances' link under 'Navigation' panel.
-
Right click on the particular instance and select 'Get Windows Password'.
Copy and paste your key pair file content in 'Private Key' field and click 'Decrypt Password'. You will get the decrypted RDP Password.
Now, you can connect to the Windows instance using RDP (Remote Desktop or the Microsoft Terminal Services Client) with the UserName: Administrator and Password: <as fetched by the above steps>
It is strongly recommended to change the password in the running instance after you first login to the instance.
NOTE: Once the password is changed, you will not be able to login with the password fetched using 'Get Default Administrator Password'.
Section 3.5 - Mount EBS Volumes
To store the backup data of your clients in Amazon S3 and the metadata information of the StoreGrid Backup/Replication Server, you need to create EBS Volumes with a storage size as required. The EBS volume will be used by StoreGrid as a temporary cache location before uploading the clients' data to Amazon S3. After creating the EBS volumes, you can mount them to the EC2 instance that you have started. The attached EBS Volume also stores the MySQL Database data.
To do this, please follow the below steps:
Go to 'Volumes' link under 'Navigation' panel.
-
To create a new volume click 'Create Volume' button. Enter the size of the Volume and select the availability zone. Please make sure that the availability zone you select here is same as the availability zone of the instance. (Ignore this step if you have already created the EBS volume)
-
Attach the EBS volume to the instance by right clicking the volume and by selecting the instance ID and the device name.
-
Once the EBS volume is attached,
In Linux, login to instance via SSH and format (only for new EBS volume) & mount the volume by execute the following commands,
yes | mkfs -t xfs /dev/sdh
mount /dev/sdh /storegridwhere /dev/sdh is the device name and /storegrid is the mount point.
Now, execute command 'df -lh' and you will find the EBS mounted in /storegrid location
In Windows,
- On the taskbar, click Start, and then click Run.
- Type diskmgmt.msc and click OK. The Disk Management utility opens.
- Right-click the Amazon EBS volume, select New Volume, and follow the on-screen prompts
Convert the EBS volume attached to the EC2 instance into a new partition using the New Partition Wizard on Windows
Once you complete the wizard, you will find the EBS volume listed in the EC2 instance like the figure below.
NOTE: If the EBS Volume mounted is an existing volume which already contains data(StoreGrid Cloud AMI configuration xml files), the EBS Volume need not be formatted and file system created, instead EBS Volume can be just attached to the instance.
In Linux, once attached, the EBS Volume should be mounted to the specified mount location.
mount /dev/sdh /storegrid
NOTE: In Linux, the Mounted Location should be '/storegrid' such that the StoreGrid automated script, recognizes the StoreGrid configurations.
Section 3.6 - Firewall Configurations for StoreGrid
To allow StoreGrid clients to connect to the StoreGrid Backup Server, you need to grant permission to the 'default' security group which determines the ports opened up for communication in your EC2 instance.
The TCP ports for which the permission should be granted are 32004, 32007. For accessing StoreGrid WebConsole remotely, you can open up 6060 and 6061 TCP Ports
Enable the firewall configurations by following the below steps:
- Click on 'Security Groups' link under 'Navigation' panel.
- Next, click on default security group.
- And, add the necessary ports under 'Allowed Connections' in default group.
The TCP port for which the permission should be granted is,
| Port Number | Description |
| 32004 | For default Backup Port |
| 32007 | For default SSL Backup Port |
| 6060 | For default HTTP Port of StoreGrid WebConsole |
| 6061 | For default HTTPS Port of StoreGrid WebConsole |
Section 4 - StoreGrid Cloud AMI Configuration
To have a persistent configuration of the StoreGrid Server, the StoreGrid configuration values and Amazon access credentials can be stored in the EBS Volume mounted for the local backup cache. This will enable the AMI when restarted as a fresh instance to load the configuration values from the EBS Volume and start the StoreGrid with minimal manual intervention. Please follow the below steps to create the configuration as needed by StoreGrid to work with Amazon S3 as the backend storage.
Create a folder 'SGCloudConf' under the attached volume in your instance.
Place the your key-pair (that you got from section 3.1 above), EC2 certificate and EC2 private key (that you got from step 7 of section 2 above) files under '<Mounted_Location>/SGCloudConf' in the name of ec2-key-pair, ec2-cert.pem and ec2-priv-key.pem respectively [You can skip this step if 'EBSSnapShots' is disabled in SGConf.xml file].
-
Create a file 'SGConf.xml' under '<Mounted_Location>/SGCloudConf'. This xml file stores the values of StoreGrid Configuration and EBS Volume snapshot schedule.
For example, if you are going to run StoreGrid as a "Backup Server" in your EC2 instance and the unique StoreGrid ID of your EC2 instance is "backupserver@datamaniacs.com" and you would like to configure "weekly" snapshot for the volume "vol-123a4bc0" on every "Sunday at 2 AM", then your SGConf.xml should be as below:
<StoreGrid>
<Configuration ID="backupserver@datamaniacs.com" StartModule="601" />
<EBSSnapShots Enabled="1">
<Schedule Type="Weekly" Day="0" Hour="2" Mins="0" />
<Volume ID="vol-123a4bc0" />
</EBSSnapShots>
</StoreGrid>
Attribute Description ID The ID attribute under Configuration tag represents the unique StoreGrid ID of the StoreGrid installation. StartModule The StartModule attribute specifies that the StoreGrid instance to be started as 601 - Backup Server or 609 - Replication Server Enabled The Enabled attribute under EBSSnapShots tag represents the enable - 1 and disable - 0 of scheduled snapshots of the attached EBS Volume. It is recommended to have this value enabled, as this would make sure that there is a backup of the attached EBS Volume. Schedule The Schedule tag fills in the details for when to schedule the EBS Volume snapshot. For example, if you wish to do snapshot of the EBS volume daily on 2:10 AM, then you can configure as follows : <Schedule Type="Daily" Day="0" Hour="2" Mins="10" />Volume ID The Volume tag's ID attribute provides the EBS Volume's id value for taking the snapshot. -
Create a file 'S3Conf.xml' under '<Mounted_Location>/SGCloudConf' as follows - For example, if your access key is 'SDFD4SFDSG43', secret key is '4GSDGTSYRSY32' and the bucket name is 'datamaniacsbucket' then your S3Conf.xml file should be as follows:
<StoreGrid>
<S3Configuration AccessKey="SDFD4SFDSG43" SecretKey="4GSDGTSYRSY32" BucketName="datamaniacsbucket" />
</StoreGrid>
Attribute Description AccessKey AccessKey attribute stores the AWS access key. SecretKey SecretKey attribute stores the AWS secret key. BucketName Enter the name of the Amazon S3 bucket to store the backup data. 
You need to create the specified bucket (should be in lower case) in S3 for your S3 storage before starting StoreGrid. You can create the bucket by using the S3Fox (Firefox addon) or Bucket Explorer.
The Amazon access credentials are encoded to a value understood by StoreGrid once after reading the values. The encoded values will be available in 'EncryptedS3Conf.xml' file.
-
A log file can be located at the following location to debug the above steps,
In Windows,
"C:/Program Files/Vembu/StoreGrid/SGCloud/SGDaemon.log"In Linux,
"/home/storegrid/Vembu/StoreGrid/SGCloud/SGDaemon.log"This will help identify the problem and resolve it.
-
The logs of the Scheduled snapshots are located at,
In Windows,
"C:/Program Files/Vembu/StoreGrid/SGCloud/SnapShot.log"In Linux,
"/home/storegrid/Vembu/StoreGrid/SGCloud/SnapShot.log"After StoreGrid is started, login to StoreGrid webconsole by entering http://<Public_DNS_name_of_the_instance>:6060 with the username of "admin" and password "storevembu".
-
Login to your StoreGrid v4.0/4.1 Backup/Replication Server running in Amazon EC2 through RDP (for Windows instance) or SSH (for Linux instance).
Stop StoreGrid process that is running. In Windows you can right click on StoreGrid's System Tray icon and select Stop StoreGrid option in Windows instance or by running /etc/init.d/storegrid stop in Linux.
Stop MySQL Server by executing the command net stop MySQL51 in a command prompt in Windows instance or by running sh mysqld.sh stop from /home/storegrid/Vembu/StoreGrid folder in Linux.
For Linux instance, unmount the EBS volume attached to the EC2 instance by executing the command umount /storegrid (Ignore this step for Windows instance)
Login into AWS Management Console from the URL, http://aws.amazon.com/console/
Select the Region on the top left corner.
-
Detach your EBS Volume which is attached to your StoreGrid v4.0/4.1 Backup/Replication Server instance. Click on Volume link under Elastic Block Store under the Navigation panel. Right click on Volume ID that you have attached to your instance and select Detach Volume.
Take a snapshot of the attached volume by right clicking on the Volume ID and select Create Snapshot from Volume and wait until the snapshot process completes
-
Launch a new instance of StoreGrid v4.2 Cloud AMI.
StoreGrid v4.2 Cloud AMIs are as follows :
US EAST
- ami-8c5589e5 (Windows 32 bit)
- ami-ba518dd3 (Windows 64 bit)
- ami-682df101 (CentOS)
US WEST NORTH CALIFORNIA
- ami-ab4f17ee (Windows 32 bit)
- ami-b74f17f2 (Windows 64 bit)
- ami-33461e76 (CentOS)
EU WEST
- ami-dd5a62a9 (Windows 32 bit)
- ami-395a624d (Windows 64 bit)
- ami-bd5c64c9 (CentOS)
ASIA PACIFIC
- ami-d81f588a (Windows 32 bit)
- ami-2a2e6978 (Windows 64 bit)
- ami-661b5c34 (CentOS)
ASIA PACIFIC(SYDNEY)
- ami-6040d65a (Windows 64 bit)
US WEST OREGON
- ami-ce0986fe (CentOS)
Click AMIs link under Navigation panel. Right click on the StoreGrid AMI ID and select Launch Instance and proceed the Instance Request Wizard. Make sure the availability zone that you select is same as the availability zone of the EBS volume that you have detached from your StoreGrid v4.0 instance.
Check the status of the instance by right clicking on the Instance ID and by selecting System Log.
-
Once the instance is ready, attach your EBS volume to the newly launched v4.2 instance. Click on Volume link under Elastic Block Store under the Navigation panel. Right click on Volume ID that you have detached from your old v4.0/4.1 instance and select Attach Volume. Select the Instance ID of StoreGrid v4.2 instance, choose the device name and click Attach.
Once the status of the volume change to in-use, login to the EC2 instance through RDP (for Windows) or SSH (for Linux instance)
-
For linux instance, mount the volume by executing the following command (Ignore this step for Windows):
mount <device name> /storegrid
where <device name> is the device name.
Ex : If the <device name> is /dev/sdh, then execute the following command :
mount /dev/sdh /storegrid
-
StoreGrid and MySQL will be started automatically in your new v4.2 instance. Your Backup/Replication Server will be migrated to v4.2.
Section 5 - Download and Install StoreGrid Client
Download the StoreGrid v4.2 Client builds for installation in the machines you want to backup.
v4.2 upgrade is now available which supports VHD Based Incremental Image Backup.
| Supported OS Versions | Size | Download Client Only Build |
 Windows® 8 / 7 / Vista / XP |
34.5 MB | |
| 33.4 MB | ||
|
RedHat Linux 8.0 and later, RedHat Enterprise Linux 4 and later, Fedora core 3 and later, CentOS 4.2 and later, SuSE Linux 9.x and Later [Build No. : 4212012092118] (Unzip the zip file and then execute the bin file) |
44.9 MB | |
|
Mandrake Linux 10.0 and Later [Build No. : 4212012092118] (Unzip the zip file and then execute the bin file) |
44.6 MB | |
|
Debian Linux 4.0 and later, Gentoo and Ubuntu 5.10 and later [Build No. : 4212012092118] (Unzip the zip file and then execute the bin file) |
45.2 MB | |
|
FREEBSD 6 [Build No. : 4212012092118] (Unzip the zip file and then execute the bin file) |
50.5 MB | |
|
FREEBSD 7 [Build No. : 4212012092118] (Unzip the zip file and then execute the bin file) |
50.4 MB | |
|
Mac OS X 10.3.x (Panther) and 10.4.x (Tiger) for PowerPcs Developer Tools not needed! [Build No. : 4212012092118] |
53.5 MB | |
|
Mac OS X 10.5.x(Leopard), 10.6.x (Snow Leopard) and 10.7 (Lion) for Intel PCs Developer Tools not needed! [Build No. : 4212012092118] |
48.6 MB | |
|
Solaris (For Solaris 10 - Intel) [Build No. : 4212012092118] (Unzip the zip file and then execute the bin file) |
44.1 MB |
Installing StoreGrid Client in Amazon EC2 for Backups for mounted EBS volumes:
If you are looking at installing the StoreGrid Client build in your existing Amazon EC2 instances which you want to backup, then please do look at the instructions for installing StoreGrid Clients in an Amazon EC2 instance and configuring backups for the mounted EBS volumes.
If you have questions or run into any issues, please email us at storegrid-cloud@vembu.com
Section 6 - Seeding Backup Data:
When backup data is huge and the client cannot directly backup the data to the Backup Server running on Amazon EC2 due to limited network bandwidth in the client network, the backup data (client does a 'same machine' backup to an local external drive) can be taken to a datacenter/network which has higher upload bandwidth. The data can then be transferred (like for example through FTP) into the Backup Server machine running on EC2. The backups can then be attached to the backup server and the data migrated to the S3 through the Server Side Local to Remote Migration feature. For information on this, please refer: Server Side Local to Remote Migration
Section 7 - Migrating from older version to StoreGrid v4.2:
Partners who are currently running StoreGrid v4.0/4.1 AMI can upgrade to the latest v4.2 by following the steps mentioned below. Partners who are running StoreGrid v3.5 AMI need to first upgrade to v4.0 AMI before upgrading it to v4.2. Partners who are running StoreGrid v2.5.5 AMI, please contact our support at storegrid-support AT Vembu DOT com; they will guide you to migrate to v4.0.
Upgrading StoreGrid Backup/Replication Server running on Amazon EC2 from v4.0/4.1 to v4.2
Please follow the steps given below to migrate your StoreGrid Amazon AMI running StoreGrid Backup or Replication Server running in v4.0/4.1 to v4.2 :
Changes to be done in the StoreGrid clients
Please note that if you are not using an Elastic IP IP for your backup server, then after migrating the StoreGrid Backup Server to v4.2, a new public DNS name will be assigned to your backup server running in v4.2. You need to point StoreGrid clients to the new public DNS name of your v4.2 backup server by doing client side Local To Remote Server Migration.
Please follow the step 11 mentioned in the Seed Backup online help document available at http://storegrid.vembu.com/online-backup/sp-help/PageHelp/server-side-seed-backup-migration.html
If you are running backup server in your datacenter and the replication server in the EC2 Cloud, then after migrating the backup server and replication server to v4.2, you need to change the replication server name in the 'Server Management -> Replication Management -> Configure Replication (tab)' in the backup server to the new public DNS name of the new replication server running v4.2.
It is recommended to have an Elastic IP address for your backup/replication server running in Amazon EC2 so that DNS name of your backup server doesn't change during a version migration (when you terminate the old EC2 instance and launch a new instance for the new version).
Please note that StoreGrid Backup Server v4.2 is tested with v4.0 and v4.1 StoreGrid clients. Therefore, your v4.0 and v4.1 clients can continue backing up to v4.2 backup server and you can upgrade your StoreGrid clients from older version to v4.2 whenever feasible. To upgrade your clients to v4.2, you can install StoreGrid v4.2 client build on top of the existing v4.0/4.1 client installation after upgrading your backup server. You can also push the v4.2 into the client machines from the StoreGrid backup server through StoreGrid's Automatic Software Update feature.
If you have any further clarifications, please email us at
© 2004 - 2013 Vembu Technologies Pvt. Ltd. All Rights Reserved.
