ABOUT
CASE STUDIES
TECHNOLOGY
PRODUCTS & MARKETS
shadow
CLOUDMOVER DEPLOYMENT GUIDE (Version 3.6.2)
Introduction
Product Use Cases
Pixspan's CloudMover is the fastest and most user-friendly copy/move engine available in the AWS Cloud for S3 Storage. CloudMover improves virtually any workflow that relies on data movement within S3, and it operates efficiently in all 18 worldwide regions where S3 is supported. The user is given control over the speed and related costs of copy/move operations, from fast to ultra fast. Use cases include:
  • Geo-Redundant Disaster Recovery
  • Follow the Sun Content/Creative Workflows
  • Data Migration
  • Hub and Spoke Content Distribution or Collection
  • Backup/Nearline
CloudMover has an additional capability for content that includes full resolution images. This capability significantly accelerates movement of content within the Cloud, while also providing substantial savings on S3 storage and inter-region transfer costs. This focuses the above use cases on the following industry verticals:
  • Media and Entertainment
  • Medical Imaging
  • Satellite (including multi-spectral imaging)
  • Surveillance
  • Oil and Gas
  • Astronomy
Deployment Overview
CloudMover is architected with a Server Node that performs the UI and Job Control functions, and Worker Nodes that perform the actual copy/move operations. In the typical deployment, the user will launch the CloudMover Server Node Instance from the AWS Marketplace into the geographically closest AWS EC2 Region. The purpose of choosing a geographically close region is so that the latency for UI functions is minimized. If that isn't a primary consideration, the Server can be launched in any region where EC2 is supported. As an example, a user in Los Angeles will typically launch the Server Node Instance in Northern California into the us-west-1 region. When the user wants to perform a copy from a S3 bucket in Northern California to a S3 bucket in London or Mumbai, he or she will create that copy job using the Server Node GUI. The Server Node will then stage the job and create Worker Nodes in the destination region to perform the copy operations. These example operations are shown in the following diagram:
Resource Requirements
Access to specific AWS resources are required in order to launch CloudMover and then to operate in the required S3 destination regions. These resources include:
System Diagrams
Server Node Launch
CloudMover is architected with a single instance Server Node that is responsible for all the configuration, user management, and job management aspects of the product. This component of CloudMover is the only AMI launched from the AWS Marketplace by the CloudFormation template, and it is launched into the region selected by the user.
During the Instance Launch sequence, the user is prompted to choose the VPC where the Server Node instance will reside in the given Region.
Intra-region Operation
CloudMover can be used for moving data between Buckets in the same region. The following diagram illustrates this use case where the Server and the Workers run within the same region, and the S3 buckets involved in the transfer are also in that same region. Note that the interactions between system components don't go outside the region.
The following diagram shows CloudMover performing a move/copy operation between buckets in the same region, but the Server is not in that same region.
Inter-region Operation
CloudMover can also be used for moving data between Buckets in different regions. The following diagram illustrates this use case where the Server and the Workers run in different S3 Regions. Note that the interactions between system components goes between Regions, and therefore goes across the AWS Backbone Network. This communication between system components utilizes TLS, and is therefore secure.
Prerequisites and Requirements
Expertise Required
There is no special expertise required to install or operate CloudMover when using the product for basic functionality. Users need to be able to execute the short questionaire-style launch procedure that is managed by CloudFormation, which is typical for launching any AWS Instance. The user must also have a general understanding of AWS capabilities and functionality, be able to operate a browser-based UI, and have an understanding of storage configurations. The person installing CloudMover should understand the basics of their company's security policies and how they map to AWS Security Groups (e.g. firewall requirements implemented with security groups). Advanced users will further have the ability to improve the security of the content operated on by CloudMover by incorporating more advanced AWS features, such as Virtual Private Clouds (VPC) and VPC Peering.
AWS User Management Best Practices
Users need an to have an AWS account in order to install and administer CloudMover. If you don't have an account, one can be created at the AWS Portal. It is important to understand how AWS manages users, and this can be studied at the IAM main page
All AWS accounts have a root user with full administrative capabilities, though best practices dictate that this user should NOT be utilized to administer the CloudMover product. Instead, a new user account should be created using the Identity and Access Management (IAM) Service. Please refer to Amazon's IAM Best Practice Guide, for detailed instructions on how to create the Role, Group, and User for CloudMover.
The first step in this process is to create an access policy for the new user through the IAM Service Dashboard (choose the Policies navigation link at the left). Choose a policy name that corresponds to the product, such as 'CloudMoverAdministrativePolicy'. The policy should be built with the following rules:
The next step is to create a Group, and attach the CloudMover Administrator Policy to that Group. IAM Groups are configured by selecting the Groups navigation link at the left and then clicking on the 'Create New Group' button at the top of the screen. Enter a Group name that identifies the group as being associated with CloudMover, such as CloudMoverAdministrativeGroup. Click on the 'Next Step' button at the bottom right of the screen, and then attach the new CloudMover Policy that was created in the previous step. Select the checkbox next to the name, then click on the 'Next Step' button on the bottom right of the screen. Review the Group and then click the 'Create Group' button in the bottom right.
The final step is to add a new user. This is initiated by selecting the Users navigation link at the left side of the page. Click on the 'Add user' button at the top of the screeen. Enter a User name in the next screen and select the checkbox next to the 'AWS Managemenet Console access' selection, and enter a password. Then click the 'Next: Permissions' button in the bottom right. Add this user to the Group that was configured in the previous step, and then click the 'Next: Tags' button at the bottom right. Click through the Add Tags selection, review the information, and then click the 'Create user' link in the bottom right.
Pre-Installation Setup Requirements
The AWS Administrator for the company needs to ensure that the AWS Account Limits for the EC2 Instance Types are consistent with the requirments of CloudMover. CloudMover requires one m5.xlarge instance for the Server Node, and it requires multiple Instances running for the Worker Nodes. The number of Worker Instances that are required is based on the CloudMover Job Delivery Option chosen. CloudMover supports three levels of delivery speeds, named Express, Plus, and Ultra, and these selections differ in the required number of EC2 Instances as follows:
  • Express Copy/Move - 4 m5.xlarge instances (transfers that contain files larger than 250MBytes may require CloudMover to request m5.2xlarge or m5.4xlarge)
  • Plus Copy/Move - 12 m5.xlarge instances (transfers that contain files larger than 250MBytes may require CloudMover to request m5.2xlarge or m5.4xlarge)
  • Ultra Copy/Move - 40 m5.xlarge instances (transfers that contain files larger than 250MBytes may require CloudMover to request m5.2xlarge or m5.4xlarge)
  • Express Image Encode/Decode - 5 c5.9xlarge instances (in rare occasions, a c5.18xlarge instance will be selected when ultra-large files are transferred)
  • Plus Image Encode/Decode - 15 c5.9xlarge instances (in rare occasions, a c5.18xlarge instance will be selected when ultra-large files are transferred)
  • Ultra Image Encode/Decode - 45 c5.9xlarge instances (in rare occasions, a c5.18xlarge instance will be selected when ultra-large files are transferred)
The AWS Administrator needs to ensure that the limits are increased to cover all of the modes that will be used. The limits on the number of EC2 instances can be found in the AWS Management Console on the Limits tab of the EC2 Dashboard, as shown in the following diagram.
To allow for transfers up to and including the Ultra (fastest) speed for content that has files of any size, the limits for the following instance types should be increased by the specified number in all regions that will be a destination for transfers:
  • m5.xlarge - 50
  • m5.2xlarge - 50
  • m5.4xlarge - 50
If the transfer includes the requirement for CloudMover to perform Pixspan-compression or Pixspan-decompression on full resolution images, or if you plan to be transferring extremely large objects (>2TB), then the following limits should be increased as well:
  • c5.9xlarge - 50
  • c5.18xlarge - 50
Pricing Information
Please see the Pixspan CloudMover Product Overview and Pricing Guide to understand the software costs associated with operating CloudMover.
Installation
Overview
The CloudMover setup procedure is very similar to the procedure to launch any commercial AMI that uses the CloudFormation procedure. AWS CloudFormation walks you through all the installation steps, as well as create the necessary collateral to launch the Instance. Note that the only component of CloudMover that is installed by the user is the CloudMover Server. Specifically, the CloudMover Worker nodes are not installed by the user. Those ephemeral resources are completely controlled by the CloudMover Server. Installation of CloudMover should take no more than 30 minutes, assuming that the installer has good knowledge of AWS facilities like VPCs, Subnets, Regions, and their S3 configuration. The installation process introduces the user configurable items in the following sections.
CloudMover Marketplace Listing
The CloudMover Marketplace Listing can be found at the following link:
The instructions that follow take you through the CloudMover installation and configuration process. The first step in this process starts at the Listing page shown directly below. Please read through the CloudMover Overview, Pricing, Usage, and Support information to determine the applicability of this product to your data movement needs in the Cloud. Then, click the orange 'Continue to Subscribe' button in the top right corner of the page.
Subscribing to the CloudMover Marketplace Listing
The purpose of the 'Subscribe' page is for you to understand the Terms and Conditions for using the CloudMover product and to verify pricing information. The tabs at the left of the following image allow you to view the per-instance Software charges associated with operating CloudMover (AWS EC2 charges are separate). The product operates with a single Head Node, and there is no software charge for running that instance. The Head Node instance will run continuously, unless it is manually stopped using the Management Console EC2 dashboard.
The Work Nodes are ephemeral, and multiple Worker instances are started and subsequently terminated by the Head Node based on the options chosen for the data movement job and the duration of the transfer. Please visit the CloudMover Pricing page at the following URL for further information about the costs that you can expect.
Please click on the End User License Agreement link (indicated in the following screen shot) to understand the terms and conditions associated with installing and operating the software. Clicking on the indicated link will launch a new tab with a PDF document with the agreement. Please read and understand this document before proceeding to the next steps.
Once you are comfortable with the terms and conditions in the EULA, please click the orange 'Accept Terms' button as shown in the following screen shot. This will activate the Continue to Configuration button in the upper right corner so that you can continue to further steps in the process.
Note that after you click the 'Accept Terms' button, there may be a delay as the Marketplace system records you EULA acceptance date. It's normal for the display to look like the following screenshot for approximately one minute (spinning indicators in the Effective Date and Expiration Date columns).
At this point, you are subscribed to CloudMover, and should receive a notification in email that states the following:
You have subscribed to the following product in AWS Marketplace:
* CloudMover Head Node for AWS S3 sold by Pixspan, Inc
Configuring CloudMover
There are two configuration items that are important for CloudMover. On the Configuration page, the 'Fulfillment Option' will be preselected for CloudMover, so this shouldn't be changed. Next, the latest software version should be chosen, and this is indicated by the highest numeric version number, along with the latest date. Finally, you should choose an AWS Region that is the phyically closest to the operations personnel who will be operating the product. The user interface for CloudMover is browser-based, and therefore is capable of being operated from any distance, though a close proximity will limit the latency and lead to a better user experience. These configuration choices are illustrated in the following screen shot.
Once you have chosen the configuration items, you can click on the orange 'Continue to Launch' button in the top right corner of the screen, as shown in the following screen shot.
Verify Before Launching
It is important to verify the options you chose in the previous steps. The fulfillment option must be set to CloudMover. Next, verify that the Software Version and the Region that you chose is indicated. CloudMover is launched through AWS CloudFormation, so the Action should indicate this. These items are shown in the following screen shot.
Before launching, it is also important to view the Usage Instructions by clicking on the 'Usage Instructions' box at the bottom of the Configuration Details section. Clicking will bring up an information box that looks like the following (note that the information in the box may differ between software versions - most notably, the login procedure changed after the version shown).
You are now ready to launch CloudMover. When ready, click the orange 'Launch' button in the lower right corner, which will start the launch process through AWS CloudFormation.
Creating a Stack, Choosing the CloudFormation Template
The first stage of the AWS CloudFormation process is to indicate the template that is used to launch the product. The Marketplace launch process for CloudMover prepopulates the information required, and the 'Specify' radio button is also preselected. This is shown in the following screen shot. The only action necessary is to click the 'Next' button in the bottom right corner of the screen.
The first item to be specified is the Stack Name associated with this AWS CloudFormation Template. By entering a name in this field, that allows you to find that Stack within the Dashboard of the CloudFormation Service. That will enable you to launch new CloudMover instances without having to go back to the Marketplace. Configuring this Stack Name is shown in the following screen shot.
Selecting the Platform
The Platform choice specifies the CloudMover software version, and the user is encouraged to choose the latest available version, that is, the version with the highest number and most recent date.
Naming the Instance
The CloudMover instance can be identified by whatever name the user requires. This allows the user to uniquely identify multiple instances of CloudMover that might be launched. The default name is set to 'CloudMover'. Once the instance of CloudMover is launched, this name can be seen on the Management Console by selecting the installed region, the EC2 Service, the 'Instances' menu item, and then looking at the 'Name' column.
Selecting the Virtual Private Cloud (VPC) and Subnet Location
The installation procedure for CloudMover allows the user to put the Server into any existing VPC within a given region. If the user requries that CloudMover be installed into its own VPC, the user must have configured this VPC in the AWS VPC Service prior to installing CloudMover.
Once a VPC is selected, a subnet that is within that VPC must be chosen. In choosing the subnet, note the IP CIDR block that is displayed with the VPC (within the parentheses), and make sure that the chosen subnet is within the VPC's CIDR block. Also, make sure that the Subnet is configured to Auto-assign public IPv4 addresses, otherwise, the newly launched Instance will not be reachable through the Browser interface.
Restricting Access to the CloudMover Server
Access to the CloudMover Server is controlled by the Security Group assigned with the Instance, which provides a firewall based on incoming and outgoing ports associated with IP CIDR Blocks. The CloudMover configuration process allows one IP CIDR block to be whitelisted for the ports that are important to the CloudMover image (shown below), covering port 22 for IT access and port 9090 for access to the UI. Additional IP Addresses can be whitelisted in the EC2 Dashboard by navigating to the Network & Security section on the left and clicking on Security Groups. Alternatively, the Security Group configuration can be accessed on the Instance display by clicking the hotlink in the Security groups cell of the table.
IT Access to the CloudMover Server
Users of CloudMover should not have the need to access the CloudMover Server operating system command line, but this option is available to set up at installation time, and is accomplished by selecting a SSH public/private Key Pair and by protecting access to the instance.
The SSH access to the instance is controlled by the user entering an IP Address Range in the SSH WhiteList edit box. Note that the IP Address range must be routable, i.e. not one of the private address ranges (e.g. 10.0.0.0 - 10.255.255.255, 172.16.0.0 - 172.31.255.255, and 192.168.0.0 - 192.168.255.255). A simple way to identify the users public IP address is to look this up on the website What Is My IP Address. For security reasons, it is highly recommended that SSH access not be provided in an unrestricted manner, i.e. the SSH WhiteList should not be set to '0.0.0.0/0', which allows anyone on the Internet to have access to the SSH Port (port 22), leaving the Private SSH Key as the only security measure. Instead, the IP Address range should be as restrictive as possible, for example '54.85.132.0/8', which gives access to just a small subnet.
In order for SSH Access to be possible, the CloudMover Server needs to be assigned a SSH Key Pair in the Region where it will be launched. If no usable key pairs exist in that region, a new one must be generated. This is accomplished on the EC2 Dashboard by selecting the Network & Security Key Pairs navigation on the left pane of the window, and then clicking on the 'Create Key Pair' button, as shown in the following screen shot.
If the Installation process was started before the desired key pair was created, then the Installation process should be restarted from the beginning.
On completion of the configuration items on this 'Specify Details' page, you can scroll to the bottom of the page and click on the 'Next' button.
CloudFormation Create Stack Options
CloudFormation allows the configuration of various options that are not important for the installation and operations of CloudMover. If there are options within this page that are integral to the IT functions of your organization, please consult with the appropriate experts in this area to select the correct values for the options. Other users should scroll down to the page and click on the 'Next' button.
CloudFormation Create Stack Review
The final step in the launch procedure is to review the selections that were made, to acknowledge the capabilities that CloudMover requires to operate, and then to launch the creation of the new Server. The Create Stack Review screen is shown in the screen shot below. This information should be verified with the selections that were made in previous screens.
If this information is valid, scroll down to the bottom of the page, where you will see that you need to select the checkbox to acknowledge that the CloudFormation Template needs to create AWS IAM (Identity and Access Management) Resources. CloudMover requires this in order to dynamically launch its Worker resources for performing the file transfers. Once you check the box, you can click on the 'Create' button in the bottom right corner of the page. This is illustrated in the following screen shot.
After clicking the 'Create' button, the CloudMover Instance will be launching, and it will take several minutes before the new EC2 Instance is available. Upon completion, the Outputs Section of the CloudFormation Template will contain the initial password for the Server as well as the URL to attach to the Server.
Navigate to the AWS Management Console EC2 Dashboard to verify the successful launch of the Instance (it will take some time before the instance is visible in the interface). Make sure that the instance passes all of the Status Checks. Note that the Instance Id is also shown on this page.
If there was a problem in the launch, and the Status Checks show errors, then terminate the new instance
(Actions->Instance State->Terminate), and run through the CloudFormation process again. If you are still having problems, contact Pixspan Support at support@pixspan.com.
It is a good practice to create a snapshot of this new image, because this saves you from having to go through the CloudFormation procedure again in the case where you need to create another CloudMover Server EC2 instance. After the instance has launched, and is in the 'running' state, select the 'Actions' pulldown, select 'Image', and then select 'Create Image', as shown in the following screen shot.
The following dialog box will be displayed. Enter an appropriate image name and description, for example, what is shown in the screen shot, and then click on the 'Create Image' button.
This newly created Snapshot can be used at a later time to launch another CloudMover Server, and this is accomplished in the EC2 Dashboard in the AMI selection in the left navigation pane. Choose the image that was created above, and then choose the 'Action' pulldown selection for 'Launch'.
Configuration
Once installation of the CloudMover Server is complete, the user must go through a short configuration procedure in order to secure the product and provide access to the user's S3 resources. The user interface for CloudMover is implemented as a browser-based GUI that is secured by TLS using HTTPS over port 9090. The AWS-assigned host name is found in the AWS EC2 Dashboard as shown in the following screen shot:
Alternatively, you may desire to have an alternative DNS address for the CloudMover Server, and this would be set up by your IT department. Once you have determined the host name, you can reach the Server using the following web address in your browser:
https://<host name>:9090
If you are having trouble reaching the CloudMover Server UI, verify that you have the host name correct. Also verify that the Subnet has routing to the Internet. You can follow the procedure in the AWS Internet Gateway Guide to make sure that the Subnet can reach the Internet. You should also verify that the Instance Security Group is configured properly such that your browser is not restricted by the Firewall. View the inbound rules on the 'Instances' navigation tab in the EC2 Dashboard, select the checkbox associated with the instance, and view the rules by clicking on the indicated link. Make sure that the Source IP range matches the IP address of the browser. Also, verify that port 9090 is open to the browser's IP address. The following screenshot illustrates these settings (though, your IP addresses will be different). Note that the inbound rules can be modified (choose to Edit) on the security group page by clicking the link directly to the left of the 'view inbound rules' link.
Once logged into CloudMover, the full user manuals are available by clicking the information icon (i) in the upper right corner of every screen. The following instructions highlight the important basic configuration settings to get CloudMover operational and performing copy/move jobs.
Initial Administor Password
For security purposes, it's not possible for a default user/administrator password to be left over in the configuration process. CloudMover has a built-in default password, which is the Instance ID for the CloudMover Server. The Instance ID is found by looking in the AWS Management Console EC2 Dashboard by selecting the Instances list, and locating the newly launched CloudMover Instance. The instance ID will have the form: i-01ae8c7822ab996c3. This password must be changed during the first login sequence, or the user will not be allowed access to the CloudMover UI. The user is informed of the strenth of the password, and is encouraged to enter a password with strong security. This configuration step is shown in the following screen shot:
Once the 'New' and 'Verify' version of the password match, the user will be able to proceed by clicking the 'Update' button.
User Role Information
CloudMover supports three user roles:
  • Admin
  • Wrangler
  • View Only
The Admin user has the ability to use all of the functionality of the UI, which includes adding new users, configuring S3 source and destination buckets, controlling the access of other users to S3 resources, viewing Job Logs, monitoring the status of Worker Nodes, and setting up Email notification facilities. Pixspan recommends that this role should be restricted to users actively performing administrative functions, so that errant administrative changes aren't made during regular operation. Wrangler users have the ability to run and monitor jobs, which is the role that the majority of users should be using. Administrative users who also have operational duties should assign themselves a separate Wrangler user/login. View-Only users just have the ability to monitor the status of jobs. This role is beneficial to internal customers who are dependent on the results of a transfer/copy job and need to be updated, but that don't directly control file movement. Please see the CloudMover User Guide for specific information about these three user roles.
User Configuration
Administrators add new users through the User Admin tab, as shown in the following screen shot. Note that clicking on the the User Admin tab will show the default action for Listing/Modifying Users. This selection allows for the modification of the role and email address. Note that there must always be at least one Administrator user configured in the system.
Users are configured with the Add New User Action selection, as shown in the following screen shot. This screen allows multiple users to be entered at the same time. This is accomplished by clicking on the plus sign for each greyed out entry that is after the live entry area(s). The administrator is required to enter a unique username, an initial password, and a role. Delivery options for that new user are selected by the assigned role, but can subsequently be changed. Adding an email address for the user is optional (allows that particular user to be chosen to receive job status information by email).
Configuring Access to S3 Buckets
The CloudMover initial state after installation contains no access to AWS S3 storage resources, and therefore, this access needs to be configured before CloudMover can perform any transfer work. You can verify this by clicking on the New Job tab, and seeing that there are no entries in the folder sections. The AWS S3 resources are configured on the System Settings tab with the S3 Settings Action selection. The Choose Folder pulldown includes an Add New S3 Folder selection, and this selection will display the configuration form, as shown in the following screen shot.
When the Add New S3 Folder drop-down is selected, the configuration items for creating access to a S3 Bucket are shown, including:
  • Top-level Folder Name - the name chosen here should be reflective of something the user will feel natural about selecting as a source or destination for CloudMover transfers. In the example shown below, the name AWS-S3-Virginin was chosen, which provides an indication to the user that the folder is in AWS. Alternatively, a generic name that obfuscates the underlying location and storage technology could be chosen, such as Nearline_1. Any properly formatted directory name can be chosen.
  • S3 Bucket Name - This bucket name must correlate exactly to the bucket name that is found in the AWS S3 Bucket console (these names are globally unique). It the example shown, this represents a bucket that is owned by Pixspan.
  • S3 Region - This entry must correspond to the AWS Region where the S3 Bucket resides. The dropdown contains all regions where AWS S3 is supported.
  • S3 Security Type - There are three S3 Bucket access mechanisms that can be employed by CloudMover. The default is to use the IAM role that was created during the CloudMover Installation process. This is the easiest mechanism to configure, as there are no other configuration items to enter. This selection only applies to S3 Buckets that are owned by the AWS Account that installed CloudMover. The two other options are typically used for transfers to or from buckets owned by external entities. Specify Credentials requires an Access Key and a Secret Access Key (created by AWS during the S3 Bucket creation procedure). Specify Temporary Credentials adds a field for Token, which is created by the AWS Security Token Service (STS). The AWS STS method is typically used when access is being provided to a S3 Bucket outside of the administrative domain where CloudMover is running.
Configuring CloudMover Workers
Each S3 Region, that could possibly be a destination for a transfer, is required to have a Virtual Private Cloud (VPC) and Subnet configured for the Worker Instances that will run in that region. The process for this configuration step is:
  • Identify the possible destination S3 Regions (e.g. N. California, Frankfurt, Mumbai, Singapore, etc.)
  • Create a list with one VPC Id/Subnet Id pair for each destination Region
  • Enter the VCP/Subnet information on the Worker EC2 Settings Action entry form for each destination Region
The VPC and Subnet Information is available in the Amazon AWS Console. To locate the VPC settings, choose the Services pull-down and click on the VPC item under Networking & Content Delivery, as shown in the following diagram:
Navigation to a list of VPCs in a particular region is performed by selecting a Region from the pull-down list of VPC Resource as shown in the following screen shot:
Choose the potential destination Region from the pull-down list. For example, hovering the mouse over the Asia Pacific (Singapore) Region will display an underline under that selection as shown in the following screen shot:
Clicking on the Singapore entry will then display the VPC Dashboard for the Singapore Region, showing all of the VPCs that can host the CloudMover Worker Instances. Choose one of the available VPCs, and note the ID for that VPC. This ID will be in the form of "vpc-xyzzy123". Note that in the following screen shot, the actual VPC ID has been purposely blurred for security purposes.
After identifying the desired VPC, the next step is to identify a subnet to be used that is within the specific VPC. This is found by clicking on the Subnets link on the left navigation bar. This action will bring up the table of available subnets. Each row of the subnet table includes the associated VPC, and care must be taken to make sure that the VPC ID listed in the table matches the VPC chosen in the previous step. Also, make sure that the Subnet allows for auto assigning public IPv4 addresses. The following screen shot of the AWS Management Console shows the subnet list (note that the Subnet IDs and the VPC IDs have been purposely blurred).
With these two pieces of information, the Worker EC2 Settings can be configured as shown in the following screen shot. Enter the VPC ID and the Subnet ID. If you are configuring a Worker in a Region that is outside of the Region where the Server is running, you must select Public IP for the Inter-Region Path to Server. To complete the operation, you must click on the 'Submit' button.
The above procedure for setting up Worker EC2 Settings needs to be repeated for every region that is a possible destination for a transfer (copy/move) operation.
Validating the Configuration
The most effective way to verify that the configuration is completed is to perform a transfer (copy/move) of content between S3 Folders or within the same folder. This is accomplished on CloudMover's New Job page, accessible by the New Job tab at the top of every page. This action can be performed by any Administrator or Wrangler user. With CloudMover configured with two buckets, one in Virginia, and one in California, a simple transfer can be initiated by dragging a sub-folder from one S3 folder and dropping it on another S3 folder, as shown in the following screen shot. It is suggested that the content used for this first verify job be limited in size, i.e. a small number of small files.
That drag-and-drop action will initiate a dialog box where the transfer can be further specified, as shown in the following diagram.
The following items should be specified for this simple configuration validation transfer job:
  • Job Name - Something that indicates that this is a validation job
  • Type - Choose 'Copy', as this is the simplest use case.
  • Destinaion Path - choose a folder name that will be recognized as a trial transfer so that it can be deleted later.
  • Recurse Directories - choose this if the directory chosen has no files at the top level.
  • Delivery Option - choose 'Express', as this is the least expensive option.
When the above entries are made, click on 'Submit Job' to start the job. To verify that this new verify job is operating correctly, view the job's progress on the Dashboard page. Note that in the process of initiating a new job, Worker resources are spun up in the destination region, and this takes approximately 30 seconds. At the conclusion of the job, the job box will disappear from the page, so to see if it ran correctly, you must click on the Show All radio button. Verify that the completed job is not indicated in either yellow or red, as that indicates a job with a warning or an errored job respectively. If the job was errored, click on the job box, and identify the error, as shown at the bottom of the dialog box under 'Error/Warning'. In the following screen shot, note the progress bar on the job, which indicates that the job is approximately 25% completed.
Each region that can be a transfer destination should be tested in the same way to validate that all operations work properly.
Security Considerations
Security of data and the infrastructure is of paramount importance when using the resources of a Public Cloud. Additionally, it's often a requirement for internal resources to only provide access to data and resources on a need-to-know basis. CloudMover and AWS include important security options, such that a safe environment can be easily maintained.
CloudMover User Role Policy
CloudMover provides the administrator the ability to control access via user account roles, including Administrator, Wrangler (perform the transfer functions), and View-only (see job progress only). The number of administrators should be kept to the absolute minimum, as administrators have access to all of the facilities of CloudMover. Even though Wrangler functions can be performed by the administrator role, administrators should avoid using that user login for data transfers, and instead, create a separate user login for themselves with the Wrangler role.
CloudMover Password Policy
User passwords in CloudMover are initiated by the user, and CloudMover does not dictate the form of the password. Note that CloudMover does not store passwords in the internal database. It salts the passwords and stores password hashes, which corresponds to best security practices for password handling.
The security approach taken is one of providing information to the administrator, not restrictive rules at the user level. The administrator should occasionally visit the User Admin page to review the list of users to make sure that users are following company policies. The user list provided with the List/Modify Users Action selection provides an insight into the strength of the password that each user has chosen. As seen in the following screen shot, the admin user and the 'daniel' user both have weak passwords, whereas the 'pat' user has a strong password. The 'iris' user is indicated as having an Admin Assigned password, which means that this user has never logged in and entered their own password. A user's password can be reset in the User Admin tab by selecting the Multiple User Actions Action, selecting the radio button for the user, and entering a password in the Reset area. This new temporary password will need to be communicated to the user in order for them to be able to log in.
Each user's password information within CloudMover's database is stored as a sha-256 hash which includes a random salt value, and the entire database record is also encrypted. Passwords are both unknowable and unrecoverable by the administrator, and they are practically unhackable by any method.
An administrator can force a user to put in a new password using the Multiple User Actions selection in the Action list. The specific (or multiple) user is selected by radio button, and the administrator enters a password in the Reset Password input box, and then hits Submit. This action will force the user to know that password, and then go through the password reset procedure that forces him or her to enter and confirm a new password. For users that continue to be uncooperative with the company's password policy, the administrator has the ability to Suspend or Delete that user. These capabilities are shown in the following screen shot.
Securing the CloudMover Server AWS Instance
The CloudFormation launch sequence will have configured a security group for the CloudMover Server Node Instance that opens the access port to universal access. The access port created is port 9090. This open access policy was chosen in order to minimize access issues after the install. Users are strongly encouraged to restrict this policy to just known hosts or networks before configuration items are entered or modified. This configuration is performed in the EC2 Dashboard under the 'Security Groups' left navigation, choosing the security group assigned to the CloudMover Server Instance. Choose the 'Inbound' tab, and then click the 'Edit' button. The following screen shot shows how multiple IP Networks or Addresses can be configured to allow access. Note that all entries with '0.0.0.0/0' have been removed, which means that the instance is only open to those networks and addresses that are explicitly specified. Even though CloudMover requires user login validation, the best security is achieved when the number of networks allowed to access CloudMover is kept to a minimum.
Troubleshooting and Tips
This section contains a discussion on the most common issues that a user will encounter when installing and using CloudMover.
EC2 Service Limits
For each region that CloudMover Workers execute in (regions where S3 destination buckets are located), CloudMover will dynamically spin up and terminate a various number of EC2 CloudMover Worker instances (dependent upon the Delivery Option selected).
Make sure that you have increased EC2 limits for every region that you will be operating in. For more detail, see the Prerequisites and Requirements – Pre-Installation Setup Requirements section (above on this page).
Public Internet Access for Head Node VPC
Upon launching CloudMover Head Node you will be asked to select a VPC to launch the EC2 instance into. If this is a single region solution or VPC Peering is employed to connect to all regions that apply to this solution a VPC without public internet access (igw-xxx…) may be selected.
However, in most multi-region CloudMover solutions, an Internet Gateway and appropriate Route Tables must be attached to the selected VPC in order for the CloudMover Workers to communicate to the CloudMover Head Node. Please check this before selecting the VPC within the CloudFormation template.
See Installation – Selecting the Virtual Private Cloud (VPC) and Subnet Location section above for more information.
Head Node Web Port Access
Access to the CloudMover Web UI is accessed via TCP port 9090. If there are any firewalls in the path between the browser client and the CloudMover browser server then port 9090 will have to be white listed.
Likewise, if there are any firewalls in the path between the CloudMover Head Node and CloudMover Workers (other regions) then port 9090 will have to be white-listed. This is because the CloudMover Workers communicate to the CloudMover Head Node via the Head Nodes’ RESTful interface over that port.
Note that the CloudMover Head Node has automatically created a Security Group attached to its instance that allows traffic to be accepted over port 9090 so no additional action is needed there.
See Installation – Restricting Access to the CloudMover Server section above for more information.
S3 Bucket Permissions
There are three permission options supported by CloudMover for accessing S3 buckets as described in Configuration – Configuring Access to S3 Buckets.
IAM Role:
  • Make sure the user that launched CloudMover Head Node has the following S3 privileges for S3:
    • Abort*, Create*, Delete*, Get*, Head*, List*, Put*
Specify Credentials:
  • Make sure that the user has permission and obtains a valid Access and Secret Access key for the bucket.
Temporary Credentials:
  • Make sure that the user has permission and obtains a valid Access, Secret Access key, and generated temporary token to be used when accessing the bucket.
If these credentials are not entered correctly, an error message will be displayed and the association between CloudMover and the S3 bucket will be disallowed. See Configuration – Configuring Access to S3 Buckets section above for more information.
EC2 Region Configuration for Multizone Operations
For single region operations, CloudMover will launch all CloudMover Workers within the same VPC and subnet that the CloudMover Head Node is running in. In that scenario, all communications between CloudMover Head Node and CloudMover Workers will be over the private IP network within that VPC/subnet and no additional configuration is required.
For multi-region operations, for virtually all scenarios (outside VPC Peering), the user should:
  • Make sure that for each region where S3 destination buckets are located that a VPC, subnet, and inter-region path to server is configured.
  • For inter-region path to server, it should typically be set to Public IP.
See Configuration – Configuring CloudMover Workers section above for more information.
Browser Support
Although Pixspan does not require a specific browser or version, it is highly recommended that the user interfaces with one of the latest versions of either Google Chrome, Mozilla Firefox®, MS Windows Internet Explorer® 10 (IE10), or Microsoft Edge® as these were the browsers that were utilized in the development and test of CloudMover.
Note that a warning prompt may appear when you first bring up CloudMover in the web browser. This warning may appear like the following prompt:
CloudMover uses HTTPS to keep the communications between CloudMover and the client browser secure, and it ships with a Pixspan-self-signed certificate. Because the certificate is self-signed, the browser warns the user that it might be invalid. The user must add an exception through the browser interface so that the certificate is accepted. Instructions to accomplish this can be found at the links associated with these searches:
The systems administrator can configure CloudMover to avoid these warnings by installing a certificate provided by an official SSL Digital Certificate Authority. Please contact your systems administrator if this is a requirement.
Amazon CloudWatch Monitoring Tool
Amazon CloudWatch is a monitoring service for AWS resources and applications.
CloudWatch offers free basic monitoring for your resources and is enabled by default. In the case of CloudMover you will be most interested in monitoring the activity of the CloudMover Head Node EC2 resource and potentially S3 resources that you typically use in your daily activity.
With CloudWatch, you can collect and track metrics, collect and monitor log files, and set alarms. An overview of CloudWatch can be found at CloudWatch Overview
In addition to understanding the details of the Free tier service of CloudWatch you might want to explore their expanded capabilities at CloudWatch Pricing
Pixspan recommends that the customer monitors these metrics for CloudMover Head Node:
  • CPU Utilization
  • Network Bytes In
  • Network Bytes Out
  • Status Check Failed (Any) Count
  • Status Check Failed (Instance) Count
  • Status Check Failed (System) Count
These metrics are enabled by default so they should not have to be enabled. They can be viewed in the AWS Management Console under EC2 Services. Select the CloudMover Head Node Instance and in the lower window of the console pane, select the Monitoring tab and view the metrics.
AWS CloudTrail Logging Tool
CloudTrail is an AWS service that records user and API activity for various AWS resources and is enabled by default but must be configured.
CloudTrail logs information on who made a request, the services used, the actions performed, parameters for actions, and the response elements returned by the AWS service. CloudTrail logs are then stored in an S3 bucket or a CloudWatch Logs log group that the customer specifies.
CloudTrail can be used to help ensure compliance and regulatory standards by tracking the activity within the account.
An overview of CloudTrail can be found at CloudTrail Overview
For pricing examples please visit the following site: CloudTrail Pricing
As this is an advanced AWS service, Pixspan leaves it to the customer to determine if CloudTrail is a beneficial service for them.
EC2 Usage/Requirements
The Server Node application is offered as a no-cost AMI ($0.00/hr), and runs on a low-powered EC2 Instance type, the m5.xlarge (AWS charges of approximately $0.22/hr). The instance running cost can be further minimized by stopping the Instance while it's not being used. If the Server instance is restarted, and an Elastic IP address has not been assigned, a new public IP address and DNS name will be assigned on the restart. Pixspan charges are introduced only when Worker Node Instances are launched to perform the copy/move operations. Pixspan charges by the hour for Worker Instances based on the desired speed of the transfer.
Operational Considerations
This Server Node is a no-charge instance, and it is intended to operate on a m5.xlarge EC2 Instance type. The cost to operate the CloudMover Server, therefore, is solely dependent on the AWS charge for the running instance. The cost of operating CloudMover for an entire year is under $2,000 (about $5.50/day) with an on-demand instance, and that cost can be reduced by employing a reserved instance. The cost can be further reduced if CloudMover isn't going to be continually used by selecting an on-demand instance, and then stopping the instance whenever it is not in use performing an operation.
CloudMover Backup and Recovery
The CloudMover system can be backed up by taking a copy of the database. This should only be performed by an IT professional who is skilled in Linux Administration. Note that in subsequent releases, Backup and Restore will be available as a UI option.
Backup
Perform the following steps for backing up the CloudMover database. A buckup can be generated at any time during the life of your CloudMover product.
  1. Open a terminal window in Linux, and ssh into the CloudMover instance using the ssh key that was generated for that instance: 'ssh -i ~/.ssh/<private key name> ec2-user@<Public DNS>'
  2. Stop the CloudMover Server: 'sudo /opt/pixspan/pixmover/bin/pixmoversrv stop'
  3. Copy the database to the EC2 User home directory: 'sudo cp /opt/pixspan/pixmover/dat/pixmoversrv.db /home/ec2-user'
  4. Change the ownership of that database: 'chown ec2-user:ec2-user pixmoversrv.db'
  5. Start the CloudMover Server: 'sudo /opt/pixspan/pixmover/bin/pixmoversrv start'
  6. Log out of the EC2 instance: 'exit'
  7. Navigate to the directory, using the 'cd' command, where you want the backup to be stored.
  8. Copy the backup file from the CloudMover Server to the local directory: 'scp -i ~/.ssh/<private key name> ec2-user@<Public DNS>:/home/ec2-user/pixmoversrv.db .'
  9. This stored DB can now be used in the restore procedure.
  10. Verify that the CloudMover Server is running by connecting using your browser (instructions above).
Restore
Perform the following steps for restoring the CloudMover Server from a database that is stored locally. This can be performed in the case where you want to bring the configured database from a previous version to a subsequent version, thereby restoring all of the user and S3 Bucket configuration items, among others.
  1. Open a terminal window on your Linux system.
  2. Navigate to the directory where the saved database is located (using the 'cd' command).
  3. Copy the database file to the CloudMover EC2 Instance in the home directory: 'scp -i ~/.ssh/<private key name> pixmoversrv.db ec2-user@<Public DNS>:/home/ec2-user'
  4. Use ssh to log into the Instance: 'ssh -i ~/.ssh/<private key name> ec2-user@<Public DNS>'.
  5. Stop the CloudMover Server: 'sudo /opt/pixspan/pixmover/bin/pixmoversrv stop'
  6. Make a backup copy of the current DB: 'sudo cp /opt/pixspan/pixmover/dat/pixmoversrv.db /opt/pixspan/pixmover/dat/pixmoversrv.db.orig'
  7. Copy the saved DB into the CloudMover database directory: 'sudo cp /home/ec2-user/pixmoversrv.db /opt/pixspan/pixmover/dat'
  8. Start the CloudMover Server: 'sudo /opt/pixspan/pixmover/bin/pixmoversrv start'
  9. Log out using the 'exit' command.
  10. Verify that the CloudMover Server is running by connecting using your browser (instructions above).
Maintaining CloudMover
It is important that the CloudMover Instance gets periodic security and operating system updates. The following steps will accomplish that:
  1. Open a terminal window in Linux, and ssh into the CloudMover instance using the ssh key that was generated for that instance: 'ssh -i ~/.ssh/<private key name> ec2-user@<Public DNS>'
  2. Perform an update with the command: 'sudo yum -y update'
  3. Logout with the 'exit' command.
Pixspan Technical Support
If you are having trouble operating CloudMover, you can contact Pixspan with your issue via email at support@pixspan.com, or create an account and submit an issue at the Pixspan Support site.
References
Please refer to the CloudMover page on the Pixspan Website for further product references.
Localization
CloudMover is currently implemented with support for the English language, and dates follow the US standard. Please contact Pixspan if you have specific localization requirements at Pixspan Support
CloudMover
Snowball Loader
Global WAN Transfer
PRODUCTS & MARKETS
TECHNOLOGY
CASE STUDIES
ABOUT