Large Scale Data Migration using AWS DataSync Agent

Introduction

Panorama Inc.(pseudonym), a leading movie production organization, needed to migrate terabytes of data from their on-premises servers to the Amazon Web Services (AWS) cloud. They have a large set of multimedia data stored in on-prem data devices from Quantum and Synology. Panorama wanted a secure, cost-efficient solution to ensure minimal downtime and maximum data integrity during transfer.

Background

Panorama Inc. faced a significant challenge in managing the growing volume of data generated. On-prem storage includes 800 TB of storage in a SAN device and another 1000 TB of backup in an on-prem  NAS device from Synology. These specialized devices need support and maintenance and incur huge CapEx and OpEx. As storage requirements grew, more users from different regions tried to access this data.

Requirements

Panorama had a crucial requirement for an efficient solution to migrate its data to the Amazon Web Services (AWS) cloud. After careful analysis, they chose High Plains Computing (HPC) as their consultant for data migration. Their specific requirements included: 

• Hosting all data in the cloud 
• Categorizing hot and cold data and storing cold data in cloud archival storage and hot data in high-performance AWS FSX for Windows 
• Pay-as-you-go model as on-prem data storage technologies like SAN are expensive. 
• Having unlimited scalability with no additional maintenance and support costs for on-premise storage 
• Automatic backup of data 
• They were ensuring security without needing specialized security staff to secure data.

Proposed Solution & Setup

The HPC team selected AWS  DataSync to migrate data to the cloud. AWS DataSync is a secure online service that automates and accelerates moving data between on-premises and AWS Storage services. DataSync can copy data between on-prem storage and various AWS cloud storage services.

• Amazon Simple Storage Service (Amazon S3) buckets, 
• Amazon Elastic File System (Amazon EFS) file systems,
• Amazon FSx for Windows File Server file systems, 

The figure below shows the High-level architecture of this data migration solution.

The solution’s components include

1. On-prem SAN is an SMB server, so Windows and MacOS clients can map various network drives to shared folders.
2. The Data sync agent was installed on one of the on-prem VMs running on Microsoft Hyper-V. The Datasync agent can run on any Linux server. We ran multiple copies of data sync agents, each with its own unique folder to sync. This sped up the sync process. We also configured file copy patterns to reject temp file copy. 
3. Copy process would write copied files on S3 objects with the object class of “Glacier Deep Archive”. This reduced the cost of storage tremendously.
4. Selected data folders that held files related to recent projects were copied to Amazon FSx for the Windows file server. This allowed content creators to access and save their content on the cloud with ease and great speed. 

Challenges

The data migration project presented significant challenges due to its massive scale. The migration process needed to maintain ongoing operations and access to essential data was not interrupted. Additionally, the utmost importance was given to maintaining the accuracy and integrity of the data during the migration, as any errors or loss could have severe consequences. Panorama aimed to find a cost-effective solution that optimized resource utilization and minimized expenses associated with the migration. The HPC team, which had ample experience in similar projects, successfully deployed the solution within the cost, quality, and time parameters.

Results

Panorama Inc. achieved remarkable results through large-scale data migration, which included several benefits:

• Zero downtime during the migration process, ensuring uninterrupted access to critical data.
• We are maintaining data integrity and reliability throughout the migration process.

Cost savings

• 1 TB of on-premises storage costs an average of $200 per year, including 5-year asset amortization and yearly support service contract costs of $400.
• 1 TB of storage in profound archival costs about $25 per year.
• 1 TB of storage in high-performance FSX volume costs $960 per year.
• By storing only 3-4% of recent data in FSX volume and the rest in deep archival storage, Panorama can save tremendous money on 1000 TB of data.3-4% data in FSX volume and the rest in deep archival storage; Panorama saved tremendously on 1000 TB of data. The table below shows CapX and Opex costs (in thousands of USD) per year and the staffing count based on 5-year ROI. AWS Cloud base solution has significantly less CapEx and OpEx spending per year and reduced staffing requirements to support infrastructure

Testimonials