Google Cloud offers a service called “Storage Transfer Service” that allows you to migrate data from one cloud storage bucket to another, from an on-premises data center to a Google Cloud Storage bucket, or even between different Google Cloud projects. This service simplifies the process of transferring data and can be particularly useful when performing data migrations.

Here’s how you can use the Google Cloud Storage Transfer Service to perform a migration:

• Set Up Google Cloud: Ensure that you have a Google Cloud account and project set up. You’ll need the necessary permissions to create and configure a transfer job.
• Activate the Service: Make sure that the Google Cloud Storage Transfer Service is enabled for your project. You can enable it in the Google Cloud Console.
• Create a Transfer Job:
  • Go to the Google Cloud Console.
  • Navigate to “Storage” and then “Transfer” under the “Storage” section.
• From here, you can create a new transfer job by providing the following information:
  • Source and Destination: Specify the source and destination for your data transfer. This can be a Google Cloud Storage bucket, an HTTP/HTTPS URL, or an on-premises location.
  • Transfer Options: Configure options such as data filters, scheduling, and overwrite behavior.
  • Notification Options: You can set up email notifications for transfer job status updates.
  • Advanced Options: You can configure additional settings, such as object attributes, encryption, and other transfer settings.

• Test the Transfer Job: Before executing the transfer job, you can run a test to ensure that the job is set up correctly and that it will transfer data as expected.
• Execute the Transfer Job: Once you’re satisfied with the job configuration and the test run, you can execute the transfer job.
• Monitor Progress: You can monitor the progress of the data transfer through the Google Cloud Console. The job status will indicate whether the transfer is ongoing, completed, or if there are any errors.
• Cleanup: After the transfer is completed, you can clean up any resources or jobs that are no longer needed.

It’s important to note that the Storage Transfer Service is a paid service, and costs depend on the volume of data transferred and the specific configuration of your transfer job.

Event-driven transfer capabilities for Storage Transfer Service

Storage Transfer Service (STS) offers a managed way for you to quickly and securely transfer data between object and file storage across Google Cloud, Amazon, Azure, on-premises, and more. We are thrilled to announce two exciting launches for STS at Google Cloud Next ‘23.  

First, STS now offers GA support for event-driven transfer — a serverless, real-time replication capability to copy data from AWS S3 to Google’s Cloud Storage and between multiple Cloud Storage buckets. With this new support, event-driven transfers can deliver data from AWS S3 to Cloud Storage within a matter of minutes. This accelerates and simplifies building multi-cloud analytics pipelines. 

Additionally, STS now offers support for effortlessly transferring data – in the order of petabytes – from on-premises Hadoop Distributed File Systems (HDFS) sources to Cloud Storage. This is currently available for trusted testers and will be publicly accessible in coming months. 

Google Cloud provides comprehensive documentation and support for the Storage Transfer Service. Additionally, consider any data transfer costs and plan your migration accordingly.