How to copy data from the AWS s3 bucket to the GCP GCS Bucket?

Prerequisites

Accounts and Permissions

• Ensure the user has active AWS and GCP accounts.
• Discuss the need for proper IAM (Identity and Access Management) roles and permissions on both AWS and GCP.
• AWS: Permissions like s3:GetObject, s3:ListBucket.
• GCP: Permissions like storage.objects.create, storage.buckets.create.

Tools and Software Requirements

• AWS CLI: A command-line interface for interacting with AWS services. Provide installation and setup instructions.
• Google Cloud SDK (gsutil): A command-line tool for interacting with GCS. Include steps to install and authenticate with GCP.

Configure the AWS CLI

To open a new Cloud Shell tab.

set the project ID as an environment variable

export PROJECT_ID=$(gcloud config get-value project)
gcloud config set project $PROJECT_ID

Download and install the AWS CLI

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Configure the AWS CLI

Make sure you are ready with Access Key ID and Secret Access Key ID

Follow docs to create https://medium.com/@aslam.muhammedclt/generate-access-key-and-secret-access-key-using-aws-iam-18fc0c39fcff

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html

Put cmd --> aws configure 
At the prompt, enter the following:

Property Value
AWS Access Key ID ____
AWS Secret Access Key ____
Default region name us-east-1
Default output format json

List Objects in S3 Bucket

aws s3 ls s3://your-s3-bucket-name

Copy Individual Files

aws s3 cp s3://your-s3-bucket-name/your-file.txt gs://your-gcs-bucket-name/your-file.txt

Copy Entire Directory

aws s3 cp s3://your-s3-bucket-name/your-directory/ gs://your-gcs-bucket-name/your-directory/ --recursive