s3hop – Cross-account S3 bucket streaming copy

A command-line tool to efficiently copy files between S3 buckets across different AWS accounts.

• Copy files between S3 buckets across different AWS accounts
• Smart file comparison to only copy new or updated files
• Progress tracking with ETA and transfer speed
• Detailed transfer summary with file type statistics
• Handles large files and directories efficiently using streaming
• Preserves file structure between source and destination

# Installpip install s3hop# Uses3hop source-profile s3://source-bucket/prefix/ dest-profile s3://dest-bucket/prefix/

You can install this package directly from PyPI:

pip install s3hop

Or install from source:

git clone https://github.com/thehamsti/s3hop.git
cd s3hop
pip install .

1. Python 3.6 or later
2. AWS credentials configured for both source and destination accounts
3. Appropriate S3 permissions in both accounts

1. Configure your AWS credentials for both accounts in ~/.aws/credentials:

[source-profile]aws_access_key_id = YOUR_SOURCE_ACCESS_KEYaws_secret_access_key = YOUR_SOURCE_SECRET_KEY[dest-profile]aws_access_key_id = YOUR_DEST_ACCESS_KEYaws_secret_access_key = YOUR_DEST_SECRET_KEY

2. Ensure you have the necessary S3 permissions:
   • Source bucket: s3:ListBucket, s3:GetObject
   • Destination bucket: s3:ListBucket, s3:PutObject

Basic usage:

s3hop source-profile s3://source-bucket/prefix/ dest-profile s3://dest-bucket/prefix/

Example with specific profiles and paths:

s3hop prod s3://prod-bucket/data/ staging s3://staging-bucket/backup/

• source-profile: AWS profile for the source account
• source_url: Source S3 URL (s3://bucket-name/prefix/)
• dest-profile: AWS profile for the destination account
• dest_url: Destination S3 URL (s3://bucket-name/prefix/)

1. Smart File Comparison

   • Compares files using ETags and last modified timestamps
   • Only copies new or updated files
   • Preserves existing files that haven’t changed

2. Progress Tracking

   • Real-time transfer speed
   • Estimated time remaining
   • Progress bar with file counts
   • Total data transferred/remaining

3. Transfer Summary

   • Start and end times
   • Duration
   • Number of files transferred/skipped
   • File type statistics
   • Failed transfers (if any)