Seeding Torrents with Amazon S3 and s3cmd on Ubuntu
Again a nice post by Eric Hammond . Hope its useful for some of you out there…
Amazon Web Services is such a huge, complex service with so many products and features that sometimes very simple but powerful features fall through the cracks when you’re reading the extensive documentation.
One of these features, which has been around for a very long time, is the ability to use AWS to seed (serve) downloadable files using the BitTorrent™ protocol. You don’t need to run EC2 instances and set up software. In fact, you don’t need to do anything except upload your files to S3 and make them publicly available.
Any file available for normal HTTP download in S3 is also available for download through a torrent. All you need to do is append the string
?torrent to the end of the URL and Amazon S3 takes care of the rest.
Let’s walk through uploading a file to S3 and accessing it with a torrent client using Ubuntu as our local system. This approach uses
s3cmd to upload the file to S3, but any other S3 software can get the job done, too.
- Install the useful
s3cmdtool and set up a configuration file for it. This is a one time step:
sudo apt-get install s3cmd s3cmd --configure
The configure phase will prompt for your AWS access key id and AWS secret access key. These are stored in
$HOME/.s3cmdwhich you should protect. You can press [Enter] for the encryption password and GPG program. I prefer “Yes” for using the HTTPS protocol, especially if I am using s3cmd from outside of EC2.
- Create an S3 bucket and upload the file with public access:
bucket=YOURBUCKETNAME filename=FILETOUPLOAD basename=$(basename $filename) s3cmd mb s3://$bucket s3cmd put --acl-public $filename s3://$bucket/$basename
- Display the URLs which can be used to access the file through normal web download and through a torrent:
cat <<EOM web: http://$bucket.s3.amazonaws.com/$basename torrent: http://$bucket.s3.amazonaws.com/$basename?torrent EOM
- The above process makes your file publicly available to anybody in the world. Don’t use this for anything you wish to keep private.
- You will pay standard S3 network charges for all downloads from S3 including the initial torrent seeding. You do not pay for network transfers between torrent peers once folks are serving the file chunks to each other.
- You cannot throttle the rate or frequency of downloads from S3. You can turn off access to prevent further downloads, but monitoring accesses and usage is not entirely real time.
- If your file is not popular enough for other torrent peers to be actively serving it, then every person who downloads it will transfer the entire content from S3’s torrent servers.
- If people know what they are doing, they can easily remove “?torrent” and download the entire file direct from S3, perhaps resulting in a higher cost to you. So as a work-around just download the ?torrent URL, save the torrent file, and upload it back to S3 as a .torrent file. Share the torrent file itself, not the ?torrent URL. Since nobody will know the URL of the original file, they can only download it via the torrent.You don’t even need to share the .torrent file using S3.SOURCE