Uploading Large files to Amazon S3 from the Command Line

Ran into a brick wall when trying to upload some of these large video files to the Amazon S3 buckets. Tried converting the 2Gb QuickTime file to an .mp4 which was a quarter of that size, but still no luck. Even tried uploading a zip-compressed version of that file, not yet knowing if I’d be able to unzip it.

Soon learned that AWS offers Amazon S3 Multi-part uploads, which can be achieved on OSX with an App like MCSTools-S3, which is fairly inexpensive at like twenty-something bucks, or – my choice: a command line library developed in Python by Michal Ludvig called s3cmd. Free and open source.

What a nice alternative to the web portal, too.

I created a user for myself via the Amazon IAM service and for permissions attached a policy that grants AmazonS3FullAccess. Then simply followed the s3cmd docs to s3cmd --configure (Configuration saved to '/Users/username/.s3cfg').

Awesomely it automatically uses Multi-part upload for files larger than 15mb – but it can be s3cmd multi-part upload disabled or configured.

In order for items to be publicly accessible via the web, permissions need to be configured accordingly, which can also be done with s3cmd. There’s a relative full list of commands here in the docs.

UPDATE: With the above policy, I was unable to s3cmd cflis so under IAM policies I added a policy, AdministratorAccess to the user I’m using s3cmd with.