September 2012
« Aug   Oct »

Synchronize Amazon S3 location to Local storage

s3tools is a great tool for automating file operation on Amazon S3 storage. We can use this tools to backup our files to S3 and mirror the S3 files to local disk or storage.

My Scenario :

I’m running an video application and FMS server and web server are on two geographical location. So it has to be need to move all the user upload videos to this FMS server. There is an option in my webserver to upload videos. These videos are directly uploaded to S3 and I need it to be moved back to FMS server.

Process involving are,

1, User upload video through the web application.
2. These videos are stored in S3 directly ie I’ve used “s3fs” program to map the S2 location as drive in Web server.
3. Application initiate a url/api call on fms server.
4. When FMS server got a url call, it execute a s3tools to complete the sync activities at FMS server.

So all the user upload videos are reached FMS and ready to stream instantly 🙂

Tools used
1. s3fs – FUSE-based file system backed by Amazon S3
2. Command line S3 client ..

How do I synchronize the S3 location to locally over S3tools

Here the command to do this,

FMSserver# /usr/bin/s3cmd sync --delete-removed   -c /ebs/scripts/.s3cfg --include '*.flv' --include '*.mp4' s3://myvideolocation/streams/help_videos/streams/_definst_/  /streamer/applications/help_videos/streams/_definst_/

-c /ebs/scripts/.s3cfg : Location of S3 credentials are copied for s3tools program.
s3://myvideolocation/streams/help_videos/streams/_definst_/ : S3location of the videos.
/streamer/applications/help_videos/streams/_definst_/ : Local video location where FMS server has been configured to play.

root@localhost#s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3
Secret Key: 7Kl+2O2X/uRCWbvsMV2m+isM3K
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP and can't be used if you're behind a proxy
Use HTTPS protocol [No]:
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't conect to S3 directly
HTTP Proxy server name:
New settings:
  Access Key: AKIAIIsdfdOTG4XQ
  Secret Key: 7Kl+2O2XvdfsdvsMfV2m+isM3K
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>