August 2012
« Jul   Sep »

Amazon : Mounting S3 bucket as drive

This is few rare scenarios, you may need to use S3 storage mounted as a drive. Here I would need this S3 storage as common network location to reach both my recorded files and user upload files in between streaming and web server.

There is a Opensource program called “s3fs” which is a Google product help to mount the S3 storage as a drive over the internet. Before start to install, We may need to install the “FUSE [ a kernel module which helps to support S3 storage system] the version 2.8.4 or higher. I found that rpmforge and other repos are still not been updated with the latest fuse version. So we need to uninstall it if you had installed.

1. Removing aged FUSE rpm packages

[root@host_web ~]yum remove fuse fuse* fuse-devel

2. Install some package require for FUSE installs.

[root@host_web ~]#yum install gcc libstdc++-devel gcc-c++ curl curl* curl-devel libxml2 libxml2* libxml2-devel openssl-devel mailcap

3.Install FUSE from the source

[root@host_web ~] wget
[root@web fuse-2.9.1]# tar -zxvf fuse-2.9.1.tar.gz
[root@web fuse-2.9.1]# cd fuse-2.9.1 && ./configure --prefix=/usr
[root@web fuse-2.9.1]# yum install make  util-linux
[root@web fuse-2.9.1]# make && make install

4.Configure FUSE in the system:

[root@reporter_web installation]# export PKG_CONFIG_PATH=/usr/lib/pkgconfig:/usr/lib64/pkgconfig/
[root@reporter_web installation]# ldconfig
[root@reporter_web installation]# modprobe fuse
[root@web fuse-2.9.1]# pkg-config --modversion fuse

5. Installing S3FS

[root@web fuse-2.9.1]# wget
[root@web fuse-2.9.1]#tar -zxvf  s3fs-1.61.tar.gz
[root@web fuse-2.9.1]#cd s3fs-1.61 && sh configure && make && make install

All the installation has been completed.

Creating a new IAM user for S3 access

Now we need to configure environment varibles for S3 credentials, There are different type of methods

1. Using the passwd_file command line option
2. Setting the AWSACCESSKEYID and AWSSECRETACCESSKEY environment variables
3. Using a .passwd-s3fs file in your home directory
4. Using the system-wide /etc/passwd-s3fs file
( I would suggest this mechanism)

I’m created new file “/etc/passwd-s3fs” and added the S3 credential details
Bucketname:Access ID:Access Key

[root@reporter_web s3fs-1.61]# cat /etc/passwd-s3fs
[root@reporter_web s3fs-1.61]#chmod 600  /etc/passwd-s3fs

Mounting S3 storage

Usage : s3fs S3storage_name mount point options

[root@reporter_web s3fs-1.61]# s3fs clouddatabank  /mnt -ouse_cache=/tmp

Verify the mount point

[root@reporter_web s3fs-1.61]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1            7.9G  1.7G  6.2G  21% /
tmpfs                 839M     0  839M   0% /dev/shm
/dev/xvdb1             50G  4.3G   43G  10% /home
s3fs                  256T     0  256T   0% /mnt
[root@reporter_web s3fs-1.61]#

More specifically, you may mount the S3 file system to be the ownership of a user or group if other programs like Apache/ftp wants to access it. In my scenario, I’m using Adobe FMS service on Amazon and I need fms user and group have full access to this S3 storage since all the files are written directly to S3.

s3fs fms-storage -o use_cache=/tmp -o allow_other,uid=500,gid=500 /mnt/fms_files

2 comments to Amazon : Mounting S3 bucket as drive

Leave a Reply

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>