Rss Feed Tweeter button Facebook button Technorati button Reddit button Linkedin button Webonews button Delicious button Digg button Flickr button Stumbleupon button Newsvine button

A Waage Blog

Ruby, Rails, Life

Archive for the ‘Amazon Web Services’ Category

How to mount an Amazon S3 bucket as virtual drive on CentOS 5.2

with 12 comments

#Note: If you are using CentOS 4, it’s the same general process. You might have more difficulty finding the packages to install fuse and dependencies.

This is a simple guide on how to mount your S3 bucket as a “virtual drive”. This is great for backing up your data to S3, or downloading a bunch of files from S3.

#First, make sure you have the fuse package installed.

#On CentOS, fuse is available from RPMforge


#Now install fuse

yum install fuse
modprobe fuse

#Download s3fs and make

cd /usr/local/src
cd s3fs

#Copy the binary to /usr/local/bin (or wherever you prefer)

cp s3fs /usr/local/bin

#Make a mount point

mkdir /mnt/s3drive

#Mount your bucket like this:


That’s it ! You can change directory to your virtual drive or start copying files !
Go ahead and use a visual client such as CyberDuck or S3Hub to verify with your own eyes that this actually worked. :)
Good luck!

Written by Andrew Waage

April 6th, 2010 at 6:38 pm

Posted in Amazon Web Services, linux

Tagged with , , , , ,

How to Set up DB server on Amazon EC2 with data stored on EBS drive formatted with XFS

with 17 comments

Working with Amazon Web Services (AWS) such as EC2, S3, and EBS takes some time to get familiar with all the tools, acronyms, and especially paradigm of thinking about servers. Once you get into it, you quickly find much old (outdated) documentation, and you can easily find yourself spending hours searching the web for blog posts to help in your quest!

After much research, I decided to create a Mysql database server on a ‘instance-store’ small ec2 server, with all the precious Mysql data stored on a EBS volume attached to my server. This way, I could easily create snapshot backups of my databases for FAST backups. Here’s the procedure I decided on. It involves symlinking Mysql config files and data directories onto the EBS volume.

Another trick I used because I needed to migrate about 20 GiB’s of data to get started, was that I initially set up an “X-tra large” instance, with 10 GiB’s RAM to handle the data import. After the data was migrated and imported to my database, I simply terminated my X-Large instance and spun up a small instance connected to the same EBS volume! All the databases were preserved nicely and I did not have to waste money paying for an X-Large instance anymore. This exemplifies the value of thinking in the “cloud” mindset – where you can spin up and down servers in a matter of seconds! Hope this article helps someone else out there!

Great Reference:

# Create small instance (I used the AMI for i.386 CentOS 5.4)

# Create EBS volume and attach it to the EC2 (as /dev/sdh in my example)

# Install mysql-server

yum install mysql-server

# Install xfsprogs so you can format EBS drive with XFS

yum install xfsprogs
modprobe xfs
mkfs.xfs /dev/sdh

# Add to your /etc/fstab so it mounts on boot

echo "/dev/sdh /vol xfs noatime 0 0" | sudo tee -a /etc/fstab
sudo mkdir -m 000 /vol
sudo mount /vol

# Create directories that will store persistant Mysql data

mkdir /vol/etc /vol/lib /vol/log

# Move the mysql config and data files to your newly mounted drive

mv /etc/my.cnf  /vol/etc/
mv /var/lib/mysql /vol/lib/
mv /var/log/mysqld.log /vol/log/

# Symlink the mysql directories so that mysql uses the mounted EBS drive

ln -s /vol/lib/mysql /var/lib/mysql
ln -s /vol/etc/my.cnf /etc/my.cnf
ln -s /vol/log/mysqld.log /var/log/mysqld.log

Now restart your Mysql server and check your logs if you have any errors!

service mysqld start

# If you have errors, watch the logs
tail -f /var/log/mysqld.log

Good luck !

Written by Andrew Waage

March 19th, 2010 at 3:56 pm