PorterTech

Phones, servers, & user experience

Successful mobile applications are enjoyable to use, offering a intuitive, useful, and responsive set of features.

Fantasy Cricket - The infrastructure.

We at Gastown Labs feel like sharing what we have worked on, so here is an outline of the infrastructure for http://fantasy.cricket.com. Fantasy Cricket lives on the Rackspace Cloud (Cloud Servers), providing on-demand resources to handle traffic spikes and growth.

A user makes a request for fantasy.cricket.com, which resolves to a network load balancer. The load balancer determines which application server will serve the request, however, the load balancer also serves the static content for the request (style sheets, images, etc). The application server will churn away at the request, pulling data from cache or querying the database. The database master handles all of the application writes, where the two replicas serve the read requests. Live cricket scores are updated via XML feeds uploaded to the load balancer and processed with a background process running on one of the application servers.

Load Balancer

Nginx is used to handle web requests and act as a reverse proxy to the application servers using its HTTP proxy module. The load balancer has a complete copy of the application, including static content which is served directly from here. Nginx uses its Memcached module to cache specific page assets, reducing future page loading times. VSFTPD is used to allow specific live cricket score feed providers to update the application, XML files are uploaded via FTP to a specified folder for processing.

Application Servers

Nginx sends requests directly to the Unicorn worker pool over a Unix Domain Socket, Unicorn workers are responsible for executing the application. There is an excellent post on Unicorn in detail at http://tomayko.com/writings/unicorn-is-unix explaining on why it's so awesome. Background processes are distributed amongst the application servers, they are responsible for processing XML feeds and updating team scores etc. Each application server runs Memcached, each a part of the cache pool (volatile) used by the application and load balancer.

Database Servers

MySQL is our relational database of choice, Fantasy Cricket uses statement-based replication. All write operations are performed to the master, which are then replicated to the two slaves. Read operations are balanced between slaves, more slaves can be added to deal with an increase in read operations. Each database server runs MemcacheDB, not to be confused with Memcached. MemcacheDB is a distributed key-value storage system for persistent storage, used to cache assets that are too expensive to place in a volatile cache.

Nginx :: Serving static content with style.

Whether your application is written in php or ruby, you need a web server to serve static content (images, stylesheets, javascript, etc) and proxy your backend (unless your using Apache with modules... yuck). Nginx happens to be my favorite http server and reverse proxy, you can find out more about Nginx Here. I thought I would share some basic performance settings that can be applied to almost any application. Most of the settings bellow can be placed in several context (http, server, location), I have labeled where I usually place them.

Context: "http {"

Sendfile uses the systems kernel sendfile support, allowing Nginx to ignore the contents of the file it's sending (doesn't use its own resources).

     sendfile on;

TCP no push is to send http response headers in a single network packet, reducing network overhead.

    tcp_nopush on;

TCP no delay is to disable the nagle buffering algorithm, only for when information can be sent without an immediate response. 

    tcp_nodelay off;

The keep alive timeout is how long Nginx will keep a connection open with a client. I tend to set this value to the time it takes to load the slowest page, but keep it short enough so not to ignore waiting clients. If Nginx is to be behind Amazon's ELB (Elastic Load Balancing) the keep alive timeout should be set to 0 to prevent sticky connections to a single EC2 instance.

    keepalive_timeout 2;

Gzip allows for real time compression of static content, squeezing those fat bits into smaller packages. Gzip reduces bandwidth consumption and allows for quicker page loads when a client has a limited/slow internet connection.

    gzip on;

    gzip_http_version 1.0;

    gzip_proxied any;

Gzip buffers is for changing the default buffer size (allowing for 32KB compressed responses) to something more accommodating, in this case 128KB (16 x 8).

    gzip_buffers 16 8k;

Gzip comp level is for specifying the compression level 1-9, 1 being the fastest but least compressed. I usually choose a level of 2 or 3, to keep the system impact to a minimum while still providing a decent compression.

    gzip_comp_level 2;

Gzip min length is for specifying the minimum file size to gzip. In this case, if a file is smaller than 360KB, don't bother with compression.

    gzip_min_length 360;

Gzip disable is for disabling compression for certain user-agents that can't handle gzip correctly such as internet explorer 6.

    gzip_disable "MSIE [1-6]\.";

Gzip types enables gzip for mime-types other than the default text/html. Compress the usual suspects.

    gzip_types text/plain text/xml text/css

                      text/comma-separated-values

                      text/javascript application/x-javascript

                      application/atom+xml;

Context "server {"

Some clients send bad (non root) favicon.ico requests, generating 404's and populating the log file. Ensure that all favicon requests are redirected to the favicon in root.

    rewrite ^/(.*)/favicon.ico$ /favicon.ico last;

Serving the same static content to a client twice is a waste of resources, send expire headers to never expire cached files.

    location ~* \.(ico|css|js|gif|jpe?g|png)(\?[0-9]+)?$ {

      expires max;

      break;

    }

Homebrew + RVM > Awesome

I recently opened a box containing my new MacBook Pro. All of my files are synced/backed up with Dropbox, allowing me to install Dropbox on the new computer and leave it for a couple of hours. The MBP is my development machine, so I needed all of my tools installed with the ability to update them with ease. In the past, I used MacPorts to take care of my MySQL, Memcached, and Ruby installions and it worked just fine. This time around however, I wanted something new and fun. Homebrew.

Homebrew is a new package manager for OS X. Unlike Fink or MacPorts, Homebrew integrates with the core operating system, reducing the number of extra libraries to install etc. Another neat feature is the ability to write software package recipes in Ruby, awesome.

Here are some raw installation instructions (clean system). I like to keep everything under user ownership to make life more enjoyable, say no to sudo.

You will need the latest version of xcode, you can get it here. After the installation is complete, you may continue.

sudo mkdir /usr/local
sudo chown -R $USER /usr/local 

(one line)

curl -Lsf http://github.com/mxcl/homebrew/tarball/master | \

tar xvz -C/usr/local --strip 1

Homebrew is installed and "/usr/local" is already part of your path, now to give it a go.

brew install git

brew update

brew install wget
brew install mysql
brew install memcached
brew install postgres
brew install redis
brew install mongodb

brew list

My homebrew experience has been a happy one, I don't see myself using MacPorts or Fink again.

Homebrew documentation can be found HERE.

RVM (Ruby Version Manager) takes care of managing my multiple Ruby and Gem versions. Leaving the OS X default installation of ruby alone, RVM installs to a users home directory, changing the required system variables. Just how Homebrew is under user ownership, RVM is quite happy running this way as well and so am I. 

To install RVM, do the following.

mkdir -p ~/.rvm/src/ && cd ~/.rvm/src && rm -rf ./rvm/ && \

git clone --depth 1 git://github.com/wayneeseguin/rvm.git && cd rvm && ./install

Add the following line to the end of your .profile or .bash_profile or .bashrc file (be sure to edit your user name).

if [[ -s /Users/YOUR_USERNAME/.rvm/scripts/rvm ]] ; then source /Users/YOUR_USERNAME/.rvm/scripts/rvm ; fi

Try out RVM.

rvm install 1.9.1

rvm install 1.8.7
rvm --default 1.9.1

ruby -v

which ruby

gem install rails

Pretty neat eh? (I'm Canadian, it's ok)

RVM documentation can be found HERE.

Fix IO bottleneck by throwing disks at it! EC2 Style.

Story time

Bob recently setup a database server on EC2, using an EBS (Elastic Block Store) volume for its data files. Everything is rock solid, persistent, and performance is predictable (appears to be), Bob feels good, he's done his job (or so Bob thinks). 

On a Monday morning, Bob receives an urgent email from customer service stating that the site is crawling slow. He immediately jumps on his home computer and checks out his graphs (Munin, because that's how Bob rolls). The application servers are not being strained, however, he notices that the database query time has shot through the roof! Bob checks on his database server, seeing high disk latency and disgusting throughput he realizes something is up with the disk.

To be continued...

The performance of an Elastic Block Store is unpredictable, an excellent word to describe them is "moody" (maybe even bipolar). You can expect an EBS volume to give you upwards of 7000 seeks per second and down to a measly, depressing, 100.

To smooth out the flaky performance of a single EBS volume, you need more "moody" disks. Stripping data across a group of EBS volumes (RAID 0) eliminates the effects of a single slow disk (probably busy crunching on someone else's bits). Amazon claims that EBS volumes are extremely redundant, many people feel comfortable stripping upwards of 30 disks. I however, am still weary of using that many disks in RAID 0, I tend to use a max of 8.

Two striped EBS volumes operating at peak have the ability to fully saturate the IO channel, filling that Gigabit network pipe. Then why use more than two disks? Because EBS volumes rarely perform at their best, adding more disks keeps the performance bar high. 

Interesting facts:

• Larger chunk size on the raid device makes a big difference in performance
• XFS and JFS are the best file systems to use
• Mounting noatime gives you around a 5% boost
• Increasing the read ahead buffer for the RAID device provides an additional boost

Instructions for Bob (Ubuntu), using 6 disks:

Update apt.

sudo apt-get update

Install the dependencies.

sudo apt-get install mdadm xfsprogs bonnie

Attach 6 EBS volumes to the EC2 instance, be sure to attach them as "/dev/sdx[1-6]". You can use the ElasticFox Firefox plugin or the Amazon EC2 console to create and attach them.

Create the RAID device (single command).

sudo mdadm --create /dev/md0 --metadata=1.1 --level=0 \

--raid-devices=6 /dev/sdx1 /dev/sdx2 /dev/sdx3 /dev/sdx4 /dev/sdx5 /dev/sdx6

Format the RAID device (XFS).

sudo mkfs.xfs /dev/md0

Increase the read ahead buffer.

sudo blockdev --setra 65536 /dev/md0

Create the mounting point.

sudo mkdir /mnt/raid

Insert the mount into fstab.

sudo echo "/dev/md0 /mnt/raid xfs noatime 0 0" | sudo tee -a /etc/fstab

Mount the RAID volume.

sudo mount /mnt/raid

Open it up to the world.

sudo chmod -R 777 /mnt/raid/

Now test it!

bonnie -fd /mnt/raid

or (RAID over 20GB)

bonnie -d /mnt/raid/ -s 20000 -m your_hostname

Because we are dealing with a database server, a higher seeks per second is key. You can expect to get results ranging from 2K to around 11K. 

Benchmark results tend to be very inconsistent, you must remember that the disk are very "moody".

How you migrate your database data files to the array is up to you :)

How to stop and delete the RAID (If you don't like it):

sudo umount /mnt/raid

sudo mdadm --stop /dev/md0

sudo mdadm --remove /dev/md0

sudo mdadm --zero-superblock /dev/sdx1

sudo mdadm --zero-superblock /dev/sdx2

sudo mdadm --zero-superblock /dev/sdx3

sudo mdadm --zero-superblock /dev/sdx4

sudo mdadm --zero-superblock /dev/sdx5

sudo mdadm --zero-superblock /dev/sdx6

Excellent posts on EBS RAID and mdadm:

http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html

http://alestic.com/2009/06/ec2-ebs-raid

http://orion.heroku.com/past/2009/7/29/io_performance_on_ebs/

http://blog.endpoint.com/2010/02/postgresql-ec2-ebs-raid0-snapshot.html

http://www.ducea.com/2009/03/08/mdadm-cheat-sheet/