PorterTech

Sensu, a monitoring framework

2011-11-01T00:00:00+00:00

At Sonian, we monitor an ever changing number of Amazon EC2 instances. As I write this post, that number is 476, expected to rise and fall before the day is done. The "elastic" nature of our infrastructure makes monitoring it a not so trivial task.

We have found the standard tools from the community toolbox to be inadequate when operating in the "cloud". Up until recently, Sonian utilized several tools in conjunction to monitor systems and collect metrics; Nagios, Collectd, Graphite, and Ganglia.

The Evolution of Nagios at Sonian.

Our servers are grouped into "stacks", providing isolated environments that are globally distributed. In the past, a Nagios server would reside in each one. The servers were responsible for monitoring the components of their stack, triggering notifications when something was amiss. Check coverage was gradually increased over time, as applications began to require more moving parts. As the number of stacks increased, a centralized view of the organization was desired. To appease the engineering teams, a distributed Nagios solution was created. The monitoring server in each stack would forward their check results to a central Nagios server running the Nagios Service Check Accepter. The central server ran the Nagios web interface, displaying the status of every client and service under our control. Notifications could only be triggered by the central server, making it easier to silence notifications for a client or one of its services.

The Neanderthal.

Nagios is NOT designed for the cloud. It expects your environment to be fairly static, with every aspect of it under your control. The initial release was May 14th, 1999. The concept of elastic Infrastructure as a Service didn't exist at the time of its creation, and it has failed to adapt.

Nagios' inability to discover clients is an excellent indication of its antiquity. Nagios must know of every client, group, and service on start. When a new server is spun up, the Nagios configuration must be updated and the service reloaded in order to begin monitoring it. Configuration Management is commonly used in this case as a semi-solution; using a method of server discovery, re-writing the configuration, and then triggering a service reload. It's only a semi-solution as the process usually only happens on a set interval or is too intensive for frequent changes. Distributing Nagios in a tiered fashion only complicates this further, making it far more difficult to begin monitoring a new server or deploy new checks. The following diagram depicts a sample of events that would require Nagios configuration changes.

Our problems with Nagios

Configuration is unpleasant & restrictive
Cannot discover new servers on its own
Easily overwhelmed with a high number of clients & checks
Difficult to extend & hack

A Brief Introduction To Sensu.

Enter Sensu, a monitoring framework that aims to be simple, malleable, and scalable.

The Building Blocks.

In this modern world of computing, we're lucky to have ever improving Configuration Management tools, such as OpsCode Chef and Puppet. These tools already gather the information needed to effectively and efficiently monitor your systems. Not only are these tools a rich source of data, but they can also handle the distribution of supporting libraries and plugins. Sensu was built with the intention of being paired with a CM tool.

Message-oriented middleware is commonly used by developers to decouple and distribute components of their applications. RabbitMQ is a robust messaging system that has proven itself at Sonian. Sensu uses RabbitMQ to securely route check requests and results, making it possible to scale out and back in on demand.

Open source key-value data stores have been around for a long time, recently gaining a lot of attention with NoSQL being all the rage. Redis is a very fast in-memory "data structure server", with keys that can contain strings, hashes, lists, sets, and sorted sets. Its support for atomic operations and ability to persist to disk has made it a common choice for new projects. Sensu uses Redis as a non-persistent database, to store client and event data.

The Concept.

The idea behind Sensu is simple, schedule the remote execution of checks and collect their results. As mentioned above, Sensu uses RabbitMQ to route check requests and results, this is the secret sauce. Checks will always have an intended target; servers with certain responsibilities, such as serving web pages (webserver) or data storage (elasticsearch). A Sensu client has a set of subscriptions based on its server’s responsibilities, the client will execute checks that are published to these subscriptions. A Sensu server has a result subscription, this is where clients publish check results. Since each component only connects to RabbitMQ, there is no need for an external discovery mechanism, new servers are monitored immediately.

Code.

Sensu is written entirely in Ruby, using the EventMachine library for single process concurrency. This has produced a fully functional, clean, and small code base.

Sonian has made it publicly available on GitHub.

Language files blank comment code ------------------------------------------------------------------------------- Ruby 9 56 0 695 ------------------------------------------------------------------------------- SUM: 9 56 0 695

Configuration.

All configuration is done with JSON files, making it easy for Configuration Management and other automation tools to create and read them. The following are configuration snippets.

Client attributes

"client": { "name": "i-424242", "address": "127.0.0.1", "subscriptions": [ "all", "webserver" ] }

Check definition

"chef_client": { "notification": "Chef client daemon is not running.", "command": "/etc/sensu/plugins/chef_client.rb", "subscribers": [ "all" ], "interval": 60 }

I hope this very shallow dive into Sensu has spiked your interest. For the nitty gritty, please check out the GitHub repository, and jump on IRC (irc.freenode.net #sensu). I realize you probably have many questions, drop them into a comment bellow and I will do my best to fill the holes.

What’s Next?

My plan for the next few days is to produce documentation (wiki) and a Vagrantfile for provisioning a VM with either OpsCode Chef or Puppet.

Phones, servers, & user experience

2010-07-19T00:00:00+00:00

Successful mobile applications are enjoyable to use, offering a intuitive, useful, and responsive set of features.

GOOD mobile developers spend a large portion of their time designing their user interface, followed by performance optimization. GREAT developers know that creating a better experience for their users is far more important than adding features.

As the demand for data-rich mobile applications grows, producing a simple, easy to use interface becomes more challenging, usually accompanied by performance issues. Application optimization can only go so far, as developers are still limited by the device. Only so much processing power can be squeezed into a phone, keeping battery life in mind and not burning a hole in your pants. This is where web API's come into play, tons of horse power sitting up in the "cloud" with buckets of data. Web API's can offer access to a vast amount of data and the ability to offload computation, reducing the responsibilities of the mobile application. As users become thirsty for more data, with the expectations that it should be delivered quickly, web API performance should be of serious concern.

While I was at Gastown Labs, we built an iPhone application that required a decent amount of geo location data, plotted on a map while a user panned and zoomed. In the beginning we had to determine the best way to handle that data. As we saw it, there were two possible solutions: store the dataset locally and parse/process it all on the phone, or build a web API to handle the geospatial processing and serve the results to the mobile application.

Storing the entire dataset locally would provide the ability to use the application completely offline, without any network connectivity. Offline usage is great, however, parsing and processing causes the phone to become completely unresponsive and an update to the dataset would require an application store submission.

We chose to build a simple, RESTful web API (SOAP sucks) and we didn't look back. Compared to storing the dataset locally, it was night and day, even with limited connectivity the performance was outstanding. Users could pan and zoom on the map, loading location data from the web API, leaving them satisfied (you can get your data and use it too). A fantastic side effect of using a web API is flexibility, data updates and additions can happen without technical intervention, completely curated by the application owner.

The web stack:

Fantasy Cricket on the Rackspace Cloud

2010-05-28T00:00:00+00:00

I feel like sharing what we've been working on at Gastown Labs, so here is an outline of the infrastructure for Fantasy Cricket. Fantasy Cricket lives on the Rackspace Cloud (Cloud Servers), providing on-demand resources to handle traffic spikes and growth.

A user makes a request for fantasy.cricket.com, which resolves to a network load balancer. The load balancer determines which application server will serve the request, however, the load balancer also serves the static content for the request (style sheets, images, etc). The application server will churn away at the request, pulling data from cache or querying the database. The database master handles all of the application writes, where the two replicas serve the read requests. Live cricket scores are updated via XML feeds uploaded to the load balancer and processed with a background process running on one of the application servers.

Load Balancer

Nginx is used to handle web requests and act as a reverse proxy to the application servers using its HTTP proxy module. The load balancer has a complete copy of the application, including static content which is served directly from here. Nginx uses its Memcached module to cache specific page assets, reducing future page loading times. VSFTPD is used to allow specific live cricket score feed providers to update the application, XML files are uploaded via FTP to a specified folder for processing.

Application Servers

Each application server uses Nginx to send requests to a Unicorn worker pool over a Unix Domain Socket. Unicorn workers are responsible for executing the application. There is an excellent post on Unicorn in detail here, explaining on why it's so awesome. Background processes are distributed amongst the application servers, they are responsible for processing XML feeds and updating team scores etc. Each application server runs Memcached, each a part of the cache pool (volatile) used by the application and load balancer.

Database Servers

MySQL is our relational database of choice, Fantasy Cricket uses statement-based replication. All write operations are performed to the master, which are then replicated to the two slaves. Read operations are balanced between slaves, more slaves can be added to deal with an increase in read operations. Each database server runs MemcacheDB, not to be confused with Memcached. MemcacheDB is a distributed key-value storage system for persistent storage, used to cache assets that are too expensive to place in a volatile cache.

Homebrew + RVM > Awesome

2010-03-26T00:00:00+00:00

I recently opened a box containing my new MacBook Pro. All of my files are synced/backed up with Dropbox, allowing me to install Dropbox on the new computer and leave it for a couple of hours. The MBP is my development machine, so I needed all of my tools installed with the ability to update them with ease. In the past, I used MacPorts to take care of my MySQL, Memcached, and Ruby installions and it worked just fine. This time around however, I wanted something new and fun. Homebrew.

Homebrew is a new package manager for OS X. Unlike Fink or MacPorts, Homebrew integrates with the core operating system, reducing the number of extra libraries to install etc. Another neat feature is the ability to write software package recipes in Ruby, awesome.

Here are some raw installation instructions (clean system). I like to keep everything under user ownership to make life more enjoyable, say no to sudo.

You will need the latest version of xcode, you can get it here. After the installation is complete, you may continue.

sudo mkdir /usr/local sudo chown -R $USER /usr/local curl -Lsf http://github.com/mxcl/homebrew/tarball/master | tar xvz -C /usr/local --strip 1

Homebrew is installed and "/usr/local" is already part of your path, now to give it a go.

brew install git brew update brew install wget brew install mysql brew install memcached brew install postgres brew install redis brew install mongodb brew list

My homebrew experience has been a happy one, I don't see myself using MacPorts or Fink again.

Homebrew documentation can be found here.

RVM (Ruby Version Manager) takes care of managing my multiple Ruby and Gem versions. Leaving the OS X default installation of ruby alone, RVM installs to a users home directory, changing the required system variables. Just how Homebrew is under user ownership, RVM is quite happy running this way as well and so am I.

To install RVM, do the following.

mkdir -p ~/.rvm/src/ && cd ~/.rvm/src && rm -rf ./rvm/ && git clone --depth 1 git://github.com/wayneeseguin/rvm.git && cd rvm && ./install

Add the following line to the end of your .profile or .bash_profile or .bashrc file (be sure to edit your user name).

if [[ -s /Users/YOURUSERNAME/.rvm/scripts/rvm ]] ; then source /Users/YOURUSERNAME/.rvm/scripts/rvm ; fi

Try out RVM.

rvm install 1.8.7 rvm install 1.9.2 rvm --default 1.9.2 ruby -v which ruby gem install rails

Pretty neat eh? (I'm Canadian, it's ok)

RVM documentation can be found here.

Perfume.com on AWS

2010-02-28T00:00:00+00:00

"Perfume.com has grown into a leading online fragrance and beauty superstore, providing more than 12,000 brand name perfume, cologne, hair care and skin care products directly to consumers around the world at discount warehouse prices. Perfume.com is an Internet Retailer Top 500 E-Retailer."

Over the past three months, I have had the opportunity at Live Current Media to build a new platform for Perfume.com. The goal of the project was to reduce cost while increasing site performance and redundancy, insert cloud marketing pitch here. The new site infrastructure had to be able to scale up and most importantly down with the rise and fall of traffic. System failures had to been handled automatically and seamlessly, minimizing downtime as much as possible.

The goals had been set and a raw path laid out, the big decision had to be made... Which cloud provider should be used? Amazon Web Services. The decision had pretty much been decided before I joined the team. Live Current Media has deployed websites using the cloud in the past. Cricket.com, for example, runs on Rackspace Cloud Servers, but Amazon's broad range of services was too tempting.

I will begin with a brief layout and explanation of the infrastructure. This is just the first of many posts to come.

Amazon Web Services to be used:

EC2 (Elastic Compute Cloud) with Auto Scaling
S3 (Simple Storage Service)
ELB (Elastic Load Balancing)
CloudWatch

To help with the explanation, I created this very pretty diagram for you.

Starting at the top, a host makes a request to www.perfume.com, a cname (alias) to an ELB (Elastic Load Balancing) host name. Amazon's ELB service provides the ability to distribute incoming traffic across your EC2 instances in an availability zone (data center) or multiple if required. ELB can detect the health of EC2 instances, distributing traffic to only healthy application servers (More about ELB). Since redundancy is important, application servers are placed into two availability zones, having ELB check their health and send the request to an application server that is ready to process it.

An "Auto Scaling" group spans both availability zones, an Amazon service to start up and shut down EC2 instances when certain conditions are met (More about Auto Scaling). The application servers are part of this Auto Scaling group to provide the ability to scale up and down by adding or reducing the number of application servers (using CPU utilization as a metric). Newly created application instances are automatically added to ELB, having traffic distributed to them when they become healthy (ready to accept requests).

When an application server requires information from the database or shared cache, it queries using an "Elastic IP Address", a static IP address associated with an account and not a particular instance (it maps to an instance). The Elastic IP can be remapped fairly quickly, allowing for a quicker database/cache failover, reducing site downtime (More about Elastic IP Addresses). A database server resides in each of the availability zones, exact replicas in case of a server or complete availability zone failure. Database failover occurs automatically, using a Heartbeat over a VPN tunnel to detect failure and a script using Amazons API to remap the Elastic IP Address.

Static website content is served from S3, Amazon's Simple Storage Service (More about S3). Moving the static content onto S3 allows for the infrastructure to focus on the dynamic components, and allow for the use of Amazons CDN (Cache Delivery Network) CloudFront, for low latency and extremely fast data transfers (More about CloudFront).

To monitor the dynamic infrastructure, CloudWatch is used, Amazons monitoring tool for cloud resources. CloudWatch provides basic visibility into resource utilization (cpu, memory, network) and is required for Auto Scaling groups (More about CloudWatch).

I hope you found this quick post interesting, perhaps I will do a follow up in the near future.