So, I sat down and fixed gridfs-fuse. You can grab my branch at GitHub. I made a few changes that make it ideal for serving files out of a GridFS DB, with a few caveats.

Installation and Configuration

Building it is relatively simple.

1. Install scons, the Python SConstruct utility (on Fedora/CentOS/RHEL, yum install scons)
2. Extract or symlink a copy of your mongodb install to /opt/mongo
3. Run scons
4. If all builds well, yay. If not, fix any missing dependencies or path issues. Edit SConstruct to change any paths that you need to.
5. Create a mount point for your GridFS filesystem; I used /mnt/gridfs (sudo mkdir /mnt/gridfs)
6. chown your mount point to your webserver’s user. If you run Apache, this is probably apache. If you run nginx, it’s probably nobody. (sudo chown nobody.nobody /mnt/gridfs)
7. Mount the database to the mount point.

   sudo -u nobody ./mount_gridfs --db=your_database --host=localhost /mnt/gridfs
   

   Change the user and db parameters as required.

8. Configure your webserver to serve files appropriately. In my case, I have carrierwave set up to write files to uploads/model/_id/filename.png, and carrierwave is configured to use /images/gfs as my base URL. This means that for a given file, I might end up with a path like /images/gfs/uploads/user/avatar/4b8475cc69e0dc57e7000005/thumb_untitled-20.png. To cause the GridFS files to be served off of the mount point, I just symlinked the mount to /images/gfs.

   cd public/images
   ln -s /mnt/gridfs gfs
   

Once that’s all set up, you should be able to use your webserver to serve images directly out of your Mongo database, and at pretty fair rates, too!

143% Unscientific Benchmarks

[chris@polaris gridfs-fuse]# ab -n 5000 -c 25 http://advice:81/images/gfs/uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/gfs/uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png
Document Length:        14332 bytes

Concurrency Level:      25
Time taken for tests:   5.029 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      72725000 bytes
HTML transferred:       71660000 bytes
Requests per second:    994.22 [#/sec] (mean)
Time per request:       25.145 [ms] (mean)
Time per request:       1.006 [ms] (mean, across all concurrent requests)
Transfer rate:          14121.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    16   25   1.4     25      52
Waiting:        2   24   1.4     24      52
Total:         17   25   1.4     25      53

Percentage of the requests served within a certain time (ms)
  50%     25
  66%     25
  75%     25
  80%     25
  90%     25
  95%     26
  98%     27
  99%     32
 100%     53 (longest request)

Caveats

To get this working, I had to hack in directory support. GridFS stores files with paths, but doesn’t store them in a hierarchy; Fuse navigates a filesystem, which is hierarchical. In order to overcome this, I made gridfs-fuse respond to directory requests as valid. For a given file, gridfs-fuse will walk the following path hierarchy:

GET /uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png
Check for uploads, directory exists
Check for uploads/user, directory exists
Check for uploads/user/avatar/, directory exists
Check for uploads/avatar/4b8347a698db740b30000057, directory exists
Check for uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png, file exists, return file.

There are two things to be aware of here:

1. The deeper your path hierarchy, the more steps gridfs-fuse will take to find your file. Less directory nesting means faster file serving. The performance difference won’t be massive, but it’s there.
2. /!\ Big giant hack. /!\ gridfs-fuse assumes that any path part with a period in it is the path leaf. This is done so that we don’t have to keep querying the DB with regexes, which degrades performance by about 90% in my testing. Always, always, always make sure your filenames have a period in them, and make sure your directories do not have a period in them. This is a rather hefty set of caveats, but if you’ll stick to them, you will be rewarded with easy GridFS file serving.

What happens if I don’t follow those rules?

A few things happen. If you put periods in directory names, you’ll get 404s. They’ll be fast 404s, but they’ll be 404s. Even if a filepath is valid, like:

/images/foo.bar/baz/bin.png

gridfs-fuse will short-circuit at images/foo.bar, assuming that is the leaf of the hierarchy.

If you don’t put a period in your filenames, then gridfs-fuse will keep returning “yup, that’s a directory”, even when your webserver requests /images/foo.bar/baz/bin.png/index.html and then /images/foo.bar/baz/bin.png/index.html/index.html and then /images/foo.bar/baz/bin.png/index.html/index.html/index.html, and so forth. There’s a built-in stop at 10 levels deep – at 10 levels, gridfs-fuse gives up and just returns a 404, but it’ll take you a relatively long time to get there, and it’s really very highly recommended that you don’t do that.

What about when gridfs-fuse isn’t running?

Never fear, that’s easily fixed. Just use a Rack or Rails Metal middleware to serve images from GridFS. This is massively slower than serving files through gridfs-fuse, but at least your visitors won’t be treated to a site full of broken images if your mount point goes away for whatever reason. I’m using the following Metal endpoint. Just throw it into app/metals/gridfs.rb, add config.metals = ["Gridfs"] into your environment.rb, and you’re off to the races.

# rails metal to be used with carrierwave (gridfs) and MongoMapper

require 'mongo'
require 'mongo/gridfs'

# Allow the metal piece to run in isolation
require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails)

class Gridfs
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/images\/gfs\/(.+)$/
      key = $1
      if ::GridFS::GridStore.exist?(MongoMapper.database, key)
        ::GridFS::GridStore.open(MongoMapper.database, key, 'r') do |file|
          [200, {'Content-Type' => file.content_type}, [file.read]]
        end
      else
        [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
      end
    else
      [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
    end
  end
end

(I didn’t write that, but I can’t find the source to give credit at the moment).

That gives you a highly performant front-end solution with a reliable fallback. For any given request, the following should happen:

1. Your webserver attempts to load the file out of GridFS. If it can’t be found (likely due to a missing mountpoint), then…
2. The request will fall through to your Metal handler. It will then attempt to serve it from GridFS.
3. If it still can’t be found, the request falls through to your Rails app.

To prevent step 3 from happening, you might want to change line 18 of the Metal handler to return a 200 and read out a generic “missing image” image of some sort. That’ll prevent 404s from invoking a hit to your app.

Stick a CDN in front of it all, and you have a high-performance file upload solution with automatic replication and sharding that you can treat like any other piece of web data. Hooray!

There are lots of benchmarks and numbers under the cut. Keep reading!

Right now, there are three options:

1. Use a “low-level” script handler, like a Rack script or Rails Metal handler to serve them out of the database
2. Use something like gridfs-fuse to mount the database as a filesystem, and read it with the Fileserver directly
3. Use something like nginx-gridfs to talk directly to MongoDB from your webserver.

I wasn’t able to get gridfs-fuse to build on my system, but I was able to build the nginx module. The question, of course, is how fast are you going be serving files with each solution?

Filesystem read through Apache

First, I’ll establish a baseline. I’m running Apache as my frontend server, and we’ll use ab to benchmark its throughput.

[chris@polaris conf]# ab -n 50000 -c 10 http://advice/images/embed/alliance-60.png

Server Software:        Apache/2.2.13
Server Hostname:        advice
Server Port:            80

Document Path:          /images/embed/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   1.904 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      159463760 bytes
HTML transferred:       158043192 bytes
Requests per second:    2625.37 [#/sec] (mean)
Time per request:       3.809 [ms] (mean)
Time per request:       0.381 [ms] (mean, across all concurrent requests)
Transfer rate:          81767.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.4      1       4
Processing:     1    3   0.5      3       6
Waiting:        0    1   0.4      1       4
Total:          2    4   0.4      4       8

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      5
  99%      5
 100%      8 (longest request)

Nice and fast, like like we’d expect.

Filesystem read through nginx

[chris@polaris conf]# ab -n 50000 -c 10 http://advice:81/images/embed/normal_alliance-60.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/embed/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   7.623 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      1590513618 bytes
HTML transferred:       1579863192 bytes
Requests per second:    6559.31 [#/sec] (mean)
Time per request:       1.525 [ms] (mean)
Time per request:       0.152 [ms] (mean, across all concurrent requests)
Transfer rate:          203763.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       9
Processing:     1    1   0.4      1      11
Waiting:        0    0   0.1      0       9
Total:          1    1   0.5      1      12

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      2
  90%      2
  95%      2
  98%      3
  99%      3
 100%     12 (longest request)

nginx screams. At 6500 requests/sec, it’s blisteringly fast.

GridFS read through nginx-gridfs

[chris@polaris conf]# ab -n 5000 -c 10 http://advice:81/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   4.613 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      158580000 bytes
HTML transferred:       157980000 bytes
Requests per second:    1083.88 [#/sec] (mean)
Time per request:       9.226 [ms] (mean)
Time per request:       0.923 [ms] (mean, across all concurrent requests)
Transfer rate:          33570.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     1    9   4.7      9     103
Waiting:        1    9   4.7      9     102
Total:          2    9   4.7      9     103

Percentage of the requests served within a certain time (ms)
  50%      9
  66%      9
  75%      9
  80%      9
  90%      9
  95%      9
  98%      9
  99%     11
 100%    103 (longest request)

Definitely a lot slower, but still very respectable. 1051 requests/sec is going to be more than adequate for most purposes, particularly if fronted with a CDN.

And finally…

Rails Metal handler

The nice thing about the Rails metal handler solution is that it’s easy. No recompiling, just drop the handler into your project and you’re off to the races. That said…

[chris@polaris nginx-gridfs]$ ab -n 250 -c 4  http://advice/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png

Server Software:        Apache/2.2.13
Server Hostname:        advice
Server Port:            80

Document Path:          /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      4
Time taken for tests:   4.646 seconds
Complete requests:      250
Failed requests:        0
Write errors:           0
Total transferred:      7960000 bytes
HTML transferred:       7899000 bytes
Requests per second:    53.81 [#/sec] (mean)
Time per request:       74.338 [ms] (mean)
Time per request:       18.585 [ms] (mean, across all concurrent requests)
Transfer rate:          1673.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    15   74  75.6     34     287
Waiting:        0   72  75.8     30     276
Total:         15   74  75.6     34     288

Percentage of the requests served within a certain time (ms)
  50%     34
  66%     39
  75%    139
  80%    192
  90%    201
  95%    210
  98%    239
  99%    245
 100%    288 (longest request)

I obviously ran far fewer requests this go-round. The reason is pretty obvious – running 5000 requests through the Ruby stack would have taken approximately forever. At 53 requests per second, this is not an attractive solution, particularly if you consider the CPU overhead that it’s incurring.

Conclusions

If you’re looking to abstract away from storing files on a filesystem, GridFS is a feasable solution. It can really crank some mean output numbers, and though it’s not up to par with a raw filesystem read, also consider that in many production environments, such a raw filesystem read might be happening via an NFS or GFS share, which is going to massively degrade the performance of that request. Given the no-hassle store-and-forget-about-it solution that GridFS offers, even when faced with the challenge of multi-server replication, it seems that you can get enough performance out of it to justify it as a solution.

In short, I didn’t want to hack around with the MongoMapper associations code, so I just implemented my own little ride-along version.

module SecretProject
  module CounterCache
    module ClassMethods
      def counter_cache(field)
        class_eval <<-EOF
          after_create "increment_counter_for_#{field}"
          after_destroy "decrement_counter_for_#{field}"
        EOF
      end
    end

    module InstanceMethods
      def method_missing(method, *args)
        if matches = method.to_s.match(/^(in|de)crement_counter_for_(.*)$/) then
          dir = matches[1] == "in" ? 1 : -1
          parent_association = matches[2]
          if parent = self.send(parent_association) then
            name = "#{self.class.to_s.tableize}_count"
            if parent.respond_to?(name)
              parent.collection.update({:_id => parent._id}, {"$inc" => {name => dir}})
            end
          end
        else
          super
        end
      end
    end

    def self.included(receiver)
      receiver.extend         ClassMethods
      receiver.send :include, InstanceMethods
    end
  end
end

Throw that into your lib directory, load it with an initializer, and then you can use it something like so:

class Foo
  include MongoMapper::Document
  include SecretProject::CounterCache

  belongs_to :user
  counter_cache :user  # Will cause a foos_count field on the owning user to be maintained when a Foo is created or deleted.
end

This’ll only increment a counter if you’ve defined one on your parent object, via key :foos_count, Integer or similar, just so that it doesn’t go around updating every model you might associate it with.

Yay.

# Controller
caches_action :show, :unless => :user?, :expires_in => 24.hours

# Sweeper
expire_action :controller => "nodes", :action => "show", :id => record.to_param

This all works dandy, but I generate pretty URLs, which means sometimes there are characters in the URL that Memcached doesn’t like. A few minutes after deploying my patch, I started getting IMs from my logger bot telling me things were unhappy.

blippr. com: [#1265856785] ArgumentError: illegal character in key "views/m.blippr.com/apps/346562-PicFo g.mobile"
blippr. com: [#1265857710] ArgumentError: illegal character in key "views/www.blippr.com/apps/336714-µTorrent  "
blippr. com: [#1265857897] ArgumentError: illegal character in key "views/www.blippr.com/apps/337076-ustre am"
blippr. com: [#1265857924] ArgumentError: illegal character in key "views/www.blippr.com/apps/336714-µTorrent  "

That’s memcached complaining about the hash keys we’re giving to it. This just won’t do. We could just regex out “bad” characters, but that means potential collisions, and potentially leaves edge cases. Why not just hash it instead?

A quick monkey patch later:

class ActionController::Caching::Actions::ActionCachePath
	def path
		@cached_path ||= Digest::SHA1.hexdigest(@path)
	end
end

And we’re all dandy. Now, rather than caching by path, the path is hashed, and the hash is used as the path key. Since hashes will always be hexadecimal characters, we know that it’ll never make memcached unhappy.

Path is blippr.com/movies/6696-The-Silence-of-the-Lambs...
Cached fragment hit: views/9111cdefca4a52cb0e3a5ebac4f618127a30efd0 (1.1ms)

There is an argument for not using this technique if you’re using file-based caching, since it means your cached bits won’t be segregated into directories, but memcached doesn’t support expiry by regex anyhow, so there’s no good reason to not use it in this case.

Enjoy!

Configuration tweaks

There’s a lot of the Rails stack that’s written in Ruby, which is great – it’s portable, it’s flexible, it works out of the box. Unfortunately, for some things, this also means it’s slow. Other times, pieces of the framework aren’t implemented as optimally as they could be. What if you could improve your app’s performance just by installing a few gems and tweaking a few config parameters? Good news – it’s not hard.

1. Replace REXML with LibXML

By default, Rails uses a Ruby-native XML library called REXML. REXML is slow. REXML is very slow. REXML is personally responsible for me almost entirely giving up on Ruby due to a bad encounter with it in my first Ruby project. Fortunately, Rails provides a very easy way to avoid using REXML.

gem install libxml-ruby

Then, in your app’s config/environment.rb

ActiveSupport::XmlMini.backend = 'LibXML'

That’s it. Now, Rails will use the very lean, very fast libxml to parse XML documents, rather than the very fat, very slow REXML. If you’re doing feed parsing, Hash.from_xml, or anything of that nature, this will save you massive amounts of pain.

2. slim_attributes

If you’re using MySQL, there’s no reason why you shouldn’t be using slim_attributes.

Slim Attributes boosts speed in Mysql/Rails ActiveRecord Models by avoiding instantiating Hashes for each result row, and lazily instantiating attributes as needed.

Pretty self-explanatory. Rather than creating massive hashes of everything the DB gives you, slim_attributes causes ActiveRecord to only create ruby objects when you actually ask for them in code. This can reduce both your app’s memory usage and time spent on database queries. It’s not a massive increase, but given that it takes exactly one line of code to add to your project, there’s no reason not to use it.

3. slim_scrooge

From the developers of slim_attributes comes another drop-in database optimization.

SlimScrooge is an optimization layer to ensure your application only fetches the database content needed to minimize wire traffic, excessive SQL queries and reduce conversion overheads to native Ruby types.

SlimScrooge implements inline query optimisation, automatically restricting the columns fetched based on what was used during previous passes through the same part of your code.

Make your ORM work for you! By only fetching the content you need from your database, you reduce over-the-wire overhead, CPU overtime due to type conversion, and other such niceties. Again, just install the gem, require it in your project, and you’re off to the races.

4. fast_xs

By default, string escaping in Rails happens in native Ruby code. This is slow. We don’t like slow. This is particularly prominent in areas like Builder::XmlMarkup, which you are using if you have any templates like foo.xml.builder lying around.

In modestly-sized document, this can result in pretty substantial slowdown in view construction. Rather than re-hashing what others have already done, I’ll point you at Speed up your feed generation in Rails for the long and short on it all. This can result in builder views running upwards of 10x as fast, and all you have to do is install the fast_xs gem – Rails will automatically detect and patch it in if it’s on the system.

5. Erubis

Erubis is an ERB implementation written in C, rather than in Ruby. As a result, it parses ERB templates very, very quickly. In fact, the Erubis benchmarks up it at upwards of 3x faster than the native ERB implementation. Installation is easy – just check the using Erubis with Ruby on Rails guide and you’re off to the races.

Do note that if you’re entirely using Haml or similar, Erubis won’t do much for you. Erubis is much faster than Haml, but Haml is much prettier than ERB. What you end up using is up to you!

Reduce action runtimes

6. Use delayed_job

Sometimes in the course of any web service, you run into some action that takes a little while to process. This is generally a pain and causes a whole host of problems, including frustrated users clicking refresh and spawning a dozen instances of your app all running the same long-running request and tying up valuable request slots. Long-running jobs, or jobs that absolutely must succeed are something of a royal pain in the patootie to handle gracefully. Fortunately, there’s DelayedJob, which is much like a double shot of Codine to ease that terrible pain.

The concept is pretty simple – rather than immediately executing a long-running task, you create a “job” for it, then use an asynchronous daemon to run your job for you.

For example, let’s say that your app wants to post to Twitter when you accomplish some task. This is all well and good if Twitter is up (ha!) and fast and isn’t experiencing any technical issues and you aren’t having any issues on your end and you don’t have any exceptions. In short, it’s fine when things don’t break, but we all know that things break and go wrong and generally end up sideways when you’re ever dealing with any kind of I/O, particularly of the remote web service kind. Rather than trying to post to Twitter in-process, we’ll create a job whose task is to post to Twitter.

Install the delayed_job gem, create the delayed_jobs table as indicated in its documentation, and write your first worker.

module Jobs
	class PostToTwitter < Struct.new(:username, :password, :tweet)
		def perform
			auth = Twitter::HTTPAuth.new(username, password)
			client = Twitter::Base.new(auth)
			client.update(tweet)
		end
	end
end

Now, in your controller code, or after_create in your model, or where ever, rather than posting to Twitter directly, just enqueue a job:

Delayed::Job.enqueue Jobs::PostToTwitter.new(params[:username], params[:password], params[:tweet])

Finally, you’ll want to fire up a DelayedJob daemon. This is pretty easy to do under Rails.

Create a file called script/worker.rb and stick the following in it:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
dir = File.expand_path(File.join(File.dirname(__FILE__), '..'))

daemon_options = {
  :multiple =>; false,
  :dir_mode => :normal,
  :dir => File.join(dir, 'tmp', 'pids'),
  :backtrace => true
}

Daemons.run_proc('job_runner', daemon_options) do
  if ARGV.include?('--')
    ARGV.slice! 0..ARGV.index('--')
  else
    ARGV.clear
  end

  Dir.chdir dir
  RAILS_ENV = ARGV.first || ENV['RAILS_ENV'] || 'development'
  require File.join('config', 'environment')

  Delayed::Worker.new.start
end

Now, all you have to do is call script/worker start and you’re up and running. Jobs will automatically be processed as they’re added to the queue. If they fail, the reason why will be logged and the job will be scheduled to be retried in the future. You can correct any mistakes and re-run the job and watch it happily succeed. If the mistake is on the remote end, then the worker will keep retrying it until it succeeds, and your user doesn’t have to sit there and wait while your app continually receives the API equivalent of the failwhale. Everyone is happy (eventually!)

Once you start using DelayedJob, you’ll find that there are lots of things you can do with it to smooth out your app’s user-response speed. Processing user avatars or large file uploads, recomputing expensive queries (like a social graph update), talking to remote web services, or even sending emails can all be moved away from the realtime and into the background with total ease.

7. Use memcached

This should probably be tip #1. Good caching can make or break a project, and memcached is a fantastic method for managing your caching.

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

By default, Rails writes page and fragment cache bits to disk. This is slow, is difficult to clean up after, adds a lot of wear-and-tear to your disk, and is generally undesirable. It’s used because it’s easy. Memcached is a far better solution – it is very much a “giant hash table in the sky”. Dump a value into memory, read it back out of memory later. It is extremely fast, and comes with some super dandy features like time-based expiration that disk caching just won’t get you.

Implementation in Rails is easy. First, install both the memcached daemon and the memcache client. Second, in your environment file, add something like so:

require_library_or_gem 'memcache'
config.cache_store = :mem_cache_store, ["localhost:11211"]

By default, memcached runs on port 11211. Point Rails at it with the above directives and restart your app and that’s it. You’re running on memcached. No more ugly disk sweeping, and you get some really nice features. You can add multiple servers to the :mem_cache_store, too, which is several flavors of awesome. The memcached client will do automatic cluster management and balancing, so you can share the same cache between any number of servers, rather than each server having to have its own copy of that cache. Sweet!

<% cache("my_custom_fragment_name:#{@record_id}", :raw => true, :expires_in => 1.hour) do %>
	<%=render :partial => "some_expensive_partial", :object => @record %>
<% end %>

This is your standard fragment cache, but the :raw and :expires_in parameters are new.

:raw tells the Ruby memcached client to not marshal the content before sticking it in memcached. Since you’re just storing a document fragment (that is, a string), marshaling a ruby string and then unmarshaling it when you want to read it back is both unnecessary and slow.

:expires_in sets a maximum lifetime for this fragment. If we generate a fragment, memcached will timestamp it, and then if we try to read it back, say, 90 minutes later, memcached will recognize “oh hey, this fragment is expired! Sorry, I don’t have anything for you!”. Our view will regenerate and re-cache that fragment, and for the next 60 minutes, rather than trying to regenerate that fragment any time that view is called, it’ll just pull the cached copy from memcached.

If you need to ever flush your cache, it’s as easy as just restarting memcached. That’s it, really. In one fell swoop, you get faster caching (yay!), easier cache management (yay!), and a cache that can scale across multiple servers (double yay!)

8. Use etags

etags are a nifty little feature that are woefully under-used by most web developers. You can think of them as a fingerprint for a given page. Consider the following process:

1. I request a page for the first time. The app generates the page and sends me both a copy of the page and a small hash finger print.
2. I request the page a second time, and send the fingerprint of my cached copy back to the server.
3. The server compares the fingerprint I sent with the fingerprint of its latest copy of the page. If they match, it just sends back a 304 Not Modified header and stops rendering

Sounds handy, right? Sure, and it’s really easy to implement in Rails. Let’s assume you have a BlogController which has a show method for showing a given blog post. You could use the following to implement etags:

def show
	@post = BlogPost.find params[:id]
	@comments = @post.comments.paginate params[:page], 25
	return unless stale? :etag => [@post, @comments]
end

Wait, that’s it? Yes, actually! What’s happening there is Rails builds a fingerprint of the object(s) you the :etag parameter of the stale? method. If the objects don’t change, then the etag doesn’t change. This means that you would get different etags for the same blog post on a different page of comments (good!), or a different etag if a comment is added (good!) or a different etag if the post is edited (good!), but as long as those objects haven’t changed since the user’s last request of that action, the etag will be the same, and the action will stop running right there and tell the browser to just display its cached copy.

On heavily-trafficked pages that aren’t easily customized on a global scale (for example, if you have custom per-user bits on the page that mean that you can’t serve the same page to everyone), this is a really decent way to prevent excessive and wasteful application work. If you don’t use the stale? method, Rails always assumes that the page is stale, and thus needs to be regenerated.

On something of a tangent, can also use stale? :last_modified => @post.updated_at to determine if a page is fresh or stale. However, this does have the drawback of not being compatible with pagination, or sorted views, or anything of that nature. By using etags, you can ensure that each unique data set gets its own etag, and thus, doesn’t have cache collisions.

Every time this comes up, everyone says “How naive! They should have been using salted hashed passwords!” This is true in any case where you don’t need to use the password again on an external service. With OAuth solutions becoming more and more popular, the need to collect and store user passwords is fortunately becoming more and more rare. However, it does need to happen sometimes, so how do you take the proper precautions when you do need to?

The first step is to encrypt your data before it is persisted into your database. This is pretty easy to do, and there are a number of methods for it. Here’s an example of something I used in a Rails app to provide encryption services.

require 'openssl'
require 'base64'
module Encryption
	class OpenSSL_Key
		PUBLIC_KEY_FILE = "#{RAILS_ROOT}/config/public.pem"
		PRIVATE_KEY_FILE = "#{RAILS_ROOT}/config/private.pem"

		def self.encrypt(data)
			@@public_key ||= OpenSSL::PKey::RSA.new(File.read(PUBLIC_KEY_FILE))
			encrypted_data = @@public_key.public_encrypt(data)
			Base64.encode64(encrypted_data)
		end

		def self.decrypt(data)
			@@private_key ||= OpenSSL::PKey::RSA.new(File.read(PRIVATE_KEY_FILE))
			decoded_data = Base64.decode64(data)
			@@private_key.private_decrypt(decoded_data)
		end
	end

	class OpenSSL_RSA
		IV64 = "xxxxxxxxxxxxxxxxxxxxxxxxxx==\n"
		KEY64 = "xxxxxxxxxxxxxxxxxxxxxxxxxx=\n"
		CIPHER = 'aes-256-cbc'

		def self.encrypt(data)
			@@iv ||= Base64.decode64(IV64)
			@@key ||= Base64.decode64(KEY64)

			cipher = OpenSSL::Cipher::Cipher.new(CIPHER)
			cipher.encrypt
			cipher.key = @@key
			cipher.iv = @@iv
			encrypted_data = cipher.update(data)
			encrypted_data << cipher.final
			Base64.encode64(encrypted_data)
		end

		def self.decrypt(data)
			@@iv ||= Base64.decode64(IV64)
			@@key ||= Base64.decode64(KEY64)

			cipher = OpenSSL::Cipher::Cipher.new(CIPHER)
			cipher.decrypt
			cipher.key = @@key
			cipher.iv = @@iv
			decrypted_data = cipher.update(Base64.decode64(data))
			decrypted_data << cipher.final
		end
	end
end

This provides two classes, Encryption::OpenSSL_Key and Encryption::OpenSSL_RSA which may be used to encrypt arbitrary strings. The OpenSSL_Key class uses a public/private keypair (in our example, read out of the Rails config directory), and the OpenSSL_RSA class uses an initialization vector and secret key. The latter is probably easier, since it means you don’t have to worry about keypairs, and since all the encrypt/decrypt is done locally, there isn’t any need for public public encryption.

Once you have that file in your project, using it is pretty simple.

# Our databse is going to have a field called encrypted_password. We'll use attr_accessor for the password itself.

class MySecretModal < ActiveRecord::Base
	before_save :encrypt_fields
	attr_accessor :password

	def password
		@decrypted_password ||= decrypt_field(:password)
	end

private

	def encrypt_fields
		write_attribute :encrypted_password, Encryption::OpenSSL_RSA.encrypt(@password)
	end

	def decrypt_field(field)
		Encryption::OpenSSL_RSA.decrypt read_attribute("encrypted_#{field}")
	end
end

The net result is that we can still get access to the raw password if we need to, but the content in the database will be RSA-encrypted against a secret key in our application. This is still vulnerable if the attacker gains access to the file containing your RSA IV/key, or if he gains access to your public/private keypair, but it is extremely resilient in the case that an attacker manages to simply dump your users table via SQL injection. You still need to practice good key management, and you absolutely should not use a technique this simplistic for storing financial data – there are a whole set of guidelines and procedures for that kind of information. However, for adding an extra layer of defense to save yourself and your customers from excess embarrassment in the case of a database breach, this is a quick, easy, and effective technique for hardening your data.

This is a rather raw implementation, and there are ways you could package it up so that you could transparently apply it to any number of models or fields, but the basic technique is solid. You could even use something like sql_crypt to easily protect sensitive fields. The technology is there, and “We needed to be able to re-use the password!” isn’t an excuse anymore. Stop storing plaintext passwords – just like backups, it’s just extra work until you need it, and then you’ll be glad you put that extra work in.

def display_name(user)
  "user.first_name #{user.last_name.slice(0,1)}"
end

Seems innocent enough, sure. Except…it doesn’t work in multibyte character sets. The first Cyrillic speaker to sign up blew that all up. When parsing an XML fragment with a name like this included, I was getting the following error:

ActionView::TemplateError: premature end of regular expression: /^\s*Елена\ �/

nokogiri (1.4.0) lib/nokogiri/xml/fragment_handler.rb:53:in `characters'

The issue, as it turned out, is that String#slice is a bytewise operation, not a character-wise operation like I’d so naively assumed. The issue is pretty easily to observe:

>> "Журинова".slice(0, 1)
=> "\320"

Fortunately, Rails has multibyte support baked in already, so it’s an easy mistake to correct:

def display_name(user)
  "user.first_name #{user.last_name.chars.first}"
end

And now…

>> "Журинова".chars.first
=> "Ж"

It’s very easy to make mistakes like this, and many times you may not even realize that they’re made unless you try to do something funny, like using it as a part of a regex. The safe operation is to never use String#slice or string subscripting on user data, but to instead treat all strings as multibyte strings. Very subtle, but the effects can be pretty nasty if you don’t.

After an extremely tedious series of debugger checks to make sure my OAuth signature was right, I decided to just edit the oauth gem on my production box and add a little debugging statement to dump the HTTP request to stdout. What I found was…surprising.

>> OAuthConsumers::Netflix.new.consumer.get_request_token
opening connection to api.netflix.com...
opened
<- "POST /oauth/request_token HTTP/1.1\r\nAccept: */*\r\nConnection: close\r\nUser-Agent: OAuth gem v0.3.6\r\nAuthorization: OAuth oauth_nonce=\"E73sq4XMkG547EbuCB9GUfG4AtsjD2QFySwLPKj0tI\", oauth_callback=\"oob\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1260049119\", oauth_consumer_key=\"xxxxxxxxxxxxx\", oauth_signature=\"QD5b5Oy8LFLvXWl%2B3R%2BQI0xlIcg%3D\", oauth_version=\"1.0\"\r\nContent-Length: 0\r\nHost: api.netflix.com\r\n\r\n"
-> "HTTP/1.1 401 Unauthorized\r\n"
-> "X-Lighty-Magnet-Uri-Path: /oauth/request_token\r\n"
-> "X-Mashery-Responder: proxyworker-i-e23bae8a.mashery.com\r\n"
-> "X-Mashery-Error-Code: ERR_401_TIMESTAMP_IS_INVALID\r\n"
-> "Content-Type: text/plain\r\n"
-> "Accept-Ranges: bytes\r\n"
-> "Content-Length: 20\r\n"
-> "Date: Sat, 05 Dec 2009 21:27:08 GMT\r\n"
-> "Server: Mashery Proxy\r\n"
-> "\r\n"
reading 20 bytes...
-> "Timestamp Is Invalid"
read 20 bytes
Conn close
OAuth::Unauthorized: 401 Unauthorized
        from /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/oauth-0.3.6/lib/oauth/consumer.rb:200:in `token_request'
        from /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/oauth-0.3.6/lib/oauth/consumer.rb:128:in `get_request_token'
        from (irb):1

Whoa there, there’s some info that the OAuth gem wasn’t giving back to me. “Timestamp is invalid.” Well then, a quick check of system time, and…oh, hey, it turns out that my system has drifted to about 10 minutes fast. Easily corrected, at least.

# ntpdate -b 0.centos.pool.ntp.org && service ntpd start

With that all done…

>> OAuthConsumers::Netflix.new.consumer.get_request_token
opening connection to api.netflix.com...
opened
<- "POST /oauth/request_token HTTP/1.1\r\nAccept: */*\r\nConnection: close\r\nUser-Agent: OAuth gem v0.3.6\r\nAuthorization: OAuth oauth_nonce=\"YIh5R3CBtAicneNREF5ZUcX80kao1zqRLLA5u8bQWA\", oauth_callback=\"oob\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1260048573\", oauth_consumer_key=\"ksfa9rxmb8dzkxg4npwr74zv\", oauth_signature=\"%2B%2Fyd5sRsJ7qmmZWNRqSlCvByYxw%3D\", oauth_version=\"1.0\"\r\nContent-Length: 0\r\nHost: api.netflix.com\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "X-Lighty-Magnet-Uri-Path: /oauth/request_token\r\n"
-> "X-Mashery-Responder: proxyworker-i-7c31a414.mashery.com\r\n"
-> "Content-Type: text/plain\r\n"
-> "Server: Mashery_Server_Adapter_Query\r\n"
-> "Date: Sat, 05 Dec 2009 21:29:32 GMT\r\n"
-> "Accept-Ranges: bytes\r\n"
-> "Content-Length: 194\r\n"
-> "\r\n"
reading 194 bytes...
-> "oauth_token=xxxxxxxx&oauth_token_secret=xxxxxxxxx&application_name=xxxxx&login_url=https%3A%2F%2Fapi-user.netflix.com%2Foauth%2Flogin%3Foauth_token%3Dczjsmzw74nk2wy274g6drmwt"
read 194 bytes
Conn close

All better. Keep those datetimes synched, sports fans. Web services are becoming more and more interconnected, and if there’s one thing I’ve learned from heist movies, it’s that the first step in any successful job is to make sure your watches are synchronized. Nobody likes that guy who shows up 10 minutes late to everything!

That sorta blows. Let’s not do it.

Introducing better_assets

better_assets is a monkeypatch to the Rails 2.3.2 AssetTagHelper to enable some additional functionality. The key points are:

* Time-based expiry of cached asset files, which is primarily useful for…
* Caching and combining of remote assets
* Finally, you can post-process combined assets with blocks passed to javascript_include_tag and stylesheet_link_tag.

Examples

It’s easy. You use it just like normal:

<%=javascript_include_tag(
  "jquery-1.3.2",
  "foo",
  :cache => "all") {|text| Packr.pack(text, :base62 => true) } %>

Whoa! What is this block madness? Why, that’s an extension to allow you to do whatever you want. In this example, we’re using jcoglan’s Packr library to automatically pack our generated Javascript. This can result in filesize being reduced by pretty massive amounts, and will result in appreciable performance benefits.

Well, that’s all fine and dandy, but it’s not my combined Javascript that’s killing me, it’s all those pesky DNS lookups for all my widget code and CSS. Never fear, you’re covered there, too.

<%=javascript_include_tag(
  "http://rpxnow.com/openid/v2/widget",
  "http://partner.googleadservices.com/gampad/google_service.js",
  "http://s3.amazonaws.com/getsatisfaction.com/feedback/feedback.js",
  "http://blippr.tags.crwdcntrl.net/cc.js",
  :cache => "remote", :lifetime => 12.hours) %>

Madness! Sheer madness! All those remote Javascript files are sucked down, combined, and cached as “remote.js”. It’ll automatically expire after 12 hours, and be re-cached after that. That way, you can get all the performance benefits of serving a single combined JS file without having to stress out that someone over at WidgetHeadquarters is going to change a piece of code and completely screw you over until you notice that your local Javascript file doesn’t match theirs six weeks later.

This, oddly enough, works for CSS files, too.

<%=stylesheet_link_tag(
  "http://s3.amazonaws.com/getsatisfaction.com/feedback/feedback.css",
  "http://s3.amazonaws.com/getsatisfaction.com/feedback/widget.css",
  :cache => "remote", :lifetime => 12.hours
) %>

No more stalling out at requests to Amazon’s S3 for CSS files! No more extraneous DNS requests or HTTP connections! No fuss, no muss, no headaches for you or you user.

All this, and it makes crispy bacon, too.*

To get it, just…wait for it. Very complex procedure ahead:

script/plugin install git://github.com/cheald/better_assets.git

Restart your app, and that’s it. Your assets are now approximately 163% more awesome, while being leaner and looking better in that fabulous summer swimsuit at the same time.

Score.

* Not really.

First, the boring documentation

All the juicy information is available in the documentation, but I’m going to just go over the key points real quick.

RUBY_HEAP_MIN_SLOTS: This is the number of “heap slots” that each Ruby instance starts up with. One heap slot can hold one Ruby object. By default, this is 10,000. By controlling this value, we can get our apps to stabilize very quickly. More on this later.

RUBY_HEAP_SLOTS_INCREMENT: Once Ruby has allocated RUBY_HEAP_MIN_SLOTS objects on its first heap, it will have to allocate a second heap to make room for more. This variable controls the size of this second heap, and sets the baseline for future heaps, as well.

RUBY_HEAP_SLOTS_GROWTH_FACTOR: For heaps #3 and onward, Ruby uses RUBY_HEAP_SLOTS_INCREMENT and this value to determine the size to allocate for the new heap. By default, this is 1.8, meaning that your third heap will end up with 10,000 * 1.8 = 18,000 slots in it.

RUBY_HEAP_FREE_MIN: After each garbage collection run, if the number of free slots is less than RUBY_HEAP_FREE_MIN, a new heap will be allocated. The default is 4096.

So, let’s look at this practically. Presume that we have a Rails process that is going to require 50,000 Ruby objects before it’s fully initialized. The allocation process, when at defaults, will look something like this:

Allocate 10,000 slots (10,000 total available)
Allocate 10,000 slots (20,000 total available)
Allocate 18,000 slots (38,000 total available)
Allocate 68,400 slots (106,400 total available)

So, we end up with about 53% more slots than we actually needed, and it took us four heap allocations to even boot the process. Surely we can do better.

Enter Scrap.

Scrap is a little Metal handler I wrote for tracking memory usage and garbage statistics over an instance’s lifetime. Installing it is trivial – just drop it into your vendor directory, restart your app, and navigate to http://yoururl.com/stats/scrap.

With this in hand, we can peek our memory usage and see what we can see.

There are some stats at the top, but for our purposes, we’re interested in the per-request garbage statistics. The newest request is near the top of the file, and the oldest request is at the bottom of the file. The last 50 requests are tracked. Each request looks something like this:

[71.92 MB] GET /apps/176568-WordPress

Number of objects    : 817571 (658305 AST nodes, 80.52%)
Heap slot size       : 20
GC cycles so far     : 503
Number of heaps      : 7
Total size of objects: 15968.18 KB
Total size of heaps  : 18036.81 KB (2068.63 KB = 11.47% unused)
Leading free slots   : 27104 (529.38 KB = 2.93%)
Trailing free slots  : 1 (0.02 KB = 0.00%)
Number of contiguous groups of 16 slots: 2829 (4.90%)
Number of terminal objects: 4307 (0.47%)

Key points here for the time being are Number of objects and Number of heaps. When we look at the number of objects – in this case, 817,000, it’s obvious that we’re going to have to allocate a number of heaps to handle all those objects. Rails’ boot-up cost is fairly significant, and the default Ruby settings just really don’t cut it here. As you can see, we’ve allocated 7 heaps, and we’re using 15.9 of 18.0 MB allocated to the heap. Once a heap is allocated, it’s never de-allocated, so we’re perma-stuck at 18 MB of heap usage. Note that this isn’t the size of all the data in the program – just the space allocated for objects. A string that contains 100MB of data will only consume 20 bytes (that’s the “heap slot size – the amount of memory each object on the heap consumes”) on the heap.

However, what if we could just allocate the whole startup cost in the initial heap, and save ourselves the problems of having to reallocate so often?

We note that we have 891k slots allocated, so we can guesstimate at a number to set our initial allocation to. In my production app, I set mine to 1,250,000 – I was observing peaks around the 1,100,000 mark, and just increased it by 10% and rounded up.

So, my first custom environment variable is

RUBY_HEAP_MIN_SLOTS=1250000

And it results in something like this on the app’s first boot:

[137.99 MB] GET /movies/7505-Star-Wars-Episode-V-The-Empire-Strikes-Back

Number of objects    : 933037 (664785 AST nodes, 71.25%)
Heap slot size       : 20
GC cycles so far     : 12
Number of heaps      : 1
Total size of objects: 18223.38 KB
Total size of heaps  : 24414.08 KB (6190.70 KB = 25.36% unused)
Leading free slots   : 316963 (6190.68 KB = 25.36%)
Trailing free slots  : 0 (0.00 KB = 0.00%)
Number of contiguous groups of 16 slots: 19810 (25.36%)
Number of terminal objects: 25941 (2.08%)

Yowza, a full 25% of my heap is unused after boot. But…well, that’s okay. We’ve only allocated 1 heap, and later on, my object allocation grows to around 1,100,000. This is still 15k under the heap size, and I’ve set RUBY_HEAP_FREE_MIN=12500 (1% of the initial size), so if I have less than 12,500 heap objects free after a GC cycle, a new heap will be allocated. Stabilizing there means that I end up with 1 heap for the lifetime of my app, and I end up sitting just under the threshold that’d cause a new heap to be born. If I have a leak, or a super heavy action or something, though, that might kick me over my limit and require a new heap. So, we come to…

RUBY_HEAP_SLOTS_INCREMENT=100000

This value says “Hey, if you have to allocate a second heap, start with this many slots”. If we go over our limit of 1.25 million slots, we’ll allocate a second heap that’s about 8% the size of the original. That seems awfully small, but consider that we’re hoping to never get to that heap.

Should we end up using that entire second heap, then we have to worry about our third setting, RUBY_HEAP_SLOTS_GROWTH_FACTOR=1. This says “Each new heap should be 1.0 as large as the previous heap.” In this case, it means I’ll keep allocating 100k-slot heaps until the cows come home. In an untuned environment, this could be bad – we would either end up having to do a ton of allocations to get to our target, or we would overallocate very badly. However, because we know our app’s memory requirements, and know about where we want it to end up, a relatively small, linear growth factor is just what the doctor ordered here.

Okay, now what?

So, we have a collection of settings with which to run our app. Great! Now, how do we use it?

Fortunately, it’s easy.

pushd `which ruby | xargs dirname`
sudo vim ruby-with-env

We’re going to create a little bash script with the following:

#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=1250000
export RUBY_HEAP_SLOTS_INCREMENT=100000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=30000000
export RUBY_HEAP_FREE_MIN=12500
exec "/opt/ree/bin/ruby" "$@"

Note that last line – the path will have to match the path to your Ruby executable, which fortunately, should be in the directory that you’re in.

Save it, don’t forget to chmod a+x ruby-with-env, and then edit your Apache or nginx configuration.

Under nginx, you’ll have a line like this:

passenger_ruby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby;

Just change it to use your new wrapper script, like so:

passenger_ruby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby-with-env;

The process is similarly easy for Apache – the line you need is something like:

PassengerRuby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby

It might be in either your httpd.conf or conf.d/passenger.conf.

Once you’re all edited up, restart your webserver, and congratulations, you’ve got a fine-tuned garbage collector humming along with your app.

Taking out the garbage

“But Chris!”, you say, “There’s a variable in there that you didn’t talk about! What gives?” You are indeed correct, astute reader. We’ve thus far avoided the RUBY_GC_MALLOC_LIMIT variable. This is a handle little setting that lets you tell Ruby how often to clean up after itself. Ruby is written in C, and C uses malloc to allocate memory. Ruby just keeps a little counter each time it allocates an object with malloc, and it runs its garbage collector after so many malloc calls have been made. I haven’t found a great way to tune this one yet, except via experimentation, but here’s what to know about it:

1. The lower this value is, the more often your garbage collector runs. Garbage collection is slow. Garbage collection is painfully slow. If a user is waiting on garbage collection, they are going to become impatient. You want as few users waiting on garbage collection as possible.
2. The higher this value is, the more memory Ruby will allocate before it tries to clean up after itself. If this value is too high, you’ll have dead objects hanging around eating up heap space, and possibly causing Ruby to crap itself and allocate a new heap. This is bad.
3. To tune this value, you want to find the happy medium, wherein you stabilize under your initial heap allocation value, but with as few garbage collection passes as possible. Read up on Evan Weaver’s blog for some more in-depth analysis of what garbage collection frequency tuning can do to your app’s performance.
4. If you have excess memory and want a faster app, err on the side of this being too high. If you are on a tight memory budget, and would prefer slower actions in exchange for not blowing your heap and allocating a whole new one, err on the side of this being too low.
5. Recommended values for this are all over the board. Evan recommends a setting of 50 million. I’m using a setting of 30 million. The Ruby default is 8 million. You’ll have to play around and find what works best for you. Just pay attention to how many requests there are in between that “GC cycles so far” number incrementing in Scrap, and you’ll be able to measure approximately how often you’re entering a GC cycle.

Good luck with it, and have fun!