It’s not too hard to set up fixtures to be inserted once per test suite run –

In your test_helper.rb, above the class ActiveSupport::TestCase definition, add the following:

Fixtures.reset_cache
fixtures_folder = File.join(RAILS_ROOT, 'test', 'fixtures')
fixtures = Dir[File.join(fixtures_folder, '*.yml')].map {|f| File.basename(f, '.yml') }
Fixtures.create_fixtures(fixtures_folder, fixtures)
Fixtures.reset_cache

Next, turn off transactional fixtures and comment out the fixtures macro:

self.use_transactional_fixtures = false
# fixtures :all

That’s all there is to it. Your fixtures will be inserted into your test database once when test_helper is included for the first time, and then not again for the rest of the test suite run. This should speed your tests up substantially.

Mobile authenticators work by transforming a seed value (called the “token secret”) + the current time into your 8-digit authentication code. FlexAuth lets you set up multiple authenticators by providing the secret, or will let you have Blizzard generate one for you.

Why would you need this?

• You want to use a mobile authenticator, but don’t want to be locked out if you ever lose your phone (just setup a new token with your registered token secret).
• You want to use multiple mechanisms to log in – maybe you need token authentication in a script, or you want to have the same authenticator values on multiple mobile phones.
• You already have a token secret from another source and want to use it on your mobile phone.

Obviously, these won’t apply to most people, but some folks will definitely find it useful.

1. Menu -> Add Account
2. Enter a name for this token/account. It can be whatever you’d like.
3. Either enter a serial + secret, or you can use the already-provided one, or generate a new one.
4. Save the token. You’ll notice that auth codes start generating right away.
5. It is highly recommended that you back up your token secret. If you uninstall the app, wipe your phone, etc, then you will lose the secret, and consequently lose the ability to generate auth codes. To back up a code, click into the token’s details, and long press on the secret to copy it. You can then paste it into a note or email or whatnot. To restore a token, simply generate a new token and use your backed up secret. It will generate compatible auth codes.

All that said, a word of caution: Never ever ever run authenticator software on the same machine that you’re logging in on. It’s bad, it’s dumb, and you shouldn’t do it. Keep your authentication token generation on a separate device if you value your account.

If any particular same-machine authentication scheme gained any measure of popularity, it would be targeted by malware and your authenticator would be useless. Don’t do it.

Other than that, enjoy!

So, I sat down and fixed gridfs-fuse. You can grab my branch at GitHub. I made a few changes that make it ideal for serving files out of a GridFS DB, with a few caveats.

Installation and Configuration

Building it is relatively simple.

1. Install scons, the Python SConstruct utility (on Fedora/CentOS/RHEL, yum install scons)
2. Extract or symlink a copy of your mongodb install to /opt/mongo
3. Run scons
4. If all builds well, yay. If not, fix any missing dependencies or path issues. Edit SConstruct to change any paths that you need to.
5. Create a mount point for your GridFS filesystem; I used /mnt/gridfs (sudo mkdir /mnt/gridfs)
6. chown your mount point to your webserver’s user. If you run Apache, this is probably apache. If you run nginx, it’s probably nobody. (sudo chown nobody.nobody /mnt/gridfs)
7. Mount the database to the mount point.

   sudo -u nobody ./mount_gridfs --db=your_database --host=localhost /mnt/gridfs
   

   Change the user and db parameters as required.

8. Configure your webserver to serve files appropriately. In my case, I have carrierwave set up to write files to uploads/model/_id/filename.png, and carrierwave is configured to use /images/gfs as my base URL. This means that for a given file, I might end up with a path like /images/gfs/uploads/user/avatar/4b8475cc69e0dc57e7000005/thumb_untitled-20.png. To cause the GridFS files to be served off of the mount point, I just symlinked the mount to /images/gfs.

   cd public/images
   ln -s /mnt/gridfs gfs
   

Once that’s all set up, you should be able to use your webserver to serve images directly out of your Mongo database, and at pretty fair rates, too!

143% Unscientific Benchmarks

[chris@polaris gridfs-fuse]# ab -n 5000 -c 25 http://advice:81/images/gfs/uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/gfs/uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png
Document Length:        14332 bytes

Concurrency Level:      25
Time taken for tests:   5.029 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      72725000 bytes
HTML transferred:       71660000 bytes
Requests per second:    994.22 [#/sec] (mean)
Time per request:       25.145 [ms] (mean)
Time per request:       1.006 [ms] (mean, across all concurrent requests)
Transfer rate:          14121.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    16   25   1.4     25      52
Waiting:        2   24   1.4     24      52
Total:         17   25   1.4     25      53

Percentage of the requests served within a certain time (ms)
  50%     25
  66%     25
  75%     25
  80%     25
  90%     25
  95%     26
  98%     27
  99%     32
 100%     53 (longest request)

Caveats

To get this working, I had to hack in directory support. GridFS stores files with paths, but doesn’t store them in a hierarchy; Fuse navigates a filesystem, which is hierarchical. In order to overcome this, I made gridfs-fuse respond to directory requests as valid. For a given file, gridfs-fuse will walk the following path hierarchy:

GET /uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png
Check for uploads, directory exists
Check for uploads/user, directory exists
Check for uploads/user/avatar/, directory exists
Check for uploads/avatar/4b8347a698db740b30000057, directory exists
Check for uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png, file exists, return file.

There are two things to be aware of here:

1. The deeper your path hierarchy, the more steps gridfs-fuse will take to find your file. Less directory nesting means faster file serving. The performance difference won’t be massive, but it’s there.
2. /!\ Big giant hack. /!\ gridfs-fuse assumes that any path part with a period in it is the path leaf. This is done so that we don’t have to keep querying the DB with regexes, which degrades performance by about 90% in my testing. Always, always, always make sure your filenames have a period in them, and make sure your directories do not have a period in them. This is a rather hefty set of caveats, but if you’ll stick to them, you will be rewarded with easy GridFS file serving.

What happens if I don’t follow those rules?

A few things happen. If you put periods in directory names, you’ll get 404s. They’ll be fast 404s, but they’ll be 404s. Even if a filepath is valid, like /images/foo.bar/baz/bin.png, gridfs-fuse will short-circuit at images/foo.bar, assuming that is the leaf of the hierarchy.

If you don’t put a period in your filenames, then gridfs-fuse will keep returning “yup, that’s a directory”, even when your webserver requests /images/foo.bar/baz/bin.png/index.html and then /images/foo.bar/baz/bin.png/index.html/index.html and then /images/foo.bar/baz/bin.png/index.html/index.html/index.html, and so forth. There’s a built-in stop at 10 levels deep – at 10 levels, gridfs-fuse gives up and just returns a 404, but it’ll take you a relatively long time to get there, and it’s really very highly recommended that you don’t do that.

What about when gridfs-fuse isn’t running?

Never fear, that’s easily fixed. Just use a Rack or Rails Metal middleware to serve images from GridFS. This is massively slower than serving files through gridfs-fuse, but at least your visitors won’t be treated to a site full of broken images if your mount point goes away for whatever reason. I’m using the following Metal endpoint. Just throw it into app/metals/gridfs.rb, add config.metals = ["Gridfs"] into your environment.rb, and you’re off to the races.

# rails metal to be used with carrierwave (gridfs) and MongoMapper

require 'mongo'
require 'mongo/gridfs'

# Allow the metal piece to run in isolation
require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails)

class Gridfs
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/images\/gfs\/(.+)$/
      key = $1
      if ::GridFS::GridStore.exist?(MongoMapper.database, key)
        ::GridFS::GridStore.open(MongoMapper.database, key, 'r') do |file|
          [200, {'Content-Type' => file.content_type}, [file.read]]
        end
      else
        [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
      end
    else
      [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
    end
  end
end

(I didn’t write that, but I can’t find the source to give credit at the moment).

That gives you a highly performant front-end solution with a reliable fallback. For any given request, the following should happen:

1. Your webserver attempts to load the file out of GridFS. If it can’t be found (likely due to a missing mountpoint), then…
2. The request will fall through to your Metal handler. It will then attempt to serve it from GridFS.
3. If it still can’t be found, the request falls through to your Rails app.

To prevent step 3 from happening, you might want to change line 18 of the Metal handler to return a 200 and read out a generic “missing image” image of some sort. That’ll prevent 404s from invoking a hit to your app.

Stick a CDN in front of it all, and you have a high-performance file upload solution with automatic replication and sharding that you can treat like any other piece of web data. Hooray!

There are lots of benchmarks and numbers under the cut. Keep reading!

Right now, there are three options:

1. Use a “low-level” script handler, like a Rack script or Rails Metal handler to serve them out of the database
2. Use something like gridfs-fuse to mount the database as a filesystem, and read it with the Fileserver directly
3. Use something like nginx-gridfs to talk directly to MongoDB from your webserver.

I wasn’t able to get gridfs-fuse to build on my system, but I was able to build the nginx module. The question, of course, is how fast are you going be serving files with each solution?

Filesystem read through Apache

First, I’ll establish a baseline. I’m running Apache as my frontend server, and we’ll use ab to benchmark its throughput.

[chris@polaris conf]# ab -n 50000 -c 10 http://advice/images/embed/alliance-60.png

Server Software:        Apache/2.2.13
Server Hostname:        advice
Server Port:            80

Document Path:          /images/embed/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   1.904 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      159463760 bytes
HTML transferred:       158043192 bytes
Requests per second:    2625.37 [#/sec] (mean)
Time per request:       3.809 [ms] (mean)
Time per request:       0.381 [ms] (mean, across all concurrent requests)
Transfer rate:          81767.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.4      1       4
Processing:     1    3   0.5      3       6
Waiting:        0    1   0.4      1       4
Total:          2    4   0.4      4       8

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      5
  99%      5
 100%      8 (longest request)

Nice and fast, like like we’d expect.

Filesystem read through nginx

[chris@polaris conf]# ab -n 50000 -c 10 http://advice:81/images/embed/normal_alliance-60.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/embed/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   7.623 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      1590513618 bytes
HTML transferred:       1579863192 bytes
Requests per second:    6559.31 [#/sec] (mean)
Time per request:       1.525 [ms] (mean)
Time per request:       0.152 [ms] (mean, across all concurrent requests)
Transfer rate:          203763.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       9
Processing:     1    1   0.4      1      11
Waiting:        0    0   0.1      0       9
Total:          1    1   0.5      1      12

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      2
  90%      2
  95%      2
  98%      3
  99%      3
 100%     12 (longest request)

nginx screams. At 6500 requests/sec, it’s blisteringly fast.

GridFS read through nginx-gridfs

[chris@polaris conf]# ab -n 5000 -c 10 http://advice:81/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   4.613 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      158580000 bytes
HTML transferred:       157980000 bytes
Requests per second:    1083.88 [#/sec] (mean)
Time per request:       9.226 [ms] (mean)
Time per request:       0.923 [ms] (mean, across all concurrent requests)
Transfer rate:          33570.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     1    9   4.7      9     103
Waiting:        1    9   4.7      9     102
Total:          2    9   4.7      9     103

Percentage of the requests served within a certain time (ms)
  50%      9
  66%      9
  75%      9
  80%      9
  90%      9
  95%      9
  98%      9
  99%     11
 100%    103 (longest request)

Definitely a lot slower, but still very respectable. 1051 requests/sec is going to be more than adequate for most purposes, particularly if fronted with a CDN.

And finally…

Rails Metal handler

The nice thing about the Rails metal handler solution is that it’s easy. No recompiling, just drop the handler into your project and you’re off to the races. That said…

[chris@polaris nginx-gridfs]$ ab -n 250 -c 4  http://advice/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png

Server Software:        Apache/2.2.13
Server Hostname:        advice
Server Port:            80

Document Path:          /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      4
Time taken for tests:   4.646 seconds
Complete requests:      250
Failed requests:        0
Write errors:           0
Total transferred:      7960000 bytes
HTML transferred:       7899000 bytes
Requests per second:    53.81 [#/sec] (mean)
Time per request:       74.338 [ms] (mean)
Time per request:       18.585 [ms] (mean, across all concurrent requests)
Transfer rate:          1673.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    15   74  75.6     34     287
Waiting:        0   72  75.8     30     276
Total:         15   74  75.6     34     288

Percentage of the requests served within a certain time (ms)
  50%     34
  66%     39
  75%    139
  80%    192
  90%    201
  95%    210
  98%    239
  99%    245
 100%    288 (longest request)

I obviously ran far fewer requests this go-round. The reason is pretty obvious – running 5000 requests through the Ruby stack would have taken approximately forever. At 53 requests per second, this is not an attractive solution, particularly if you consider the CPU overhead that it’s incurring.

Conclusions

If you’re looking to abstract away from storing files on a filesystem, GridFS is a feasable solution. It can really crank some mean output numbers, and though it’s not up to par with a raw filesystem read, also consider that in many production environments, such a raw filesystem read might be happening via an NFS or GFS share, which is going to massively degrade the performance of that request. Given the no-hassle store-and-forget-about-it solution that GridFS offers, even when faced with the challenge of multi-server replication, it seems that you can get enough performance out of it to justify it as a solution.

In short, I didn’t want to hack around with the MongoMapper associations code, so I just implemented my own little ride-along version.

module SecretProject
  module CounterCache
    module ClassMethods
      def counter_cache(field)
        class_eval <<-EOF
          after_create "increment_counter_for_#{field}"
          after_destroy "decrement_counter_for_#{field}"
        EOF
      end
    end

    module InstanceMethods
      def method_missing(method, *args)
        if matches = method.to_s.match(/^(in|de)crement_counter_for_(.*)$/) then
          dir = matches[1] == "in" ? 1 : -1
          parent_association = matches[2]
          if parent = self.send(parent_association) then
            name = "#{self.class.to_s.tableize}_count"
            if parent.respond_to?(name)
              parent.collection.update({:_id => parent._id}, {"$inc" => {name => dir}})
            end
          end
        else
          super
        end
      end
    end

    def self.included(receiver)
      receiver.extend         ClassMethods
      receiver.send :include, InstanceMethods
    end
  end
end

Throw that into your lib directory, load it with an initializer, and then you can use it something like so:

class Foo
  include MongoMapper::Document
  include SecretProject::CounterCache

  belongs_to :user
  counter_cache :user  # Will cause a foos_count field on the owning user to be maintained when a Foo is created or deleted.
end

This’ll only increment a counter if you’ve defined one on your parent object, via key :foos_count, Integer or similar, just so that it doesn’t go around updating every model you might associate it with.

Yay.

# Controller
caches_action :show, :unless => :user?, :expires_in => 24.hours

# Sweeper
expire_action :controller => "nodes", :action => "show", :id => record.to_param

This all works dandy, but I generate pretty URLs, which means sometimes there are characters in the URL that Memcached doesn’t like. A few minutes after deploying my patch, I started getting IMs from my logger bot telling me things were unhappy.

blippr. com: [#1265856785] ArgumentError: illegal character in key "views/m.blippr.com/apps/346562-PicFo g.mobile"
blippr. com: [#1265857710] ArgumentError: illegal character in key "views/www.blippr.com/apps/336714-µTorrent  "
blippr. com: [#1265857897] ArgumentError: illegal character in key "views/www.blippr.com/apps/337076-ustre am"
blippr. com: [#1265857924] ArgumentError: illegal character in key "views/www.blippr.com/apps/336714-µTorrent  "

That’s memcached complaining about the hash keys we’re giving to it. This just won’t do. We could just regex out “bad” characters, but that means potential collisions, and potentially leaves edge cases. Why not just hash it instead?

A quick monkey patch later:

class ActionController::Caching::Actions::ActionCachePath
	def path
		@cached_path ||= Digest::SHA1.hexdigest(@path)
	end
end

And we’re all dandy. Now, rather than caching by path, the path is hashed, and the hash is used as the path key. Since hashes will always be hexadecimal characters, we know that it’ll never make memcached unhappy.

Path is blippr.com/movies/6696-The-Silence-of-the-Lambs...
Cached fragment hit: views/9111cdefca4a52cb0e3a5ebac4f618127a30efd0 (1.1ms)

There is an argument for not using this technique if you’re using file-based caching, since it means your cached bits won’t be segregated into directories, but memcached doesn’t support expiry by regex anyhow, so there’s no good reason to not use it in this case.

Enjoy!

def display_name(user)
  "user.first_name #{user.last_name.slice(0,1)}"
end

Seems innocent enough, sure. Except…it doesn’t work in multibyte character sets. The first Cyrillic speaker to sign up blew that all up. When parsing an XML fragment with a name like this included, I was getting the following error:

ActionView::TemplateError: premature end of regular expression: /^\s*Елена\ �/

nokogiri (1.4.0) lib/nokogiri/xml/fragment_handler.rb:53:in `characters'

The issue, as it turned out, is that String#slice is a bytewise operation, not a character-wise operation like I’d so naively assumed. The issue is pretty easily to observe:

>> "Журинова".slice(0, 1)
=> "\320"

Fortunately, Rails has multibyte support baked in already, so it’s an easy mistake to correct:

def display_name(user)
  "user.first_name #{user.last_name.chars.first}"
end

And now…

>> "Журинова".chars.first
=> "Ж"

It’s very easy to make mistakes like this, and many times you may not even realize that they’re made unless you try to do something funny, like using it as a part of a regex. The safe operation is to never use String#slice or string subscripting on user data, but to instead treat all strings as multibyte strings. Very subtle, but the effects can be pretty nasty if you don’t.

After an extremely tedious series of debugger checks to make sure my OAuth signature was right, I decided to just edit the oauth gem on my production box and add a little debugging statement to dump the HTTP request to stdout. What I found was…surprising.

>> OAuthConsumers::Netflix.new.consumer.get_request_token
opening connection to api.netflix.com...
opened
<- "POST /oauth/request_token HTTP/1.1\r\nAccept: */*\r\nConnection: close\r\nUser-Agent: OAuth gem v0.3.6\r\nAuthorization: OAuth oauth_nonce=\"E73sq4XMkG547EbuCB9GUfG4AtsjD2QFySwLPKj0tI\", oauth_callback=\"oob\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1260049119\", oauth_consumer_key=\"xxxxxxxxxxxxx\", oauth_signature=\"QD5b5Oy8LFLvXWl%2B3R%2BQI0xlIcg%3D\", oauth_version=\"1.0\"\r\nContent-Length: 0\r\nHost: api.netflix.com\r\n\r\n"
-> "HTTP/1.1 401 Unauthorized\r\n"
-> "X-Lighty-Magnet-Uri-Path: /oauth/request_token\r\n"
-> "X-Mashery-Responder: proxyworker-i-e23bae8a.mashery.com\r\n"
-> "X-Mashery-Error-Code: ERR_401_TIMESTAMP_IS_INVALID\r\n"
-> "Content-Type: text/plain\r\n"
-> "Accept-Ranges: bytes\r\n"
-> "Content-Length: 20\r\n"
-> "Date: Sat, 05 Dec 2009 21:27:08 GMT\r\n"
-> "Server: Mashery Proxy\r\n"
-> "\r\n"
reading 20 bytes...
-> "Timestamp Is Invalid"
read 20 bytes
Conn close
OAuth::Unauthorized: 401 Unauthorized
        from /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/oauth-0.3.6/lib/oauth/consumer.rb:200:in `token_request'
        from /opt/ruby-enterprise-1.8.7-2009.10/lib/ruby/gems/1.8/gems/oauth-0.3.6/lib/oauth/consumer.rb:128:in `get_request_token'
        from (irb):1

Whoa there, there’s some info that the OAuth gem wasn’t giving back to me. “Timestamp is invalid.” Well then, a quick check of system time, and…oh, hey, it turns out that my system has drifted to about 10 minutes fast. Easily corrected, at least.

# ntpdate -b 0.centos.pool.ntp.org && service ntpd start

With that all done…

>> OAuthConsumers::Netflix.new.consumer.get_request_token
opening connection to api.netflix.com...
opened
<- "POST /oauth/request_token HTTP/1.1\r\nAccept: */*\r\nConnection: close\r\nUser-Agent: OAuth gem v0.3.6\r\nAuthorization: OAuth oauth_nonce=\"YIh5R3CBtAicneNREF5ZUcX80kao1zqRLLA5u8bQWA\", oauth_callback=\"oob\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1260048573\", oauth_consumer_key=\"ksfa9rxmb8dzkxg4npwr74zv\", oauth_signature=\"%2B%2Fyd5sRsJ7qmmZWNRqSlCvByYxw%3D\", oauth_version=\"1.0\"\r\nContent-Length: 0\r\nHost: api.netflix.com\r\n\r\n"
-> "HTTP/1.1 200 OK\r\n"
-> "X-Lighty-Magnet-Uri-Path: /oauth/request_token\r\n"
-> "X-Mashery-Responder: proxyworker-i-7c31a414.mashery.com\r\n"
-> "Content-Type: text/plain\r\n"
-> "Server: Mashery_Server_Adapter_Query\r\n"
-> "Date: Sat, 05 Dec 2009 21:29:32 GMT\r\n"
-> "Accept-Ranges: bytes\r\n"
-> "Content-Length: 194\r\n"
-> "\r\n"
reading 194 bytes...
-> "oauth_token=xxxxxxxx&oauth_token_secret=xxxxxxxxx&application_name=xxxxx&login_url=https%3A%2F%2Fapi-user.netflix.com%2Foauth%2Flogin%3Foauth_token%3Dczjsmzw74nk2wy274g6drmwt"
read 194 bytes
Conn close

All better. Keep those datetimes synched, sports fans. Web services are becoming more and more interconnected, and if there’s one thing I’ve learned from heist movies, it’s that the first step in any successful job is to make sure your watches are synchronized. Nobody likes that guy who shows up 10 minutes late to everything!

That sorta blows. Let’s not do it.

Introducing better_assets

better_assets is a monkeypatch to the Rails 2.3.2 AssetTagHelper to enable some additional functionality. The key points are:

* Time-based expiry of cached asset files, which is primarily useful for…
* Caching and combining of remote assets
* Finally, you can post-process combined assets with blocks passed to javascript_include_tag and stylesheet_link_tag.

Examples

It’s easy. You use it just like normal:

<%=javascript_include_tag(
  "jquery-1.3.2",
  "foo",
  :cache => "all") {|text| Packr.pack(text, :base62 => true) } %>

Whoa! What is this block madness? Why, that’s an extension to allow you to do whatever you want. In this example, we’re using jcoglan’s Packr library to automatically pack our generated Javascript. This can result in filesize being reduced by pretty massive amounts, and will result in appreciable performance benefits.

Well, that’s all fine and dandy, but it’s not my combined Javascript that’s killing me, it’s all those pesky DNS lookups for all my widget code and CSS. Never fear, you’re covered there, too.

<%=javascript_include_tag(
  "http://rpxnow.com/openid/v2/widget",
  "http://partner.googleadservices.com/gampad/google_service.js",
  "http://s3.amazonaws.com/getsatisfaction.com/feedback/feedback.js",
  "http://blippr.tags.crwdcntrl.net/cc.js",
  :cache => "remote", :lifetime => 12.hours) %>

Madness! Sheer madness! All those remote Javascript files are sucked down, combined, and cached as “remote.js”. It’ll automatically expire after 12 hours, and be re-cached after that. That way, you can get all the performance benefits of serving a single combined JS file without having to stress out that someone over at WidgetHeadquarters is going to change a piece of code and completely screw you over until you notice that your local Javascript file doesn’t match theirs six weeks later.

This, oddly enough, works for CSS files, too.

<%=stylesheet_link_tag(
  "http://s3.amazonaws.com/getsatisfaction.com/feedback/feedback.css",
  "http://s3.amazonaws.com/getsatisfaction.com/feedback/widget.css",
  :cache => "remote", :lifetime => 12.hours
) %>

No more stalling out at requests to Amazon’s S3 for CSS files! No more extraneous DNS requests or HTTP connections! No fuss, no muss, no headaches for you or you user.

All this, and it makes crispy bacon, too.*

To get it, just…wait for it. Very complex procedure ahead:

script/plugin install git://github.com/cheald/better_assets.git

Restart your app, and that’s it. Your assets are now approximately 163% more awesome, while being leaner and looking better in that fabulous summer swimsuit at the same time.

Score.

* Not really.

First, the boring documentation

All the juicy information is available in the documentation, but I’m going to just go over the key points real quick.

RUBY_HEAP_MIN_SLOTS: This is the number of “heap slots” that each Ruby instance starts up with. One heap slot can hold one Ruby object. By default, this is 10,000. By controlling this value, we can get our apps to stabilize very quickly. More on this later.

RUBY_HEAP_SLOTS_INCREMENT: Once Ruby has allocated RUBY_HEAP_MIN_SLOTS objects on its first heap, it will have to allocate a second heap to make room for more. This variable controls the size of this second heap, and sets the baseline for future heaps, as well.

RUBY_HEAP_SLOTS_GROWTH_FACTOR: For heaps #3 and onward, Ruby uses RUBY_HEAP_SLOTS_INCREMENT and this value to determine the size to allocate for the new heap. By default, this is 1.8, meaning that your third heap will end up with 10,000 * 1.8 = 18,000 slots in it.

RUBY_HEAP_FREE_MIN: After each garbage collection run, if the number of free slots is less than RUBY_HEAP_FREE_MIN, a new heap will be allocated. The default is 4096.

So, let’s look at this practically. Presume that we have a Rails process that is going to require 50,000 Ruby objects before it’s fully initialized. The allocation process, when at defaults, will look something like this:

Allocate 10,000 slots (10,000 total available)
Allocate 10,000 slots (20,000 total available)
Allocate 18,000 slots (38,000 total available)
Allocate 68,400 slots (106,400 total available)

So, we end up with about 53% more slots than we actually needed, and it took us four heap allocations to even boot the process. Surely we can do better.

Enter Scrap.

Scrap is a little Metal handler I wrote for tracking memory usage and garbage statistics over an instance’s lifetime. Installing it is trivial – just drop it into your vendor directory, restart your app, and navigate to http://yoururl.com/stats/scrap.

With this in hand, we can peek our memory usage and see what we can see.

There are some stats at the top, but for our purposes, we’re interested in the per-request garbage statistics. The newest request is near the top of the file, and the oldest request is at the bottom of the file. The last 50 requests are tracked. Each request looks something like this:

[71.92 MB] GET /apps/176568-WordPress

Number of objects    : 817571 (658305 AST nodes, 80.52%)
Heap slot size       : 20
GC cycles so far     : 503
Number of heaps      : 7
Total size of objects: 15968.18 KB
Total size of heaps  : 18036.81 KB (2068.63 KB = 11.47% unused)
Leading free slots   : 27104 (529.38 KB = 2.93%)
Trailing free slots  : 1 (0.02 KB = 0.00%)
Number of contiguous groups of 16 slots: 2829 (4.90%)
Number of terminal objects: 4307 (0.47%)

Key points here for the time being are Number of objects and Number of heaps. When we look at the number of objects – in this case, 817,000, it’s obvious that we’re going to have to allocate a number of heaps to handle all those objects. Rails’ boot-up cost is fairly significant, and the default Ruby settings just really don’t cut it here. As you can see, we’ve allocated 7 heaps, and we’re using 15.9 of 18.0 MB allocated to the heap. Once a heap is allocated, it’s never de-allocated, so we’re perma-stuck at 18 MB of heap usage. Note that this isn’t the size of all the data in the program – just the space allocated for objects. A string that contains 100MB of data will only consume 20 bytes (that’s the “heap slot size – the amount of memory each object on the heap consumes”) on the heap.

However, what if we could just allocate the whole startup cost in the initial heap, and save ourselves the problems of having to reallocate so often?

We note that we have 891k slots allocated, so we can guesstimate at a number to set our initial allocation to. In my production app, I set mine to 1,250,000 – I was observing peaks around the 1,100,000 mark, and just increased it by 10% and rounded up.

So, my first custom environment variable is

RUBY_HEAP_MIN_SLOTS=1250000

And it results in something like this on the app’s first boot:

[137.99 MB] GET /movies/7505-Star-Wars-Episode-V-The-Empire-Strikes-Back

Number of objects    : 933037 (664785 AST nodes, 71.25%)
Heap slot size       : 20
GC cycles so far     : 12
Number of heaps      : 1
Total size of objects: 18223.38 KB
Total size of heaps  : 24414.08 KB (6190.70 KB = 25.36% unused)
Leading free slots   : 316963 (6190.68 KB = 25.36%)
Trailing free slots  : 0 (0.00 KB = 0.00%)
Number of contiguous groups of 16 slots: 19810 (25.36%)
Number of terminal objects: 25941 (2.08%)

Yowza, a full 25% of my heap is unused after boot. But…well, that’s okay. We’ve only allocated 1 heap, and later on, my object allocation grows to around 1,100,000. This is still 15k under the heap size, and I’ve set RUBY_HEAP_FREE_MIN=12500 (1% of the initial size), so if I have less than 12,500 heap objects free after a GC cycle, a new heap will be allocated. Stabilizing there means that I end up with 1 heap for the lifetime of my app, and I end up sitting just under the threshold that’d cause a new heap to be born. If I have a leak, or a super heavy action or something, though, that might kick me over my limit and require a new heap. So, we come to…

RUBY_HEAP_SLOTS_INCREMENT=100000

This value says “Hey, if you have to allocate a second heap, start with this many slots”. If we go over our limit of 1.25 million slots, we’ll allocate a second heap that’s about 8% the size of the original. That seems awfully small, but consider that we’re hoping to never get to that heap.

Should we end up using that entire second heap, then we have to worry about our third setting, RUBY_HEAP_SLOTS_GROWTH_FACTOR=1. This says “Each new heap should be 1.0 as large as the previous heap.” In this case, it means I’ll keep allocating 100k-slot heaps until the cows come home. In an untuned environment, this could be bad – we would either end up having to do a ton of allocations to get to our target, or we would overallocate very badly. However, because we know our app’s memory requirements, and know about where we want it to end up, a relatively small, linear growth factor is just what the doctor ordered here.

Okay, now what?

So, we have a collection of settings with which to run our app. Great! Now, how do we use it?

Fortunately, it’s easy.

pushd `which ruby | xargs dirname`
sudo vim ruby-with-env

We’re going to create a little bash script with the following:

#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=1250000
export RUBY_HEAP_SLOTS_INCREMENT=100000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=30000000
export RUBY_HEAP_FREE_MIN=12500
exec "/opt/ree/bin/ruby" "$@"

Note that last line – the path will have to match the path to your Ruby executable, which fortunately, should be in the directory that you’re in.

Save it, don’t forget to chmod a+x ruby-with-env, and then edit your Apache or nginx configuration.

Under nginx, you’ll have a line like this:

passenger_ruby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby;

Just change it to use your new wrapper script, like so:

passenger_ruby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby-with-env;

The process is similarly easy for Apache – the line you need is something like:

PassengerRuby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby

It might be in either your httpd.conf or conf.d/passenger.conf.

Once you’re all edited up, restart your webserver, and congratulations, you’ve got a fine-tuned garbage collector humming along with your app.

Taking out the garbage

“But Chris!”, you say, “There’s a variable in there that you didn’t talk about! What gives?” You are indeed correct, astute reader. We’ve thus far avoided the RUBY_GC_MALLOC_LIMIT variable. This is a handle little setting that lets you tell Ruby how often to clean up after itself. Ruby is written in C, and C uses malloc to allocate memory. Ruby just keeps a little counter each time it allocates an object with malloc, and it runs its garbage collector after so many malloc calls have been made. I haven’t found a great way to tune this one yet, except via experimentation, but here’s what to know about it:

1. The lower this value is, the more often your garbage collector runs. Garbage collection is slow. Garbage collection is painfully slow. If a user is waiting on garbage collection, they are going to become impatient. You want as few users waiting on garbage collection as possible.
2. The higher this value is, the more memory Ruby will allocate before it tries to clean up after itself. If this value is too high, you’ll have dead objects hanging around eating up heap space, and possibly causing Ruby to crap itself and allocate a new heap. This is bad.
3. To tune this value, you want to find the happy medium, wherein you stabilize under your initial heap allocation value, but with as few garbage collection passes as possible. Read up on Evan Weaver’s blog for some more in-depth analysis of what garbage collection frequency tuning can do to your app’s performance.
4. If you have excess memory and want a faster app, err on the side of this being too high. If you are on a tight memory budget, and would prefer slower actions in exchange for not blowing your heap and allocating a whole new one, err on the side of this being too low.
5. Recommended values for this are all over the board. Evan recommends a setting of 50 million. I’m using a setting of 30 million. The Ruby default is 8 million. You’ll have to play around and find what works best for you. Just pay attention to how many requests there are in between that “GC cycles so far” number incrementing in Scrap, and you’ll be able to measure approximately how often you’re entering a GC cycle.

Good luck with it, and have fun!