1. Set up object creation tracking on the Set class
2. Find the leaky instance of our set using GDB
3. Locate what code created that Set instance

First things first. Check out Sean Bradly’s GCTrack module. I’m using a modified version of it to include object IDs in the tracker struct, so that we can perform reverse-lookups on an instance.

# Sean Bradly (rhythmx at gmail) - 2010
# Modified by Chris Heald (cheald@gmail.com) - August 2010
module GCTrack
  # Class to track an object's lifecycle and where it was created
  class Tracepoint < Struct.new(:num, :call, :insp, :id)
    include Comparable
    # make it so the obj.num works as a simple key
    def <=>(obj)
      if obj.class <= self.class
        self.num <=> obj.num
      else
        self.num <=> obj
      end
    end
  end

  def self.included(klass)

    # Hook initialize to track this object
    klass.class_eval do
      alias_method :old_init, :initialize

      def initialize(*args)
        # Call original init
        old_init(*args)
        # trace creation and setup finalizer callback
        final_proc = self.class.gct_setup_finalizer(self, caller, inspect)
        # define the finalizer
        ObjectSpace.define_finalizer(self, final_proc)
      end
    end

    # Add some metaclass foo to track all the objects.
    # You cant track in an instance w/o creating circular
    # references and breaking Ruby's GC

    (
    class << klass;
      self;
    end).class_eval do
      def gct_setup_finalizer(obj, caller, insp)
        # create/log a tracepoint for this object
        num = gct_objnum
        tpoint = Tracepoint.new(num, caller, insp, obj.object_id)
        gct_add(tpoint)
        # call option init cb
        if self.respond_to?(:created)
          self.send(:created, tpoint)
        end
        obj = nil # don't let the proc track a ref to 'obj'
        # this callback happens at GC time
        final_proc = Proc.new do
          gct_del(tpoint)
          # call optional final cb
          if self.respond_to?(:deleted)
            self.send(:deleted, tpoint)
          end
        end
      end

      def gct_objnum
        @num ||= -1
        @num  +=  1
      end

      def gct_addlog
        @addlog ||= []
      end

      def gct_dellog
        @dellog ||= []
      end

      def gct_add(num)
        gct_addlog << num
      end

      def gct_del(num)
        gct_dellog << num
      end

      def gct_active
        gct_addlog - gct_dellog
      end

      def gct_origin(id)
        x = gct_addlog.select {|g| g.id == id }.first
        x.nil? ? nil : x.call.join("\n")
      end

      def gct_orphan_report
        # Get the leaked objects
        rpt     = ""
        leaked  = gct_active
        callers = leaked.map { |o| o.call }.sort.uniq
        # Iterate over source lines that have leaked objs
        callers.each do |calla|
          leaked_here = leaked.find_all { |o| o.call == calla }
          rpt << "==== #{leaked_here.size} leaked objects from:\n\n"
          calla.each { |l| rpt << "    #{l}\n" }
          rpt << "\n"
          rpt << "    == object data:\n\n"

          leaked_here.each do |o|
            rpt << "    num: #{o.num}, inspect: #{o.insp}\n"
          end
          rpt << "\n"
        end
        rpt
      end

    end
  end
end

Additionally, you’ll want to patch the object you’re interested in tracking.

class Set
  include GCTrack
end

You’ll want to include both of those at the top of your config.rb, before your initializer block. If you’re tracking an object defined in your Rails app, just include GCTrack in the object definition, rather than in a monkeypatch.

Next, you need to be sure that you have your GDB macros set up properly. /root/.gdbinit should have the following:

define eval
  call(rb_p(rb_eval_string_protect($arg0,(int*)0)))
end

define redirect_stdout
  call rb_eval_string("$_old_stdout, $stdout = $stdout, File.open('/tmp/ruby-debug.' + Process.pid.to_s, 'a'); $stdout.sync = true")
end

(re)start your application, hit the leaky action a few times, and find its PID using ps ax or top. Once you have that, attach to the process with gdb, redirect your process’s stdout to the tmp file with redirect_stdout and dump all Sets in your object space.

(gdb) attach 12345
(gdb) redirect_stdout
$8 = 2
(gdb) eval "GC.start"
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" if o.is_a?(Set) }; puts '----'"

Now that we have that, let’s look at the tmp file, and locate the leaked set instance.

Set, #<Set: {Achievement, Achievement, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment}> -- 78376680

There, at the end, is our object ID. Now back over to gdb.

(gdb) eval "puts Set.gct_addlog.select {|g| g.id == 78376680}.first.call.join(\"\n\")"

Your tmp file should now have a stack trace for that object’s allocation now. Track it down and beat it into submission.

/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/mongo_mapper-0.8.3/lib/mongo_mapper/support/descendant_appends.rb:15:in `new'

You can do this for just about any object you’d like. Just mix in your GCTrack and easily find where in code a particular instance of an object was created. Just like that, debugging memory leaks goes from hunting guppies in the Atlantic to shooting fish in a barrel.

Have fun!

I set out to track down the cause tonight, and have both fixed the problem and gotten some handy experience debugging Rails apps with gdb.

The Solution

First off, if you just want the fix, here it is. You probably have a middleware to clear your identity maps already. We’re just going to modify that. In my case, it’s lib/mongo_mapper/per_request_identity_map_clear.rb.

module MongoMapper
  class PerRequestMapClear
    def initialize(app)
      @app = app
    end

    def call(env)
      if Rails.configuration.cache_classes
        MongoMapper::Plugins::IdentityMap.clear
      else
        MongoMapper::Document.descendants.each do |m|
          m.descendants.clear if m.respond_to? :descendants
        end
        MongoMapper::Document.descendants.clear
        MongoMapper::EmbeddedDocument.descendants.clear
        MongoMapper::Plugins::IdentityMap.clear
        MongoMapper::Plugins::IdentityMap.models.clear
      end
      @app.call(env)
    end
  end
end

In particular, these two lines:

  MongoMapper::Document.descendants.clear
  MongoMapper::Plugins::IdentityMap.models.clear

Make sure you get that into your middleware stack and all your MongoMapper memory leaking issues will magically disappear.

The Problem

You see, MongoMapper is doing some dag-nasty evil stuff with class references. Namely, it’s holding onto them in class variables that don’t get reloaded per-request in Rails. Simply:

1. Rails loads your User model. This loads up a ton of Validations, MongoMapper::Keys, procs, and whatever else you have in your model definition. It’s not really light. That’s okay because we load it only once in production.
2. This is a MongoMapper::Document, which causes User (the class reference) to be inserted into the MongoMapper::Document.descendants class attribute, which happens to be a Set object.
3. Next time you refresh a page, Rails re-loads your User model. This creates a new and separate User class reference. It does not replace the previous User class reference. It is not equal to your previous User class reference. They are, as far as Ruby cares, separate objects.
4. MongoMapper happily sticks that class reference into its descendants class attribute. You now have two separate copies of User. Since MongoMapper is holding onto a reference to your old User class, Ruby can never garbage collect it. The old User class and all of its huge cascading chain of referenced objects are now “leaked”.
5. Your memory usage is increasing by 4mb per request now.

I had app instances reaching nearly 1GB of RAM usage after light testing. I finally noticed it when my development machine kicked into swap and actions that took 80ms to run were taking 8000ms to run. Hm. That might be a problem!

Debugging leaks with GDB

GDB is an amazing tool. With a few macros, you can make hacking around in a live Ruby instance pretty painless. Crack open your /root/.gdbinit file and add a few macros:

define eval
  call(rb_p(rb_eval_string_protect($arg0,(int*)0)))
end

define redirect_stdout
  call rb_eval_string("$_old_stdout, $stdout = $stdout, File.open('/tmp/ruby-debug.' + Process.pid.to_s, 'a'); $stdout.sync = true")
end

Now, we’re going to attach to your running Ruby process. This needs to be done as root.

[root@polaris ~]# gdb
(gdb) attach 12019
(gdb) redirect_stdout
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" unless o.is_a?(String) }; puts '----'"

This will effectively dump all non-String objects in the attached Ruby process to tmp/ruby-debug.12019. This takes a little bit, but it lets you come up with some handy data for parsing later.

To get data we can compare, we’ll need to dump the environment for multiple requests:

[root@polaris ~]# gdb
(gdb) attach 12019
(gdb) redirect_stdout
(gdb) eval "GC.start"
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" unless o.is_a?(String) }; puts '----'"
(gdb) detach

(run some requests)

(gdb) attach 12019
(gdb) redirect_stdout
(gdb) eval "GC.start"
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" unless o.is_a?(String) }; puts '----'"
(gdb) detach

At this point, you’ll have two ObjectSpace dumps in your temp file. For my purposes, I hacked up a quick little script to parse those dumps, and to output all objects that were not present in both dumps. Since I’m invoking GC.start, in theory, this should help me find my leaked objects.

runs = [[]]
open(ARGV[0]).each do |line|
  if line == "----\n" then
    runs << []
  elsif line.match "--" then
    runs.last << line.strip
  end
end

diff = []
runs.each do |run|
  diff = (diff - run) | (run - diff)
end
diff.sort.map {|d| puts d }

Not very pretty, but it does the job. Just a quick invocation to ruby find_leaked.rb /tmp/ruby-debug.12019 > leaked (well, not that quick, it took a minute to run) and I effectively had an ObjectSpace diff I could pore through.

There’s a lot of stuff in there. In particular, you’re going to notice that you have a LOT of Array, Hash, and MatchData objects (perhaps potential optimization points for future Rails releases?). While we may be interested in those, try to cull out the things that obviously aren’t a problem just for readability’s sake.

I pored through the diff looking for things related to MongoMapper or my models. After not too long, I came across these lines:

Set, #<Set: {Achievement, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment, Achievement, Achievement, EmbeddedComment, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment, Achievement, Achievement, Achievement, EmbeddedComment, EmbeddedComment, Achievement}> -- 92034680
Set, #<Set: {Achievement, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment, Achievement, Achievement, EmbeddedComment, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment, Achievement, Achievement, EmbeddedComment, Achievement, Achievement, EmbeddedComment, EmbeddedComment, Achievement}> -- 92034680

Whoa there. What? Why do I have Sets with multiple references to Achievement and EmbeddedComment? That doesn’t smell right. I suspect the problem lies in MongoMapper, so let’s grep the MongoMapper codebase for Set.

[chris@polaris lib]$ grep Set * -R
mongo_mapper/plugins/identity_map.rb:        @models ||= Set.new
mongo_mapper/plugins/modifiers.rb:          modifier_update('$addToSet', args)
mongo_mapper/plugins/protected.rb:          self.write_inheritable_attribute(:attr_protected, Set.new(attrs) + (protected_attributes || []))
mongo_mapper/plugins/accessible.rb:          self.write_inheritable_attribute(:attr_accessible, Set.new(attrs) + (accessible_attributes || []))
mongo_mapper/support/descendant_appends.rb:        @descendants ||= Set.new
mongo_mapper/connection.rb:      raise 'Set config before connecting. MongoMapper.config = {...}' unless defined?(@@config)
mongo_mapper/connection.rb:      raise 'Set config before connecting. MongoMapper.config = {...}' if config.blank?
mongo_mapper/extensions/set.rb:    module Set
mongo_mapper/extensions/set.rb:class Set
mongo_mapper/extensions/set.rb:  extend MongoMapper::Extensions::Set

Great, we have a hit list to look through. IdentityMap is worth looking at; a new set is created there, and its naming indicates that it’s for holding models, probably model references (it is). mongo_mapper/support/descendant_appends.rb is much the same deal. We can ignore mongo_mapper/plugins/accessible.rb since we can guess that the attrs being passed are symbols, rather than those class references we saw in the ObjectSpace diff.

Let’s crack open descendant_appends.rb

module MongoMapper
  module Support
    module DescendantAppends
      def included(model)
        extra_extensions.each { |extension| model.extend(extension) }
        extra_inclusions.each { |inclusion| model.send(:include, inclusion) }
        descendants << model
      end

      # @api public
      def descendants
        @descendants ||= Set.new
      end

Oh dear. And there it is. Every time MongoMapper::Support::DescendantAppends is included in a model (which is via MongoMapper::Document and MongoMapper:EmbeddedDocument), a reference to the including class is stored in a class variable.

Since we know that Rails reloads models per-request in development mode, and we know that each copy of a model’s class is not considered equivalent to the other copies of that class, it’s easy to see what happens here: We end up with sets like in our object dump, with however many orphaned old copies of our models, and all their various and sundry associated models.

And so, we arrive at our solution. By clearing the descendants set on every request where we are reloading our models, we ensure that there are not references to old copies of models left hanging around leaking memory.

My development instances are now running solid at 68mb apiece, rather than 1gb apiece. As you can imagine, the difference in response speed (and thus, productivity) is substantial.

Hope this helps. Have fun with gdb – it’s an obscenely powerful tool, and used properly, can give you a purely nutty amount of information which can be invaluable in tracking down memory leaks and related problems.

I recently discovered some very cool functionality it includes – the WillPaginate::Collection class can be used as a custom paginator for effectively any enumerable collection. It’s very simple, too. I recently used it to build pages of the most popular tags on posts in my database. My data store is MongoDB, and I’m fetching an array consisting of two-element arrays, [tag, tag_count]. To use will_paginate’s functionality with this, I just use the following:

tags = Post.tag_counts(nil, {:sort => ["value", "descending"]}) # Return an array of tag/count pairs. Custom function, so it can't leverage the finder on Post.
@topics = WillPaginate::Collection.create(current_page, 20, tags.length) do |pager|
	pager.replace(tags.slice(pager.offset, pager.offset + pager.per_page))
end

current_page is a helper that derives the current page from the request parameters. The rest of it is self-explanitory. I can now use @topics in my page just as I’d use a paginated result set from the database.

- @topics.each do |topic|
    # ...
=will_paginate @topics

Bam. Doesn’t get much easier than that. You can get exceptionally creative with it, too. Effectively, all you need to know is:

• WillPaginate::Collection#new takes 3 parameters: the current page, the per-page count, and optionally, the total number of entries.
• The pager block variable exposes offset and per_page properties, prime for passing into a DB query or slicing an enumerable with
• Call pager.replace(sub-array) with the current page’s set of elements.

That’s literally all there is to it. Now you can have easy pagination on just about any collection you can conceive of. Let WillPaginate handle all the heavy lifting and such. If you’ve done enough pagination by hand, you’ll probably appreciate the easy beauty of this particular method.

Configuration tweaks

There’s a lot of the Rails stack that’s written in Ruby, which is great – it’s portable, it’s flexible, it works out of the box. Unfortunately, for some things, this also means it’s slow. Other times, pieces of the framework aren’t implemented as optimally as they could be. What if you could improve your app’s performance just by installing a few gems and tweaking a few config parameters? Good news – it’s not hard.

1. Replace REXML with LibXML

By default, Rails uses a Ruby-native XML library called REXML. REXML is slow. REXML is very slow. REXML is personally responsible for me almost entirely giving up on Ruby due to a bad encounter with it in my first Ruby project. Fortunately, Rails provides a very easy way to avoid using REXML.

gem install libxml-ruby

Then, in your app’s config/environment.rb

ActiveSupport::XmlMini.backend = 'LibXML'

That’s it. Now, Rails will use the very lean, very fast libxml to parse XML documents, rather than the very fat, very slow REXML. If you’re doing feed parsing, Hash.from_xml, or anything of that nature, this will save you massive amounts of pain.

2. slim_attributes

If you’re using MySQL, there’s no reason why you shouldn’t be using slim_attributes.

Slim Attributes boosts speed in Mysql/Rails ActiveRecord Models by avoiding instantiating Hashes for each result row, and lazily instantiating attributes as needed.

Pretty self-explanatory. Rather than creating massive hashes of everything the DB gives you, slim_attributes causes ActiveRecord to only create ruby objects when you actually ask for them in code. This can reduce both your app’s memory usage and time spent on database queries. It’s not a massive increase, but given that it takes exactly one line of code to add to your project, there’s no reason not to use it.

3. slim_scrooge

From the developers of slim_attributes comes another drop-in database optimization.

SlimScrooge is an optimization layer to ensure your application only fetches the database content needed to minimize wire traffic, excessive SQL queries and reduce conversion overheads to native Ruby types.

SlimScrooge implements inline query optimisation, automatically restricting the columns fetched based on what was used during previous passes through the same part of your code.

Make your ORM work for you! By only fetching the content you need from your database, you reduce over-the-wire overhead, CPU overtime due to type conversion, and other such niceties. Again, just install the gem, require it in your project, and you’re off to the races.

4. fast_xs

By default, string escaping in Rails happens in native Ruby code. This is slow. We don’t like slow. This is particularly prominent in areas like Builder::XmlMarkup, which you are using if you have any templates like foo.xml.builder lying around.

In modestly-sized document, this can result in pretty substantial slowdown in view construction. Rather than re-hashing what others have already done, I’ll point you at Speed up your feed generation in Rails for the long and short on it all. This can result in builder views running upwards of 10x as fast, and all you have to do is install the fast_xs gem – Rails will automatically detect and patch it in if it’s on the system.

5. Erubis

Erubis is an ERB implementation written in C, rather than in Ruby. As a result, it parses ERB templates very, very quickly. In fact, the Erubis benchmarks up it at upwards of 3x faster than the native ERB implementation. Installation is easy – just check the using Erubis with Ruby on Rails guide and you’re off to the races.

Do note that if you’re entirely using Haml or similar, Erubis won’t do much for you. Erubis is much faster than Haml, but Haml is much prettier than ERB. What you end up using is up to you!

Reduce action runtimes

6. Use delayed_job

Sometimes in the course of any web service, you run into some action that takes a little while to process. This is generally a pain and causes a whole host of problems, including frustrated users clicking refresh and spawning a dozen instances of your app all running the same long-running request and tying up valuable request slots. Long-running jobs, or jobs that absolutely must succeed are something of a royal pain in the patootie to handle gracefully. Fortunately, there’s DelayedJob, which is much like a double shot of Codine to ease that terrible pain.

The concept is pretty simple – rather than immediately executing a long-running task, you create a “job” for it, then use an asynchronous daemon to run your job for you.

For example, let’s say that your app wants to post to Twitter when you accomplish some task. This is all well and good if Twitter is up (ha!) and fast and isn’t experiencing any technical issues and you aren’t having any issues on your end and you don’t have any exceptions. In short, it’s fine when things don’t break, but we all know that things break and go wrong and generally end up sideways when you’re ever dealing with any kind of I/O, particularly of the remote web service kind. Rather than trying to post to Twitter in-process, we’ll create a job whose task is to post to Twitter.

Install the delayed_job gem, create the delayed_jobs table as indicated in its documentation, and write your first worker.

module Jobs
	class PostToTwitter < Struct.new(:username, :password, :tweet)
		def perform
			auth = Twitter::HTTPAuth.new(username, password)
			client = Twitter::Base.new(auth)
			client.update(tweet)
		end
	end
end

Now, in your controller code, or after_create in your model, or where ever, rather than posting to Twitter directly, just enqueue a job:

Delayed::Job.enqueue Jobs::PostToTwitter.new(params[:username], params[:password], params[:tweet])

Finally, you’ll want to fire up a DelayedJob daemon. This is pretty easy to do under Rails.

Create a file called script/worker.rb and stick the following in it:

#!/usr/bin/env ruby
require 'rubygems'
require 'daemons'
dir = File.expand_path(File.join(File.dirname(__FILE__), '..'))

daemon_options = {
  :multiple =>; false,
  :dir_mode => :normal,
  :dir => File.join(dir, 'tmp', 'pids'),
  :backtrace => true
}

Daemons.run_proc('job_runner', daemon_options) do
  if ARGV.include?('--')
    ARGV.slice! 0..ARGV.index('--')
  else
    ARGV.clear
  end

  Dir.chdir dir
  RAILS_ENV = ARGV.first || ENV['RAILS_ENV'] || 'development'
  require File.join('config', 'environment')

  Delayed::Worker.new.start
end

Now, all you have to do is call script/worker start and you’re up and running. Jobs will automatically be processed as they’re added to the queue. If they fail, the reason why will be logged and the job will be scheduled to be retried in the future. You can correct any mistakes and re-run the job and watch it happily succeed. If the mistake is on the remote end, then the worker will keep retrying it until it succeeds, and your user doesn’t have to sit there and wait while your app continually receives the API equivalent of the failwhale. Everyone is happy (eventually!)

Once you start using DelayedJob, you’ll find that there are lots of things you can do with it to smooth out your app’s user-response speed. Processing user avatars or large file uploads, recomputing expensive queries (like a social graph update), talking to remote web services, or even sending emails can all be moved away from the realtime and into the background with total ease.

7. Use memcached

This should probably be tip #1. Good caching can make or break a project, and memcached is a fantastic method for managing your caching.

Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.

By default, Rails writes page and fragment cache bits to disk. This is slow, is difficult to clean up after, adds a lot of wear-and-tear to your disk, and is generally undesirable. It’s used because it’s easy. Memcached is a far better solution – it is very much a “giant hash table in the sky”. Dump a value into memory, read it back out of memory later. It is extremely fast, and comes with some super dandy features like time-based expiration that disk caching just won’t get you.

Implementation in Rails is easy. First, install both the memcached daemon and the memcache client. Second, in your environment file, add something like so:

require_library_or_gem 'memcache'
config.cache_store = :mem_cache_store, ["localhost:11211"]

By default, memcached runs on port 11211. Point Rails at it with the above directives and restart your app and that’s it. You’re running on memcached. No more ugly disk sweeping, and you get some really nice features. You can add multiple servers to the :mem_cache_store, too, which is several flavors of awesome. The memcached client will do automatic cluster management and balancing, so you can share the same cache between any number of servers, rather than each server having to have its own copy of that cache. Sweet!

<% cache("my_custom_fragment_name:#{@record_id}", :raw => true, :expires_in => 1.hour) do %>
	<%=render :partial => "some_expensive_partial", :object => @record %>
<% end %>

This is your standard fragment cache, but the :raw and :expires_in parameters are new.

:raw tells the Ruby memcached client to not marshal the content before sticking it in memcached. Since you’re just storing a document fragment (that is, a string), marshaling a ruby string and then unmarshaling it when you want to read it back is both unnecessary and slow.

:expires_in sets a maximum lifetime for this fragment. If we generate a fragment, memcached will timestamp it, and then if we try to read it back, say, 90 minutes later, memcached will recognize “oh hey, this fragment is expired! Sorry, I don’t have anything for you!”. Our view will regenerate and re-cache that fragment, and for the next 60 minutes, rather than trying to regenerate that fragment any time that view is called, it’ll just pull the cached copy from memcached.

If you need to ever flush your cache, it’s as easy as just restarting memcached. That’s it, really. In one fell swoop, you get faster caching (yay!), easier cache management (yay!), and a cache that can scale across multiple servers (double yay!)

8. Use etags

etags are a nifty little feature that are woefully under-used by most web developers. You can think of them as a fingerprint for a given page. Consider the following process:

1. I request a page for the first time. The app generates the page and sends me both a copy of the page and a small hash finger print.
2. I request the page a second time, and send the fingerprint of my cached copy back to the server.
3. The server compares the fingerprint I sent with the fingerprint of its latest copy of the page. If they match, it just sends back a 304 Not Modified header and stops rendering

Sounds handy, right? Sure, and it’s really easy to implement in Rails. Let’s assume you have a BlogController which has a show method for showing a given blog post. You could use the following to implement etags:

def show
	@post = BlogPost.find params[:id]
	@comments = @post.comments.paginate params[:page], 25
	return unless stale? :etag => [@post, @comments]
end

Wait, that’s it? Yes, actually! What’s happening there is Rails builds a fingerprint of the object(s) you the :etag parameter of the stale? method. If the objects don’t change, then the etag doesn’t change. This means that you would get different etags for the same blog post on a different page of comments (good!), or a different etag if a comment is added (good!) or a different etag if the post is edited (good!), but as long as those objects haven’t changed since the user’s last request of that action, the etag will be the same, and the action will stop running right there and tell the browser to just display its cached copy.

On heavily-trafficked pages that aren’t easily customized on a global scale (for example, if you have custom per-user bits on the page that mean that you can’t serve the same page to everyone), this is a really decent way to prevent excessive and wasteful application work. If you don’t use the stale? method, Rails always assumes that the page is stale, and thus needs to be regenerated.

On something of a tangent, can also use stale? :last_modified => @post.updated_at to determine if a page is fresh or stale. However, this does have the drawback of not being compatible with pagination, or sorted views, or anything of that nature. By using etags, you can ensure that each unique data set gets its own etag, and thus, doesn’t have cache collisions.

Every time this comes up, everyone says “How naive! They should have been using salted hashed passwords!” This is true in any case where you don’t need to use the password again on an external service. With OAuth solutions becoming more and more popular, the need to collect and store user passwords is fortunately becoming more and more rare. However, it does need to happen sometimes, so how do you take the proper precautions when you do need to?

The first step is to encrypt your data before it is persisted into your database. This is pretty easy to do, and there are a number of methods for it. Here’s an example of something I used in a Rails app to provide encryption services.

require 'openssl'
require 'base64'
module Encryption
	class OpenSSL_Key
		PUBLIC_KEY_FILE = "#{RAILS_ROOT}/config/public.pem"
		PRIVATE_KEY_FILE = "#{RAILS_ROOT}/config/private.pem"

		def self.encrypt(data)
			@@public_key ||= OpenSSL::PKey::RSA.new(File.read(PUBLIC_KEY_FILE))
			encrypted_data = @@public_key.public_encrypt(data)
			Base64.encode64(encrypted_data)
		end

		def self.decrypt(data)
			@@private_key ||= OpenSSL::PKey::RSA.new(File.read(PRIVATE_KEY_FILE))
			decoded_data = Base64.decode64(data)
			@@private_key.private_decrypt(decoded_data)
		end
	end

	class OpenSSL_RSA
		IV64 = "xxxxxxxxxxxxxxxxxxxxxxxxxx==\n"
		KEY64 = "xxxxxxxxxxxxxxxxxxxxxxxxxx=\n"
		CIPHER = 'aes-256-cbc'

		def self.encrypt(data)
			@@iv ||= Base64.decode64(IV64)
			@@key ||= Base64.decode64(KEY64)

			cipher = OpenSSL::Cipher::Cipher.new(CIPHER)
			cipher.encrypt
			cipher.key = @@key
			cipher.iv = @@iv
			encrypted_data = cipher.update(data)
			encrypted_data << cipher.final
			Base64.encode64(encrypted_data)
		end

		def self.decrypt(data)
			@@iv ||= Base64.decode64(IV64)
			@@key ||= Base64.decode64(KEY64)

			cipher = OpenSSL::Cipher::Cipher.new(CIPHER)
			cipher.decrypt
			cipher.key = @@key
			cipher.iv = @@iv
			decrypted_data = cipher.update(Base64.decode64(data))
			decrypted_data << cipher.final
		end
	end
end

This provides two classes, Encryption::OpenSSL_Key and Encryption::OpenSSL_RSA which may be used to encrypt arbitrary strings. The OpenSSL_Key class uses a public/private keypair (in our example, read out of the Rails config directory), and the OpenSSL_RSA class uses an initialization vector and secret key. The latter is probably easier, since it means you don’t have to worry about keypairs, and since all the encrypt/decrypt is done locally, there isn’t any need for public public encryption.

Once you have that file in your project, using it is pretty simple.

# Our databse is going to have a field called encrypted_password. We'll use attr_accessor for the password itself.

class MySecretModal < ActiveRecord::Base
	before_save :encrypt_fields
	attr_accessor :password

	def password
		@decrypted_password ||= decrypt_field(:password)
	end

private

	def encrypt_fields
		write_attribute :encrypted_password, Encryption::OpenSSL_RSA.encrypt(@password)
	end

	def decrypt_field(field)
		Encryption::OpenSSL_RSA.decrypt read_attribute("encrypted_#{field}")
	end
end

The net result is that we can still get access to the raw password if we need to, but the content in the database will be RSA-encrypted against a secret key in our application. This is still vulnerable if the attacker gains access to the file containing your RSA IV/key, or if he gains access to your public/private keypair, but it is extremely resilient in the case that an attacker manages to simply dump your users table via SQL injection. You still need to practice good key management, and you absolutely should not use a technique this simplistic for storing financial data – there are a whole set of guidelines and procedures for that kind of information. However, for adding an extra layer of defense to save yourself and your customers from excess embarrassment in the case of a database breach, this is a quick, easy, and effective technique for hardening your data.

This is a rather raw implementation, and there are ways you could package it up so that you could transparently apply it to any number of models or fields, but the basic technique is solid. You could even use something like sql_crypt to easily protect sensitive fields. The technology is there, and “We needed to be able to re-use the password!” isn’t an excuse anymore. Stop storing plaintext passwords – just like backups, it’s just extra work until you need it, and then you’ll be glad you put that extra work in.

First, the boring documentation

All the juicy information is available in the documentation, but I’m going to just go over the key points real quick.

RUBY_HEAP_MIN_SLOTS: This is the number of “heap slots” that each Ruby instance starts up with. One heap slot can hold one Ruby object. By default, this is 10,000. By controlling this value, we can get our apps to stabilize very quickly. More on this later.

RUBY_HEAP_SLOTS_INCREMENT: Once Ruby has allocated RUBY_HEAP_MIN_SLOTS objects on its first heap, it will have to allocate a second heap to make room for more. This variable controls the size of this second heap, and sets the baseline for future heaps, as well.

RUBY_HEAP_SLOTS_GROWTH_FACTOR: For heaps #3 and onward, Ruby uses RUBY_HEAP_SLOTS_INCREMENT and this value to determine the size to allocate for the new heap. By default, this is 1.8, meaning that your third heap will end up with 10,000 * 1.8 = 18,000 slots in it.

RUBY_HEAP_FREE_MIN: After each garbage collection run, if the number of free slots is less than RUBY_HEAP_FREE_MIN, a new heap will be allocated. The default is 4096.

So, let’s look at this practically. Presume that we have a Rails process that is going to require 50,000 Ruby objects before it’s fully initialized. The allocation process, when at defaults, will look something like this:

Allocate 10,000 slots (10,000 total available)
Allocate 10,000 slots (20,000 total available)
Allocate 18,000 slots (38,000 total available)
Allocate 68,400 slots (106,400 total available)

So, we end up with about 53% more slots than we actually needed, and it took us four heap allocations to even boot the process. Surely we can do better.

Enter Scrap.

Scrap is a little Metal handler I wrote for tracking memory usage and garbage statistics over an instance’s lifetime. Installing it is trivial – just drop it into your vendor directory, restart your app, and navigate to http://yoururl.com/stats/scrap.

With this in hand, we can peek our memory usage and see what we can see.

There are some stats at the top, but for our purposes, we’re interested in the per-request garbage statistics. The newest request is near the top of the file, and the oldest request is at the bottom of the file. The last 50 requests are tracked. Each request looks something like this:

[71.92 MB] GET /apps/176568-WordPress

Number of objects    : 817571 (658305 AST nodes, 80.52%)
Heap slot size       : 20
GC cycles so far     : 503
Number of heaps      : 7
Total size of objects: 15968.18 KB
Total size of heaps  : 18036.81 KB (2068.63 KB = 11.47% unused)
Leading free slots   : 27104 (529.38 KB = 2.93%)
Trailing free slots  : 1 (0.02 KB = 0.00%)
Number of contiguous groups of 16 slots: 2829 (4.90%)
Number of terminal objects: 4307 (0.47%)

Key points here for the time being are Number of objects and Number of heaps. When we look at the number of objects – in this case, 817,000, it’s obvious that we’re going to have to allocate a number of heaps to handle all those objects. Rails’ boot-up cost is fairly significant, and the default Ruby settings just really don’t cut it here. As you can see, we’ve allocated 7 heaps, and we’re using 15.9 of 18.0 MB allocated to the heap. Once a heap is allocated, it’s never de-allocated, so we’re perma-stuck at 18 MB of heap usage. Note that this isn’t the size of all the data in the program – just the space allocated for objects. A string that contains 100MB of data will only consume 20 bytes (that’s the “heap slot size – the amount of memory each object on the heap consumes”) on the heap.

However, what if we could just allocate the whole startup cost in the initial heap, and save ourselves the problems of having to reallocate so often?

We note that we have 891k slots allocated, so we can guesstimate at a number to set our initial allocation to. In my production app, I set mine to 1,250,000 – I was observing peaks around the 1,100,000 mark, and just increased it by 10% and rounded up.

So, my first custom environment variable is

RUBY_HEAP_MIN_SLOTS=1250000

And it results in something like this on the app’s first boot:

[137.99 MB] GET /movies/7505-Star-Wars-Episode-V-The-Empire-Strikes-Back

Number of objects    : 933037 (664785 AST nodes, 71.25%)
Heap slot size       : 20
GC cycles so far     : 12
Number of heaps      : 1
Total size of objects: 18223.38 KB
Total size of heaps  : 24414.08 KB (6190.70 KB = 25.36% unused)
Leading free slots   : 316963 (6190.68 KB = 25.36%)
Trailing free slots  : 0 (0.00 KB = 0.00%)
Number of contiguous groups of 16 slots: 19810 (25.36%)
Number of terminal objects: 25941 (2.08%)

Yowza, a full 25% of my heap is unused after boot. But…well, that’s okay. We’ve only allocated 1 heap, and later on, my object allocation grows to around 1,100,000. This is still 15k under the heap size, and I’ve set RUBY_HEAP_FREE_MIN=12500 (1% of the initial size), so if I have less than 12,500 heap objects free after a GC cycle, a new heap will be allocated. Stabilizing there means that I end up with 1 heap for the lifetime of my app, and I end up sitting just under the threshold that’d cause a new heap to be born. If I have a leak, or a super heavy action or something, though, that might kick me over my limit and require a new heap. So, we come to…

RUBY_HEAP_SLOTS_INCREMENT=100000

This value says “Hey, if you have to allocate a second heap, start with this many slots”. If we go over our limit of 1.25 million slots, we’ll allocate a second heap that’s about 8% the size of the original. That seems awfully small, but consider that we’re hoping to never get to that heap.

Should we end up using that entire second heap, then we have to worry about our third setting, RUBY_HEAP_SLOTS_GROWTH_FACTOR=1. This says “Each new heap should be 1.0 as large as the previous heap.” In this case, it means I’ll keep allocating 100k-slot heaps until the cows come home. In an untuned environment, this could be bad – we would either end up having to do a ton of allocations to get to our target, or we would overallocate very badly. However, because we know our app’s memory requirements, and know about where we want it to end up, a relatively small, linear growth factor is just what the doctor ordered here.

Okay, now what?

So, we have a collection of settings with which to run our app. Great! Now, how do we use it?

Fortunately, it’s easy.

pushd `which ruby | xargs dirname`
sudo vim ruby-with-env

We’re going to create a little bash script with the following:

#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=1250000
export RUBY_HEAP_SLOTS_INCREMENT=100000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=30000000
export RUBY_HEAP_FREE_MIN=12500
exec "/opt/ree/bin/ruby" "$@"

Note that last line – the path will have to match the path to your Ruby executable, which fortunately, should be in the directory that you’re in.

Save it, don’t forget to chmod a+x ruby-with-env, and then edit your Apache or nginx configuration.

Under nginx, you’ll have a line like this:

passenger_ruby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby;

Just change it to use your new wrapper script, like so:

passenger_ruby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby-with-env;

The process is similarly easy for Apache – the line you need is something like:

PassengerRuby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby

It might be in either your httpd.conf or conf.d/passenger.conf.

Once you’re all edited up, restart your webserver, and congratulations, you’ve got a fine-tuned garbage collector humming along with your app.

Taking out the garbage

“But Chris!”, you say, “There’s a variable in there that you didn’t talk about! What gives?” You are indeed correct, astute reader. We’ve thus far avoided the RUBY_GC_MALLOC_LIMIT variable. This is a handle little setting that lets you tell Ruby how often to clean up after itself. Ruby is written in C, and C uses malloc to allocate memory. Ruby just keeps a little counter each time it allocates an object with malloc, and it runs its garbage collector after so many malloc calls have been made. I haven’t found a great way to tune this one yet, except via experimentation, but here’s what to know about it:

1. The lower this value is, the more often your garbage collector runs. Garbage collection is slow. Garbage collection is painfully slow. If a user is waiting on garbage collection, they are going to become impatient. You want as few users waiting on garbage collection as possible.
2. The higher this value is, the more memory Ruby will allocate before it tries to clean up after itself. If this value is too high, you’ll have dead objects hanging around eating up heap space, and possibly causing Ruby to crap itself and allocate a new heap. This is bad.
3. To tune this value, you want to find the happy medium, wherein you stabilize under your initial heap allocation value, but with as few garbage collection passes as possible. Read up on Evan Weaver’s blog for some more in-depth analysis of what garbage collection frequency tuning can do to your app’s performance.
4. If you have excess memory and want a faster app, err on the side of this being too high. If you are on a tight memory budget, and would prefer slower actions in exchange for not blowing your heap and allocating a whole new one, err on the side of this being too low.
5. Recommended values for this are all over the board. Evan recommends a setting of 50 million. I’m using a setting of 30 million. The Ruby default is 8 million. You’ll have to play around and find what works best for you. Just pay attention to how many requests there are in between that “GC cycles so far” number incrementing in Scrap, and you’ll be able to measure approximately how often you’re entering a GC cycle.

Good luck with it, and have fun!

It’s up at github – installation is dead easy (assuming you’re on Rails 2.3+, of course) – just install the plugin, restart your app, and hit <your url>/stats/scrap in your browser. Bam, instant juicy memory goodness about your app at your fingertips. If you’d like an example of the output, good news! Check it out at http://tachyonsix.com/scrap.htm.

You can use it to troubleshoot heap leaks – just run a few requests, hit your Scrap URL, and see what your deltas look like. Seeing a huge growth in a certain type of object? Chances are pretty good that you have a heap leak, and can start tracking it down.

The request history can help you locate certain actions that might be causing spikes in memory usage. It’ll show the last N requests, along with memory and heap statistics before each request. If there’s a consistent memory usage leap after a certain action, chances are that it’s doing something naughty.

Want to get a bigger picture on what objects are hanging around? You can use the config/scrap.yml file to get Scrap to spit out more detailed reports on instances of a given class. There’s full documentation on it in the README.

Anyhow, give it a shot, let me know what you think.

There are a number of monitoring solutions that are popular these days, such as monit, god, and Nagios. They’re all fantastic, but sometimes you just want something simple and to-the-point, right?

With jabberish in your project, this becomes a no-brainer. I’m already using Jabberish in my project, so I whipped up a little script that checks system load, available memory, and any changes in swap usage and shoots me an IM under certain conditions. My monitoring still handles automated maintenance in the case of a runaway process or whatnot, but this keeps me instantly informed of any problems that my system might be running in to.

require 'rubygems'
require 'drb'
require 'daemons'

MAX_MEMORY = 95
MAX_LOAD = 4.0
DELIVER_TO = "cheald@gmail.com"
JABBERISH_SERVER = "druby://localhost:35505"

$warned = {}
$hostname = `hostname`.strip

def im
	$im_service ||= DRbObject.new(nil, JABBERISH_SERVER)
end

def deliver(msg)
	im.deliver DELIVER_TO, "[#{$hostname}] #{msg}"
end

def check_stats
	meminfo = open("/proc/meminfo").read

	mtotal = meminfo.match(/MemTotal:\s+(\d+)/)[1].to_i
	mfree = meminfo.match(/MemFree:\s+(\d+)/)[1].to_i
	mused = mtotal - mfree

	stotal = meminfo.match(/SwapTotal:\s+(\d+)/)[1].to_i
	sfree = meminfo.match(/SwapFree:\s+(\d+)/)[1].to_i
	sused = stotal - sfree

	begin
		if $warned[:swap] and sused > $warned[:swap] then
			deliver "WARNING: Swap has increased from #{$warned[:swap]} to #{sused}"
		end
		$warned[:swap] = sused

		pct = mused / mtotal.to_f * 100.0
		if pct > MAX_MEMORY then
			unless $warned[:memory]
				deliver sprintf("ALERT: Memory free: %2.2fmb (%2.2f%% used)", mfree / 1024.0, pct)
				$warned[:memory] = true
			end
		else
			$warned[:memory] = false
		end

		load = open("/proc/loadavg").read.split(" ").first
		if load > MAX_LOAD then
			unless $warned[:load]
				deliver sprintf("WARNING: Load average is %s", load)
				$warned[:load] = true
			end
		else
			$warned[:load] = false
		end
	rescue
		puts "Error: #{$!}"
	end
end

Daemons.daemonize(:backtrace => true)

loop {
	check_stats
	sleep(10)
}

Not too bad, huh? This is written for a CentOS installation, so you may need to change things like the meminfo regexes depending on your system. It could probably use a YAML config file to be truly correct – configuration options in constants works, but is a little ugly.

Now I get alerts like these via instant message:

[iceman.tagteam] WARNING: Load average is 4.44
[iceman.tagteam] ALERT: Memory free: 99.82mb (93.38% used)
[polaris.tagteam] ALERT: Memory free: 72.20mb (95.14% used)

This lets me respond to changing system conditions extremely rapidly, and serves as a high-level alert log when when I’m not at the keyboard – when I get back, I check my messages from blippr, and can see when and how often certain marginal conditions are being met.

Hope it’s useful!

if foo then
	do_stuff
end

Ruby will let you clean that up with:

do_stuff if foo

This works just nearly all the time, but I ran into an odd problem today, where the trailing conditions were producing behavior I didn’t want.

>> foobar
NameError: undefined local variable or method `foobar' for #<Object:0x92bc998>
        from (irb#1):2
>> foobar = true unless defined?(foobar)
=> nil
>> foobar
=> nil
>> unless defined?(foobar); foobar = true; end
=> true
>> foobar
=> true

Wait, what? Using the trailing conditional changes the order in which Ruby parses the statement, resulting in something like the following operations:

1. Define foobar because it’s referenced, set it to nil
2. Parse the unless conditional
3. If the condition is true, set foobar to true

The kicker here is that because foobar‘s assignment is the first thing parsed, it’s always initialized before you ever get to the defined? statement. So instead, we run the second piece of code:

unless defined?(foobar); foobar = true; end

This runs something like the following:

1. Parse the unless condition.
2. Define foobar because it’s referenced, set it to nil
3. If the condition is true, set foobar to true

Obviously this is the desired behavior. Several lessons here:

• Ruby initializes variables when they are parsed, not when the code path that contains them is run (in fact, it’ll even initialize variables that are in unreachable code paths!)
• if condition then do_stuff end is not always the same as do_stuff if condition

It’s a bit of an edge case, but it’s an edge case that had me baffled. Hopefully this post saves you some frustration.

That’ll work, but if speed is a factor, it might not be your best option.

ActiveRecord makes interface to the DB very easy, but it doesn’t necessarily make it fast. Instantiating an ActiveRecord object is costly, and if you do a lot of ‘em, that’s going to cause you to bump up against the garbage collector, which will significantly hinder performance. There are several options, though, depending on how much speed you need.

There are benchmarks at the bottom of the post, so if you’re just interested in those, scroll down.

Option 1: Use transactions

This is definitely the easiest method, and while you’ll realize gains from it, you aren’t going to be breaking any speed records using only this method. However, it’s well worth it if you are doing mass inserts via ActiveRecord.

Instead of

1000.times { Model.create(options) }

You want:

ActiveRecord::Base.transaction do
  1000.times { Model.create(options) }
end

The net effect is that the database performs all of your inserts in a single transaction, rather than starting and committing a new transaction for every request.

Options 2: Get down and dirty with the raw SQL

If you know that your data is valid and can afford to skip validations, you can save a lot of time by just jumping directly to raw SQL.

Imagine, for example, that you’re running the following:

1000.times {|i| Foo.create(:counter => i) }

That’s going to create 1000 ActiveRecord objects, run validations, generate the insert SQL, and dump it into the database. You can realize large performance gains by just jumping directly to the generated SQL.

1000.times do |i|
  Foo.connection.execute "INSERT INTO foos (counter) values (#{i})"
end

You should use sanitize_sql and such as necessary to sanitize input values if they are not already sanitized, but with this technique you can realize extremely large performance gains. Of course, wrapping all those inserts in a single transaction, as in Option 1 gets you even more performance.

Foo.transaction do
  1000.times do |i|
    Foo.connection.execute "INSERT INTO foos (counter) values (#{i})"
  end
end

Option 3: A single mass insert

Many databases support mass inserts of data in a single insert statement. They are able to significantly optimize this operation under the hood, and if you’re comfortable using it, will be your fastest option by far.

inserts = []
TIMES.times do
  inserts.push "(3.0, '2009-01-23 20:21:13', 2, 1)"
end
sql = "INSERT INTO user_node_scores (`score`, `updated_at`, `node_id`, `user_id`) VALUES #{inserts.join(", ")}"

No transaction block is necessary here, since it’s just a single statement, and the DB will wrap it in a transaction. We build an array of value sets to include, then just join them into the INSERT statement. We don’t use string concatenation, since it will result in significantly more string garbage generated, which could potentially get us into the GC, which we’re trying to avoid (and hey, memory savings are always good).

Option 4: ActiveRecord::Extensions

njero in #rubyonrails pointed me at this nifty little gem and I decided to include it. It seems to try to intelligently do mass inserts of data. I wasn’t able to get it to emulate the single mass insert for a MySQL database, but it does provide a significant speed increase without much additional work, and can preserve your validations and such.

There’s the obvious added benefit that you stay in pure Ruby, and don’t have to get into the raw SQL.

columns = [:score, :node_id, :user_id]
values = []
TIMES.times do
	values.push [3, 2, 1]
end

UserNodeScore.import columns, values

Benchmarks

I used a simple script to test each of the methods described here.

require "ar-extensions"

CONN = ActiveRecord::Base.connection
TIMES = 10000

def do_inserts
	TIMES.times { UserNodeScore.create(:user_id => 1, :node_id => 2, :score => 3) }
end

def raw_sql
	TIMES.times { CONN.execute "INSERT INTO `user_node_scores` (`score`, `updated_at`, `node_id`, `user_id`) VALUES(3.0, '2009-01-23 20:21:13', 2, 1)" }
end

def mass_insert
	inserts = []
	TIMES.times do
		inserts.push "(3.0, '2009-01-23 20:21:13', 2, 1)"
	end
	sql = "INSERT INTO user_node_scores (`score`, `updated_at`, `node_id`, `user_id`) VALUES #{inserts.join(", ")}"
	CONN.execute sql
end

def activerecord_extensions_mass_insert(validate = true)
	columns = [:score, :node_id, :user_id]
	values = []
	TIMES.times do
		values.push [3, 2, 1]
	end

	UserNodeScore.import columns, values, {:validate => validate}
end

puts "Testing various insert methods for #{TIMES} inserts\n"
puts "ActiveRecord without transaction:"
puts base = Benchmark.measure { do_inserts }

puts "ActiveRecord with transaction:"
puts bench = Benchmark.measure { ActiveRecord::Base.transaction { do_inserts } }
puts sprintf("  %2.2fx faster than base", base.real / bench.real)

puts "Raw SQL without transaction:"
puts bench = Benchmark.measure { raw_sql }
puts sprintf("  %2.2fx faster than base", base.real / bench.real)

puts "Raw SQL with transaction:"
puts bench = Benchmark.measure { ActiveRecord::Base.transaction { raw_sql } }
puts sprintf("  %2.2fx faster than base", base.real / bench.real)

puts "Single mass insert:"
puts bench = Benchmark.measure { mass_insert }
puts sprintf("  %2.2fx faster than base", base.real / bench.real)

puts "ActiveRecord::Extensions mass insert:"
puts bench = Benchmark.measure { activerecord_extensions_mass_insert }
puts sprintf("  %2.2fx faster than base", base.real / bench.real)

puts "ActiveRecord::Extensions mass insert without validations:"
puts bench = Benchmark.measure { activerecord_extensions_mass_insert(true)  }
puts sprintf("  %2.2fx faster than base", base.real / bench.real)

And the results:

Testing various insert methods for 10000 inserts
ActiveRecord without transaction:
 14.930000   0.640000  15.570000 ( 18.898352)
ActiveRecord with transaction:
 13.420000   0.310000  13.730000 ( 14.619136)
  1.29x faster than base
Raw SQL without transaction:
  0.920000   0.170000   1.090000 (  3.731032)
  5.07x faster than base
Raw SQL with transaction:
  0.870000   0.150000   1.020000 (  1.648834)
  11.46x faster than base
Single mass insert:
  0.000000   0.000000   0.000000 (  0.268634)
  70.35x faster than base
ActiveRecord::Extensions mass insert:
  6.580000   0.280000   6.860000 (  9.409169)
  2.01x faster than base
ActiveRecord::Extensions mass insert without validations:
  6.550000   0.240000   6.790000 (  9.446273)
  2.00x faster than base

The results are fairly self-explainatory, but of particular note is the specific single INSERT statement. At 70x faster than the non-transactional ActiveRecord insert, if you need speed, it’s hard to beat.

Conclusions

ActiveRecord is great, but sometimes it’ll hold you back. Finding the balance between ease of use (full ActiveRecord) and performance (bare metal mass inserts) can have a profound effect on the performance of your app.