It’s not too hard to set up fixtures to be inserted once per test suite run –

In your test_helper.rb, above the class ActiveSupport::TestCase definition, add the following:

Fixtures.reset_cache
fixtures_folder = File.join(RAILS_ROOT, 'test', 'fixtures')
fixtures = Dir[File.join(fixtures_folder, '*.yml')].map {|f| File.basename(f, '.yml') }
Fixtures.create_fixtures(fixtures_folder, fixtures)
Fixtures.reset_cache

Next, turn off transactional fixtures and comment out the fixtures macro:

self.use_transactional_fixtures = false
# fixtures :all

That’s all there is to it. Your fixtures will be inserted into your test database once when test_helper is included for the first time, and then not again for the rest of the test suite run. This should speed your tests up substantially.

Enter IE.

“Oh, crap”, you think.

Modern web design in IE is a pain in the rear. Fortunately, we have modern tools that make it a not-pain.

• SASS (Syntactically Awesome Stylesheets) is a macro language for CSS. It lets you express CSS as nested rules, and gives you mix-ins, functionality extensions, variables, partials, and a whole lot more.
• CSSPie is a set of behaviors for Internet Explorer that gives you CSS3 visual styles without really slow Javascript hacks like CurvyCorners.

When combined, the two are a shot of liquid awesome injected directly into your brain.

I’ve settled on a fairly standard setup for my projects. I have:

• My application.sass file.
• My _mixins.sass partial.
• My PIE.htc behavior file.

Macros is very straightforward:

@mixin pie
  behavior: url(/behaviors/PIE.htc)
.pie
  +pie

@mixin shadows($color: #aaa, $x: 1px, $y: 2px, $spread: 2px)
  @extend .pie
  -moz-box-shadow: $color $x $y $spread
  -webkit-box-shadow: $color $x $y $spread
  box-shadow: $color $x $y $spread

@mixin inset-shadows($color: #aaa, $x: 1px, $y: 1px, $spread: 1px)
  @extend .pie
  -moz-box-shadow: inset $x $y $spread $color
  -webkit-box-shadow: inset $x $y $spread $color
  box-shadow: inset $x $y $spread $color

@mixin corners($tl: 5px, $tr: nil, $br: nil, $bl: nil)
  @extend .pie
  @if $tr == nil
    $tr: $tl
  @if $br == nil
    $br: $tl
  @if $bl == nil
    $bl: $tl
  -moz-border-radius: $tl $tr $br $bl
  -webkit-border-top-left-radius: $tl
  -webkit-border-bottom-left-radius: $bl
  -webkit-border-top-right-radius: $tr
  -webkit-border-bottom-right-radius: $br
  border-radius: $tl $tr $br $bl

@mixin vertical-gradient($start: #000, $end: #ccc)
  @extend .pie
  background: $end
  background: -webkit-gradient( linear, left top, left bottom, color-stop(0, $start), color-stop(1, $end) )
  background: -moz-linear-gradient(center top, $start 0%, $end 100%)
  -pie-background: linear-gradient(90deg, $start, $end)

What’s going on here? We’re defining several mix-ins for Sass:

+shadows([color, [x, [y, [spread]]]])

+inset-shadows([color, [x, [y, [spread]]]])

+corners(size)

+corners(topleft, topright, bottomright, bottomleft)

+vertical-gradient(start, end)

Now, in your CSS, you can just do the following:

body
  font:
    family: Arial
    size: 10pt

.box
  +corners
  +shadows(#ccc)
  +vertical-gradient(#eee, #fff)

  h3
    color: #444

.dark-box
  +corners(20px)
  +shadows(#888, 4px, 4px, 4px)
  +vertical-gradient(#444, #000)
  color: #fff
  h3
    color: #fff

.box, .dark-box
  padding: 1em
  margin-bottom: 1em

This expands to:

.pie, .box, .dark-box {
  behavior: url(/projects/PIE.htc);
}

body {
  font-family: Arial;
  font-size: 10pt;
}

.box {
  -moz-border-radius: 5px 5px 5px 5px;
  -webkit-border-top-left-radius: 5px;
  -webkit-border-bottom-left-radius: 5px;
  -webkit-border-top-right-radius: 5px;
  -webkit-border-bottom-right-radius: 5px;
  border-radius: 5px 5px 5px 5px;
  -moz-box-shadow: #cccccc 1px 2px 2px;
  -webkit-box-shadow: #cccccc 1px 2px 2px;
  box-shadow: #cccccc 1px 2px 2px;
  background: white;
  background: -webkit-gradient(linear, left top, left bottom, color-stop(0, #eeeeee), color-stop(1, white));
  background: -moz-linear-gradient(center top, #eeeeee 0%, white 100%);
  -pie-background: linear-gradient(270deg, #eeeeee, white);
}
.box h3 {
  color: #444444;
}

.dark-box {
  -moz-border-radius: 20px 20px 20px 20px;
  -webkit-border-top-left-radius: 20px;
  -webkit-border-bottom-left-radius: 20px;
  -webkit-border-top-right-radius: 20px;
  -webkit-border-bottom-right-radius: 20px;
  border-radius: 20px 20px 20px 20px;
  -moz-box-shadow: #888888 4px 4px 4px;
  -webkit-box-shadow: #888888 4px 4px 4px;
  box-shadow: #888888 4px 4px 4px;
  background: black;
  background: -webkit-gradient(linear, left top, left bottom, color-stop(0, #444444), color-stop(1, black));
  background: -moz-linear-gradient(center top, #444444 0%, black 100%);
  -pie-background: linear-gradient(270deg, #444444, black);
  color: white;
}
.dark-box h3 {
  color: white;
}

.box, .dark-box {
  padding: 1em;
  margin-bottom: 1em;
}

Check out the live demo.

Here are screenshots of the demo in Chrome 6, Firefox 4.0b3, Internet Explorer 8. Can you tell which browser is which?

1. Set up object creation tracking on the Set class
2. Find the leaky instance of our set using GDB
3. Locate what code created that Set instance

First things first. Check out Sean Bradly’s GCTrack module. I’m using a modified version of it to include object IDs in the tracker struct, so that we can perform reverse-lookups on an instance.

# Sean Bradly (rhythmx at gmail) - 2010
# Modified by Chris Heald (cheald@gmail.com) - August 2010
module GCTrack
  # Class to track an object's lifecycle and where it was created
  class Tracepoint < Struct.new(:num, :call, :insp, :id)
    include Comparable
    # make it so the obj.num works as a simple key
    def <=>(obj)
      if obj.class <= self.class
        self.num <=> obj.num
      else
        self.num <=> obj
      end
    end
  end

  def self.included(klass)

    # Hook initialize to track this object
    klass.class_eval do
      alias_method :old_init, :initialize

      def initialize(*args)
        # Call original init
        old_init(*args)
        # trace creation and setup finalizer callback
        final_proc = self.class.gct_setup_finalizer(self, caller, inspect)
        # define the finalizer
        ObjectSpace.define_finalizer(self, final_proc)
      end
    end

    # Add some metaclass foo to track all the objects.
    # You cant track in an instance w/o creating circular
    # references and breaking Ruby's GC

    (
    class << klass;
      self;
    end).class_eval do
      def gct_setup_finalizer(obj, caller, insp)
        # create/log a tracepoint for this object
        num = gct_objnum
        tpoint = Tracepoint.new(num, caller, insp, obj.object_id)
        gct_add(tpoint)
        # call option init cb
        if self.respond_to?(:created)
          self.send(:created, tpoint)
        end
        obj = nil # don't let the proc track a ref to 'obj'
        # this callback happens at GC time
        final_proc = Proc.new do
          gct_del(tpoint)
          # call optional final cb
          if self.respond_to?(:deleted)
            self.send(:deleted, tpoint)
          end
        end
      end

      def gct_objnum
        @num ||= -1
        @num  +=  1
      end

      def gct_addlog
        @addlog ||= []
      end

      def gct_dellog
        @dellog ||= []
      end

      def gct_add(num)
        gct_addlog << num
      end

      def gct_del(num)
        gct_dellog << num
      end

      def gct_active
        gct_addlog - gct_dellog
      end

      def gct_origin(id)
        x = gct_addlog.select {|g| g.id == id }.first
        x.nil? ? nil : x.call.join("\n")
      end

      def gct_orphan_report
        # Get the leaked objects
        rpt     = ""
        leaked  = gct_active
        callers = leaked.map { |o| o.call }.sort.uniq
        # Iterate over source lines that have leaked objs
        callers.each do |calla|
          leaked_here = leaked.find_all { |o| o.call == calla }
          rpt << "==== #{leaked_here.size} leaked objects from:\n\n"
          calla.each { |l| rpt << "    #{l}\n" }
          rpt << "\n"
          rpt << "    == object data:\n\n"

          leaked_here.each do |o|
            rpt << "    num: #{o.num}, inspect: #{o.insp}\n"
          end
          rpt << "\n"
        end
        rpt
      end

    end
  end
end

Additionally, you’ll want to patch the object you’re interested in tracking.

class Set
  include GCTrack
end

You’ll want to include both of those at the top of your config.rb, before your initializer block. If you’re tracking an object defined in your Rails app, just include GCTrack in the object definition, rather than in a monkeypatch.

Next, you need to be sure that you have your GDB macros set up properly. /root/.gdbinit should have the following:

define eval
  call(rb_p(rb_eval_string_protect($arg0,(int*)0)))
end

define redirect_stdout
  call rb_eval_string("$_old_stdout, $stdout = $stdout, File.open('/tmp/ruby-debug.' + Process.pid.to_s, 'a'); $stdout.sync = true")
end

(re)start your application, hit the leaky action a few times, and find its PID using ps ax or top. Once you have that, attach to the process with gdb, redirect your process’s stdout to the tmp file with redirect_stdout and dump all Sets in your object space.

(gdb) attach 12345
(gdb) redirect_stdout
$8 = 2
(gdb) eval "GC.start"
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" if o.is_a?(Set) }; puts '----'"

Now that we have that, let’s look at the tmp file, and locate the leaked set instance.

Set, #<Set: {Achievement, Achievement, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment}> -- 78376680

There, at the end, is our object ID. Now back over to gdb.

(gdb) eval "puts Set.gct_addlog.select {|g| g.id == 78376680}.first.call.join(\"\n\")"

Your tmp file should now have a stack trace for that object’s allocation now. Track it down and beat it into submission.

/opt/ruby-enterprise-1.8.7-2010.02/lib/ruby/gems/1.8/gems/mongo_mapper-0.8.3/lib/mongo_mapper/support/descendant_appends.rb:15:in `new'

You can do this for just about any object you’d like. Just mix in your GCTrack and easily find where in code a particular instance of an object was created. Just like that, debugging memory leaks goes from hunting guppies in the Atlantic to shooting fish in a barrel.

Have fun!

I set out to track down the cause tonight, and have both fixed the problem and gotten some handy experience debugging Rails apps with gdb.

The Solution

First off, if you just want the fix, here it is. You probably have a middleware to clear your identity maps already. We’re just going to modify that. In my case, it’s lib/mongo_mapper/per_request_identity_map_clear.rb.

module MongoMapper
  class PerRequestMapClear
    def initialize(app)
      @app = app
    end

    def call(env)
      if Rails.configuration.cache_classes
        MongoMapper::Plugins::IdentityMap.clear
      else
        MongoMapper::Document.descendants.each do |m|
          m.descendants.clear if m.respond_to? :descendants
        end
        MongoMapper::Document.descendants.clear
        MongoMapper::EmbeddedDocument.descendants.clear
        MongoMapper::Plugins::IdentityMap.clear
        MongoMapper::Plugins::IdentityMap.models.clear
      end
      @app.call(env)
    end
  end
end

In particular, these two lines:

  MongoMapper::Document.descendants.clear
  MongoMapper::Plugins::IdentityMap.models.clear

Make sure you get that into your middleware stack and all your MongoMapper memory leaking issues will magically disappear.

The Problem

You see, MongoMapper is doing some dag-nasty evil stuff with class references. Namely, it’s holding onto them in class variables that don’t get reloaded per-request in Rails. Simply:

1. Rails loads your User model. This loads up a ton of Validations, MongoMapper::Keys, procs, and whatever else you have in your model definition. It’s not really light. That’s okay because we load it only once in production.
2. This is a MongoMapper::Document, which causes User (the class reference) to be inserted into the MongoMapper::Document.descendants class attribute, which happens to be a Set object.
3. Next time you refresh a page, Rails re-loads your User model. This creates a new and separate User class reference. It does not replace the previous User class reference. It is not equal to your previous User class reference. They are, as far as Ruby cares, separate objects.
4. MongoMapper happily sticks that class reference into its descendants class attribute. You now have two separate copies of User. Since MongoMapper is holding onto a reference to your old User class, Ruby can never garbage collect it. The old User class and all of its huge cascading chain of referenced objects are now “leaked”.
5. Your memory usage is increasing by 4mb per request now.

I had app instances reaching nearly 1GB of RAM usage after light testing. I finally noticed it when my development machine kicked into swap and actions that took 80ms to run were taking 8000ms to run. Hm. That might be a problem!

Debugging leaks with GDB

GDB is an amazing tool. With a few macros, you can make hacking around in a live Ruby instance pretty painless. Crack open your /root/.gdbinit file and add a few macros:

define eval
  call(rb_p(rb_eval_string_protect($arg0,(int*)0)))
end

define redirect_stdout
  call rb_eval_string("$_old_stdout, $stdout = $stdout, File.open('/tmp/ruby-debug.' + Process.pid.to_s, 'a'); $stdout.sync = true")
end

Now, we’re going to attach to your running Ruby process. This needs to be done as root.

[root@polaris ~]# gdb
(gdb) attach 12019
(gdb) redirect_stdout
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" unless o.is_a?(String) }; puts '----'"

This will effectively dump all non-String objects in the attached Ruby process to tmp/ruby-debug.12019. This takes a little bit, but it lets you come up with some handy data for parsing later.

To get data we can compare, we’ll need to dump the environment for multiple requests:

[root@polaris ~]# gdb
(gdb) attach 12019
(gdb) redirect_stdout
(gdb) eval "GC.start"
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" unless o.is_a?(String) }; puts '----'"
(gdb) detach

(run some requests)

(gdb) attach 12019
(gdb) redirect_stdout
(gdb) eval "GC.start"
(gdb) eval "ObjectSpace.each_object {|o| puts \"#{o.class.name}, #{o.inspect} -- #{o.object_id}\" unless o.is_a?(String) }; puts '----'"
(gdb) detach

At this point, you’ll have two ObjectSpace dumps in your temp file. For my purposes, I hacked up a quick little script to parse those dumps, and to output all objects that were not present in both dumps. Since I’m invoking GC.start, in theory, this should help me find my leaked objects.

runs = [[]]
open(ARGV[0]).each do |line|
  if line == "----\n" then
    runs << []
  elsif line.match "--" then
    runs.last << line.strip
  end
end

diff = []
runs.each do |run|
  diff = (diff - run) | (run - diff)
end
diff.sort.map {|d| puts d }

Not very pretty, but it does the job. Just a quick invocation to ruby find_leaked.rb /tmp/ruby-debug.12019 > leaked (well, not that quick, it took a minute to run) and I effectively had an ObjectSpace diff I could pore through.

There’s a lot of stuff in there. In particular, you’re going to notice that you have a LOT of Array, Hash, and MatchData objects (perhaps potential optimization points for future Rails releases?). While we may be interested in those, try to cull out the things that obviously aren’t a problem just for readability’s sake.

I pored through the diff looking for things related to MongoMapper or my models. After not too long, I came across these lines:

Set, #<Set: {Achievement, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment, Achievement, Achievement, EmbeddedComment, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment, Achievement, Achievement, Achievement, EmbeddedComment, EmbeddedComment, Achievement}> -- 92034680
Set, #<Set: {Achievement, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment, Achievement, Achievement, EmbeddedComment, EmbeddedComment, Achievement, EmbeddedComment, EmbeddedComment, Achievement, Achievement, EmbeddedComment, Achievement, Achievement, EmbeddedComment, EmbeddedComment, Achievement}> -- 92034680

Whoa there. What? Why do I have Sets with multiple references to Achievement and EmbeddedComment? That doesn’t smell right. I suspect the problem lies in MongoMapper, so let’s grep the MongoMapper codebase for Set.

[chris@polaris lib]$ grep Set * -R
mongo_mapper/plugins/identity_map.rb:        @models ||= Set.new
mongo_mapper/plugins/modifiers.rb:          modifier_update('$addToSet', args)
mongo_mapper/plugins/protected.rb:          self.write_inheritable_attribute(:attr_protected, Set.new(attrs) + (protected_attributes || []))
mongo_mapper/plugins/accessible.rb:          self.write_inheritable_attribute(:attr_accessible, Set.new(attrs) + (accessible_attributes || []))
mongo_mapper/support/descendant_appends.rb:        @descendants ||= Set.new
mongo_mapper/connection.rb:      raise 'Set config before connecting. MongoMapper.config = {...}' unless defined?(@@config)
mongo_mapper/connection.rb:      raise 'Set config before connecting. MongoMapper.config = {...}' if config.blank?
mongo_mapper/extensions/set.rb:    module Set
mongo_mapper/extensions/set.rb:class Set
mongo_mapper/extensions/set.rb:  extend MongoMapper::Extensions::Set

Great, we have a hit list to look through. IdentityMap is worth looking at; a new set is created there, and its naming indicates that it’s for holding models, probably model references (it is). mongo_mapper/support/descendant_appends.rb is much the same deal. We can ignore mongo_mapper/plugins/accessible.rb since we can guess that the attrs being passed are symbols, rather than those class references we saw in the ObjectSpace diff.

Let’s crack open descendant_appends.rb

module MongoMapper
  module Support
    module DescendantAppends
      def included(model)
        extra_extensions.each { |extension| model.extend(extension) }
        extra_inclusions.each { |inclusion| model.send(:include, inclusion) }
        descendants << model
      end

      # @api public
      def descendants
        @descendants ||= Set.new
      end

Oh dear. And there it is. Every time MongoMapper::Support::DescendantAppends is included in a model (which is via MongoMapper::Document and MongoMapper:EmbeddedDocument), a reference to the including class is stored in a class variable.

Since we know that Rails reloads models per-request in development mode, and we know that each copy of a model’s class is not considered equivalent to the other copies of that class, it’s easy to see what happens here: We end up with sets like in our object dump, with however many orphaned old copies of our models, and all their various and sundry associated models.

And so, we arrive at our solution. By clearing the descendants set on every request where we are reloading our models, we ensure that there are not references to old copies of models left hanging around leaking memory.

My development instances are now running solid at 68mb apiece, rather than 1gb apiece. As you can imagine, the difference in response speed (and thus, productivity) is substantial.

Hope this helps. Have fun with gdb – it’s an obscenely powerful tool, and used properly, can give you a purely nutty amount of information which can be invaluable in tracking down memory leaks and related problems.

I recently discovered some very cool functionality it includes – the WillPaginate::Collection class can be used as a custom paginator for effectively any enumerable collection. It’s very simple, too. I recently used it to build pages of the most popular tags on posts in my database. My data store is MongoDB, and I’m fetching an array consisting of two-element arrays, [tag, tag_count]. To use will_paginate’s functionality with this, I just use the following:

tags = Post.tag_counts(nil, {:sort => ["value", "descending"]}) # Return an array of tag/count pairs. Custom function, so it can't leverage the finder on Post.
@topics = WillPaginate::Collection.create(current_page, 20, tags.length) do |pager|
	pager.replace(tags.slice(pager.offset, pager.offset + pager.per_page))
end

current_page is a helper that derives the current page from the request parameters. The rest of it is self-explanitory. I can now use @topics in my page just as I’d use a paginated result set from the database.

- @topics.each do |topic|
    # ...
=will_paginate @topics

Bam. Doesn’t get much easier than that. You can get exceptionally creative with it, too. Effectively, all you need to know is:

• WillPaginate::Collection#new takes 3 parameters: the current page, the per-page count, and optionally, the total number of entries.
• The pager block variable exposes offset and per_page properties, prime for passing into a DB query or slicing an enumerable with
• Call pager.replace(sub-array) with the current page’s set of elements.

That’s literally all there is to it. Now you can have easy pagination on just about any collection you can conceive of. Let WillPaginate handle all the heavy lifting and such. If you’ve done enough pagination by hand, you’ll probably appreciate the easy beauty of this particular method.

MongoDB 1.6 was released today, and it includes, among other things it includes support for the incredible sexy replica sets feature – basically master/slave replication on crack with automatic failover and the like. I’m setting it up, and figured I’d document the pieces as I walk through them.

My test deploy is going to consist of two nodes and one arbiter; production will have several more potential nodes. We aren’t worrying about sharding at this point, but 1.6 brings automatic sharding with it, as well, so we can enable that at a later point if we need to.

Installation

Installation is very easy. 10gen offers a yum repo, so it’s as easy as adding the repo to /etc/yum.repos.d and then running yum install mongo-stable mongo-server-stable.

Once installed, mongo --version confirms that we’re on 1.6. Time to boot up our nodes.

Configuration

For staging, we’re going to run both replica nodes and the arbiter on a single machine. This means 3 configs.

I have 3 config files in /etc/mongod/ – mongo.node1.conf, mongo.node2.conf, and mongo.arbiter.conf. As follows:

# mongo.node1.conf
replSet=my_replica_set
logpath=/var/log/mongo/mongod.node1.log
port = 27017
logappend=true
dbpath=/var/lib/mongo/node1
fork = true
rest = true

# mongo.node2.conf
replSet=my_replica_set
logpath=/var/log/mongo/mongod.node2.log
port = 27018
logappend=true
dbpath=/var/lib/mongo/node2
fork = true

# mongo.arbiter.conf
replSet=my_replica_set
logpath=/var/log/mongo/mongod.arbiter.log
port = 27019
logappend=true
dbpath=/var/lib/mongo/arbiter
fork = true
oplogSize = 1

Starting it up

Then we just fire up our daemons:

mongod -f /etc/mongod/mongo.node1.conf
mongod -f /etc/mongod/mongo.node2.conf
mongod -f /etc/mongod/mongo.arbiter.conf

Once we spin up the servers, they need a bit to allocate files and start listening. I tried to connect a bit too early, and got the following:

[root@261668-db3 mongo]# mongo
MongoDB shell version: 1.6.0
connecting to: test
Fri Aug  6 03:48:40 Error: couldn't connect to server 127.0.0.1} (anon):1137
exception: connect failed

Configuring replica set members

Once you can connect to the mongo console, and we need to set up the replica set. If you have a compliant configuration, then you can just call rs.initiate() and everything will get spun up. If you don’t, though, you’ll need to specify your initial configuration.

This is where I hit my first problem; the hostname as the system defines it didn’t resolve. This was resulting in the following:

[root@261668-db3 init.d]# mongo --port 27017
MongoDB shell version: 1.6.0
connecting to: 127.0.0.1:27017/test
> rs.initiate();
{
        "info2" : "no configuration explicitly specified -- making one",
        "errmsg" : "couldn't initiate : need members up to initiate, not ok : 261668-db3.db3.domain.com:27017",
        "ok" : 0
}

The solution, then, is to specify the members, and to use a resolvable internal name. Note that you do NOT include the arbiter’s information; you don’t want to add it to the replica set early as a full-fledged member.

> cfg = {_id: "my_replica_set", members: [{_id: 0, host: "db3:27017"}, {_id: 1, host: "db3:27018"}] }
> rs.initiate(cfg);
{
        "info" : "Config now saved locally.  Should come online in about a minute.",
        "ok" : 1
}

Bingo. We’re in business.

Configuring the replica set arbiter

If the replica set master fails, a new master is elected. To be elected, a replica master needs to have at least floor(n / 2) + 1 votes, where n is the number of active nodes in the cluster. In a paired setup, if the master were to fail, then the remaining slave wouldn’t be able to elect itself to the new master, since it would only have 1 vote. Thus, we run an arbiter, which is a special lightweight, no-data-contained node whose only job is to be a tiebreaker. It will vote with the orphaned slave and elect it to the new master, so that the slave can continue duties while the old master is offline.

> rs.addArb("db3:27019")
{
        "startupStatus" : 6,
        "errmsg" : "Received replSetInitiate - should come online shortly.",
        "ok" : 0
}

Updated driver usage

Once we’re set up, the Ruby Mongo connection code is updated to connect to a replica set rather than a single server.

Before:

MongoMapper.connection = Mongo::Connection.new("db3", 27017)

After

MongoMapper.connection = Mongo::Connection.multi([["db3", 27017], ["db3", 27018]])

This will attempt to connect to each of the defined servers, and get a list of all the visible nodes, then find the master. Since you don’t have to specify the full list, you don’t have to update your connection info each time you change the machines in the set. All it needs is at least one connectable server (even a slave) and the driver will figure out the master from there.

Conclusion

That’s about all there is to it! We’re now up and running with a replica set. We can add new slaves to the replica set, force a new master, take nodes in the cluster down, and all that jazz without impacting your app. You can even set up replica slaves in other data centers for zero-effort offsite backup. If your DB server exploded, you could point your app at the external datacenter’s node and keep running while you replace your local database server. Once your new server is up, just bring it online and re-add its node back into your replica set. Data will be transparently synched back to your local node. Once the sync is complete, you can re-elect your local node as the master, and all is well again.

Congratulations – enjoy your new replica set!

First things first. If this is TL;DR for you, here’s the demo, or click the “Click Heatmap” button in the corner of this page.

My first iteration was an endpoint in my current Rails app, which handled saving/sending of click data. That worked fine, but for something as lightweight and common as a click, I didn’t want to be invoking my full Rails stack. I’ve also been meaning to play with node.js…well hey, there’s an opportunity here!

First, I had to get the client code working. I modified the existing heatmap code into a jQuery plugin, which handled all the setup/transmission/rendering of data. This enables you to do something like so:

$("#body_wrapper").clickhax({ trigger: "#showHeatmap", endpoint: "/map" });

What that does is attach the handlers to your wrapper element, and sets up the HTML5 canvas to display over that. Events will be sent to /map (which, in this example, is ProxyPassed to my node.js daemon), and clicking an element with an id of showHeatmap causes the heatmap data to be fetched and rendered on the client. The client itself just takes a raw JSON dataset and performs smoothing and rendering with it. It’s fairly basic canvas work – the majority of the heavy lifting is non-graphical – but still, it’ll only work in browsers that support the canvas tag. Sorry, oldschool IE users.

Okay, great, that’s working, what about the backend now?

Node.js is remarkably easy to get up and running on, and with the addition of the Express package, it behaves an awful lot like Sinatra. I’m using MongoDB as my backend store for this, which is handy, since it natively speaks JSON, and there are client libraries for node. Using the npm utility, I quickly had them installed and was up and running.

You can see the code on GitHub, but I’ll touch on the key points here first.

The biggest gotcha I ran into this was in my treatment of the database connection handling. It took me a little while to recognize that the calls being made are asynchronous. This is important. This is very important. Rather than writing it top-down like a Ruby script, I had to, as you can see, use the provided callback chains. In particular, in my get(“/”) handler, I was performing the query and then immediately trying to iterate the cursor – this doesn’t work! You have to iterate in the callback. (In my defense, it was late and my brain was foggy!)

The code is pretty straightforward, though. When you post to your endpoint, it accepts x and y parameters and parses out the referring URL as the click target page. The plugin computes the click as an offset from the top left corner of your wrapped element, so if you have a fixed-width wrapper, your click data remains consistent even with differing monitor sizes. Data is quantitized to 5px before storage, and storage is done with upserts and MongoDB’s atomic increment; multiple clicks in the same 5px square will simply increment a counter in that record, rather than saving a record per click.

Positions are indicated by assuming a maximum width of 3000px. This allows us to store positions as single integers, rather than position pairs or strings. The client plugin is aware of this, and can reverse a given index into an x/y pair accordingly. The getter simply constructs a hash of {position: click_count} and sends that to the client. The client then applies a blur pattern on top of those points to generate a smoothed heightfield, and then normalizes that heightfield to the 0..255 range. Those heights are then mapped to colors and rendered onto the canvas. That’s all there is to it!

Quantitizing to 5px squares means that for my 600×400 demo, I have 9600 potential squares, and each square takes 6-11 bytes of JSON to represent. Thus, even for a fully saturated clickmap, I should only ever have to receive/compute/render 103kb worth of data. That number of obviously increases as you increase the size of the target area – 960×2000 would be a maximum of 825kb of data for a fully saturated clickmap. However, in practice, full saturation should be a non-concern. Your clicks will be focused around interactable elements, and due to the atomic increment counters, heatmaps should remain light and snappy both for inserts and fetches, regardless of the number of clicks in a page.

If you don’t already have node.js and mongodb, the setup may be a bit more involved, but you could use PHP/MySQL, or Rails with SQLite or whatever as your endpoint server. The front and back ends are relatively decoupled, and can be re-used independently of each other.

Mobile authenticators work by transforming a seed value (called the “token secret”) + the current time into your 8-digit authentication code. FlexAuth lets you set up multiple authenticators by providing the secret, or will let you have Blizzard generate one for you.

Why would you need this?

• You want to use a mobile authenticator, but don’t want to be locked out if you ever lose your phone (just setup a new token with your registered token secret).
• You want to use multiple mechanisms to log in – maybe you need token authentication in a script, or you want to have the same authenticator values on multiple mobile phones.
• You already have a token secret from another source and want to use it on your mobile phone.

Obviously, these won’t apply to most people, but some folks will definitely find it useful.

1. Menu -> Add Account
2. Enter a name for this token/account. It can be whatever you’d like.
3. Either enter a serial + secret, or you can use the already-provided one, or generate a new one.
4. Save the token. You’ll notice that auth codes start generating right away.
5. It is highly recommended that you back up your token secret. If you uninstall the app, wipe your phone, etc, then you will lose the secret, and consequently lose the ability to generate auth codes. To back up a code, click into the token’s details, and long press on the secret to copy it. You can then paste it into a note or email or whatnot. To restore a token, simply generate a new token and use your backed up secret. It will generate compatible auth codes.

All that said, a word of caution: Never ever ever run authenticator software on the same machine that you’re logging in on. It’s bad, it’s dumb, and you shouldn’t do it. Keep your authentication token generation on a separate device if you value your account.

If any particular same-machine authentication scheme gained any measure of popularity, it would be targeted by malware and your authenticator would be useless. Don’t do it.

Other than that, enjoy!

So, I sat down and fixed gridfs-fuse. You can grab my branch at GitHub. I made a few changes that make it ideal for serving files out of a GridFS DB, with a few caveats.

Installation and Configuration

Building it is relatively simple.

1. Install scons, the Python SConstruct utility (on Fedora/CentOS/RHEL, yum install scons)
2. Extract or symlink a copy of your mongodb install to /opt/mongo
3. Run scons
4. If all builds well, yay. If not, fix any missing dependencies or path issues. Edit SConstruct to change any paths that you need to.
5. Create a mount point for your GridFS filesystem; I used /mnt/gridfs (sudo mkdir /mnt/gridfs)
6. chown your mount point to your webserver’s user. If you run Apache, this is probably apache. If you run nginx, it’s probably nobody. (sudo chown nobody.nobody /mnt/gridfs)
7. Mount the database to the mount point.

   sudo -u nobody ./mount_gridfs --db=your_database --host=localhost /mnt/gridfs
   

   Change the user and db parameters as required.

8. Configure your webserver to serve files appropriately. In my case, I have carrierwave set up to write files to uploads/model/_id/filename.png, and carrierwave is configured to use /images/gfs as my base URL. This means that for a given file, I might end up with a path like /images/gfs/uploads/user/avatar/4b8475cc69e0dc57e7000005/thumb_untitled-20.png. To cause the GridFS files to be served off of the mount point, I just symlinked the mount to /images/gfs.

   cd public/images
   ln -s /mnt/gridfs gfs
   

Once that’s all set up, you should be able to use your webserver to serve images directly out of your Mongo database, and at pretty fair rates, too!

143% Unscientific Benchmarks

[chris@polaris gridfs-fuse]# ab -n 5000 -c 25 http://advice:81/images/gfs/uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/gfs/uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png
Document Length:        14332 bytes

Concurrency Level:      25
Time taken for tests:   5.029 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      72725000 bytes
HTML transferred:       71660000 bytes
Requests per second:    994.22 [#/sec] (mean)
Time per request:       25.145 [ms] (mean)
Time per request:       1.006 [ms] (mean, across all concurrent requests)
Transfer rate:          14121.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    16   25   1.4     25      52
Waiting:        2   24   1.4     24      52
Total:         17   25   1.4     25      53

Percentage of the requests served within a certain time (ms)
  50%     25
  66%     25
  75%     25
  80%     25
  90%     25
  95%     26
  98%     27
  99%     32
 100%     53 (longest request)

Caveats

To get this working, I had to hack in directory support. GridFS stores files with paths, but doesn’t store them in a hierarchy; Fuse navigates a filesystem, which is hierarchical. In order to overcome this, I made gridfs-fuse respond to directory requests as valid. For a given file, gridfs-fuse will walk the following path hierarchy:

GET /uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png
Check for uploads, directory exists
Check for uploads/user, directory exists
Check for uploads/user/avatar/, directory exists
Check for uploads/avatar/4b8347a698db740b30000057, directory exists
Check for uploads/user/avatar/4b8347a698db740b30000057/thumb_adrine-big.png, file exists, return file.

There are two things to be aware of here:

1. The deeper your path hierarchy, the more steps gridfs-fuse will take to find your file. Less directory nesting means faster file serving. The performance difference won’t be massive, but it’s there.
2. /!\ Big giant hack. /!\ gridfs-fuse assumes that any path part with a period in it is the path leaf. This is done so that we don’t have to keep querying the DB with regexes, which degrades performance by about 90% in my testing. Always, always, always make sure your filenames have a period in them, and make sure your directories do not have a period in them. This is a rather hefty set of caveats, but if you’ll stick to them, you will be rewarded with easy GridFS file serving.

What happens if I don’t follow those rules?

A few things happen. If you put periods in directory names, you’ll get 404s. They’ll be fast 404s, but they’ll be 404s. Even if a filepath is valid, like /images/foo.bar/baz/bin.png, gridfs-fuse will short-circuit at images/foo.bar, assuming that is the leaf of the hierarchy.

If you don’t put a period in your filenames, then gridfs-fuse will keep returning “yup, that’s a directory”, even when your webserver requests /images/foo.bar/baz/bin.png/index.html and then /images/foo.bar/baz/bin.png/index.html/index.html and then /images/foo.bar/baz/bin.png/index.html/index.html/index.html, and so forth. There’s a built-in stop at 10 levels deep – at 10 levels, gridfs-fuse gives up and just returns a 404, but it’ll take you a relatively long time to get there, and it’s really very highly recommended that you don’t do that.

What about when gridfs-fuse isn’t running?

Never fear, that’s easily fixed. Just use a Rack or Rails Metal middleware to serve images from GridFS. This is massively slower than serving files through gridfs-fuse, but at least your visitors won’t be treated to a site full of broken images if your mount point goes away for whatever reason. I’m using the following Metal endpoint. Just throw it into app/metals/gridfs.rb, add config.metals = ["Gridfs"] into your environment.rb, and you’re off to the races.

# rails metal to be used with carrierwave (gridfs) and MongoMapper

require 'mongo'
require 'mongo/gridfs'

# Allow the metal piece to run in isolation
require(File.dirname(__FILE__) + "/../../config/environment") unless defined?(Rails)

class Gridfs
  def self.call(env)
    if env["PATH_INFO"] =~ /^\/images\/gfs\/(.+)$/
      key = $1
      if ::GridFS::GridStore.exist?(MongoMapper.database, key)
        ::GridFS::GridStore.open(MongoMapper.database, key, 'r') do |file|
          [200, {'Content-Type' => file.content_type}, [file.read]]
        end
      else
        [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
      end
    else
      [404, {'Content-Type' => 'text/plain'}, ['File not found.']]
    end
  end
end

(I didn’t write that, but I can’t find the source to give credit at the moment).

That gives you a highly performant front-end solution with a reliable fallback. For any given request, the following should happen:

1. Your webserver attempts to load the file out of GridFS. If it can’t be found (likely due to a missing mountpoint), then…
2. The request will fall through to your Metal handler. It will then attempt to serve it from GridFS.
3. If it still can’t be found, the request falls through to your Rails app.

To prevent step 3 from happening, you might want to change line 18 of the Metal handler to return a 200 and read out a generic “missing image” image of some sort. That’ll prevent 404s from invoking a hit to your app.

Stick a CDN in front of it all, and you have a high-performance file upload solution with automatic replication and sharding that you can treat like any other piece of web data. Hooray!

There are lots of benchmarks and numbers under the cut. Keep reading!

Right now, there are three options:

1. Use a “low-level” script handler, like a Rack script or Rails Metal handler to serve them out of the database
2. Use something like gridfs-fuse to mount the database as a filesystem, and read it with the Fileserver directly
3. Use something like nginx-gridfs to talk directly to MongoDB from your webserver.

I wasn’t able to get gridfs-fuse to build on my system, but I was able to build the nginx module. The question, of course, is how fast are you going be serving files with each solution?

Filesystem read through Apache

First, I’ll establish a baseline. I’m running Apache as my frontend server, and we’ll use ab to benchmark its throughput.

[chris@polaris conf]# ab -n 50000 -c 10 http://advice/images/embed/alliance-60.png

Server Software:        Apache/2.2.13
Server Hostname:        advice
Server Port:            80

Document Path:          /images/embed/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   1.904 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      159463760 bytes
HTML transferred:       158043192 bytes
Requests per second:    2625.37 [#/sec] (mean)
Time per request:       3.809 [ms] (mean)
Time per request:       0.381 [ms] (mean, across all concurrent requests)
Transfer rate:          81767.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.4      1       4
Processing:     1    3   0.5      3       6
Waiting:        0    1   0.4      1       4
Total:          2    4   0.4      4       8

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      5
  99%      5
 100%      8 (longest request)

Nice and fast, like like we’d expect.

Filesystem read through nginx

[chris@polaris conf]# ab -n 50000 -c 10 http://advice:81/images/embed/normal_alliance-60.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/embed/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   7.623 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      1590513618 bytes
HTML transferred:       1579863192 bytes
Requests per second:    6559.31 [#/sec] (mean)
Time per request:       1.525 [ms] (mean)
Time per request:       0.152 [ms] (mean, across all concurrent requests)
Transfer rate:          203763.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       9
Processing:     1    1   0.4      1      11
Waiting:        0    0   0.1      0       9
Total:          1    1   0.5      1      12

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      2
  90%      2
  95%      2
  98%      3
  99%      3
 100%     12 (longest request)

nginx screams. At 6500 requests/sec, it’s blisteringly fast.

GridFS read through nginx-gridfs

[chris@polaris conf]# ab -n 5000 -c 10 http://advice:81/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   4.613 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      158580000 bytes
HTML transferred:       157980000 bytes
Requests per second:    1083.88 [#/sec] (mean)
Time per request:       9.226 [ms] (mean)
Time per request:       0.923 [ms] (mean, across all concurrent requests)
Transfer rate:          33570.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     1    9   4.7      9     103
Waiting:        1    9   4.7      9     102
Total:          2    9   4.7      9     103

Percentage of the requests served within a certain time (ms)
  50%      9
  66%      9
  75%      9
  80%      9
  90%      9
  95%      9
  98%      9
  99%     11
 100%    103 (longest request)

Definitely a lot slower, but still very respectable. 1051 requests/sec is going to be more than adequate for most purposes, particularly if fronted with a CDN.

And finally…

Rails Metal handler

The nice thing about the Rails metal handler solution is that it’s easy. No recompiling, just drop the handler into your project and you’re off to the races. That said…

[chris@polaris nginx-gridfs]$ ab -n 250 -c 4  http://advice/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png

Server Software:        Apache/2.2.13
Server Hostname:        advice
Server Port:            80

Document Path:          /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      4
Time taken for tests:   4.646 seconds
Complete requests:      250
Failed requests:        0
Write errors:           0
Total transferred:      7960000 bytes
HTML transferred:       7899000 bytes
Requests per second:    53.81 [#/sec] (mean)
Time per request:       74.338 [ms] (mean)
Time per request:       18.585 [ms] (mean, across all concurrent requests)
Transfer rate:          1673.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    15   74  75.6     34     287
Waiting:        0   72  75.8     30     276
Total:         15   74  75.6     34     288

Percentage of the requests served within a certain time (ms)
  50%     34
  66%     39
  75%    139
  80%    192
  90%    201
  95%    210
  98%    239
  99%    245
 100%    288 (longest request)

I obviously ran far fewer requests this go-round. The reason is pretty obvious – running 5000 requests through the Ruby stack would have taken approximately forever. At 53 requests per second, this is not an attractive solution, particularly if you consider the CPU overhead that it’s incurring.

Conclusions

If you’re looking to abstract away from storing files on a filesystem, GridFS is a feasable solution. It can really crank some mean output numbers, and though it’s not up to par with a raw filesystem read, also consider that in many production environments, such a raw filesystem read might be happening via an NFS or GFS share, which is going to massively degrade the performance of that request. Given the no-hassle store-and-forget-about-it solution that GridFS offers, even when faced with the challenge of multi-server replication, it seems that you can get enough performance out of it to justify it as a solution.