Here’s the call graph for #strip_tags (as profiled in an application I’m working on). As you can see, it tokenizes the entire string, and then iterates the tokens, likely pushing and popping sections onto and off of a stack as tags are opened and closed.

This is a lot more than a quick little regex to strip out tags; it’s actually parsing the full HTML document. Fortunately, there are already tools to do that, and they have their slow parts written as C extensions. Nokogiri is my weapon of choice in this regard – it’s battle-tested and generally rocks at parsing markup, even when it’s poorly-formed.

So, let’s benchmark a “strip all the markup out of a string” use case with #strip_tags and nokogiri.

require 'rubygems'
require 'action_view'
require 'nokogiri'

include ActionView::Helpers::SanitizeHelper

f = open("news").read

LOOPS = 1000
Benchmark.bmbm do |x|
  x.report("strip_tags") { LOOPS.times { strip_tags f }}
  x.report("nokogiri") { LOOPS.times { Nokogiri::HTML(f).text }}
end

The data file in this case is a snapshot of the current page of Hacker News. It’s a 23kb HTML file. Nothing too huge, but certainly not small, either. Let’s run it through the benchmark:

[chris@luna projects]$ ruby strip.rb
Rehearsal ----------------------------------------------
strip_tags  33.070000   0.010000  33.080000 ( 33.092638)
nokogiri     3.220000   0.020000   3.240000 (  3.241090)
------------------------------------ total: 36.320000sec

                 user     system      total        real
strip_tags  33.010000   0.010000  33.020000 ( 33.056551)
nokogiri     3.190000   0.000000   3.190000 (  3.200680)

Yikes. It’s not just slower, it’s ~10x slower.

Don’t use strip_tags. Also, profile your code. But just because it’s convenient doesn’t mean you should use it.

Turns out there’s a pretty easy fix in the form of a couple of module parameters:

In /etc/defaults/grub, add the following kernel parameters:

video.brightness_switch_enabled=1 video.use_bios_initial_backlight=0

(You may also want to add radeon.modeset=1 and acpi_osi=Linux for this particular machine, but they aren’t related to the brightness fix.)

Then update your grub2 config:

 grub2-mkconfig > /boot/grub2/grub.cfg 

Reboot, and your brightness controls should work as expected. The brightness slider in GNOME still doesn’t work, but I’m content with hardware brightness controls over no brightness controls.

However, PixelPerfect doesn’t work all that well (well, at all) anymore, and Firefox is no longer my browser of choice. There weren’t any good options for Chrome, so I wrote one.

You can grab it straight from the Chrome store if you want (it’s free!), or you might be interested in perusing the source code.

Comps is very lightweight, and gets straight to the point – click the button, drop your comp into the webpage, and position it to make direct comparisons. Mousewheel over the comp to change its opacity, or you can toggle it on and off using the thumbnail drawer or the Comps button. It’ll remember your settings between sessions (and even pageloads!) so that you can tweak and refresh to your heart’s content, and instantly have your comp right there for comparison.

Give it a shot, let me know if/how you like it.

Fortunately, Ruby is deeply entangled with another more portable serialization format: YAML.

Rails manages its session cookies through the MessageVerifier. Easy enough – we can just write our own MessageVerifier that uses YAML rather than Marshal.

module ActiveSupport
  class YamlMessageVerifier < MessageVerifier
    def verify(signed_message)
      raise InvalidSignature if signed_message.blank?

      data, digest = signed_message.split("--")
      if data.present? && digest.present? && secure_compare(digest, generate_digest(data))
        str = ActiveSupport::Base64.decode64(data)
        if str[0..2] == '---'
          YAML::load str
        else # Handle old Marshal.dump'd session
          Marshal.load(str)
        end
      else
        raise InvalidSignature
      end
    end

    def generate(value)
      data = ActiveSupport::Base64.encode64s(YAML::dump value)
      "#{data}--#{generate_digest(data)}"
    end
  end
end

You’ll notice that verify() can accept a Marshaled session as well; this lets you transparently transition existing cookies to the new format without any kind of session breakage. Nice.

Now, to use the verifier, we monkeypatch CookieStore:

module ActionController
  module Session
    class CookieStore
      def verifier_for(secret, digest)
        key = secret.respond_to?(:call) ? secret.call : secret
        ActiveSupport::YamlMessageVerifier.new(key, digest)
      end
    end
  end
end

Now, this will work…at least at first glance, until you try to use the flash. This is a particularly nasty little problem, and it stems from the fact that Ruby’s YAML implementation serializes Hash objects without their instance variables, and FlashHash inherits from Hash, and thus inherits its serialization/deserialization strategy. I worked for a while to monkeypatch those strategies, but I didn’t like the result, and it felt a little hacky. Instead, I just took advantage of the YAML load lifecycle to make sure the FlashHash initializes properly:

module ActionController
  module Flash
    class FlashHash
      def update_with_initializer(h)
        @used ||= {}
        update_without_initializer(h)
      end
      alias_method_chain :update, :initializer
    end
  end
end

The core problem is that YAML::load calls Hash#update, and FlashHash presumes that the @used instance variable is present and initialized to an empty hash. To fix that, I just aliased in an initializer to make sure that variable is set.

Note that if you are storing other Hash subclasses with instance variables that rely on those variables being persisted across sessions, they will break. However, you should only be storing primitive/array/hash data in the session if possible. FlashHash is sort of a nasty violation of this principle.

At this point, your session should be serializing to and from YAML. We’ll want to read it from PHP, naturally. I’m using SPYC in the PHP project, which gets us Close Enough(TM). It doesn’t handle symbol keys, but we’ll handle those in the PHP itself.

Reading from PHP

Reading the data back out is surprisingly simple. We have to verify the authenticity of the data, of course, by checking the hash, but then you just base64 decode the data, load it with spyc, and perform some simple transformation to turn symbols into strings. If you wanted to make it even easier, you could monkeypatch the cookie store to call #stringify_keys! on your session hash before serializing it (and then call #with_indifferent_access on the hash when you deserialize it. Be aware of the speed impact of such a decision before you do it.)

function explode_symbols($arr) {
  $result = array();
  foreach($arr as $key => $val) {
    if(is_numeric($key) && $val[0] == ":") {
      $bits = explode(":", $val, 3);
      $result[trim($bits[1])] = trim($bits[2]);
    } elseif (is_array($val)) {
      $result[$key] = explode_symbols($val);
    } else {
      $result[$key] = $val;
    }
  }
  return $result;
}

function deserialize_session($session_key, $secret) {
  list($session64, $hash) = explode("--", $_COOKIE[$session_key], 2);
  if(hash_hmac("SHA1", $session64, $secret) == $hash) {
    $session = base64_decode($session64);
    return explode_symbols(spyc_load($session));
  } else {
    throw new Exception("Invalid session signature");
  }
}

$rails_session = deserialize_session("your_session_cookie_name", $your_session_cookie_secret);

Caveats

• Be aware that YAML is slower than Marshal
• Be aware that storing Hash subclasses in the session is likely going to Not Work.

And that’s all there is to it. You can now share data between the two apps via the session cookie.

check file resque-restart.txt with path /path/to/your/app/tmp/resque-restart.txt
  if changed timestamp then
    exec "/usr/bin/monit restart resque-worker"

Ties in nicely with deploy tasks, and you don’t have to end up leaving root access SSH keypairs laying around.

In practice, I’ve found that this costs about 1ms per 1000 records in your counted result set. This is really bad in concert with will_paginate, which Plucky (which is used by MongoMapper) exposes an interface to. It naively takes your query, performs a count() on it, and then performs the query again with limiters to get the records for the current page. This is a standard and quickly-accepted way to do this sort of thing.

NewRelic is a great tool to help profile your applications, and in this case, it’s making the problem abundantly clear:

You see that purple? That’s how long it takes to run those count() operations. What a big fat pile of suck.

I don’t have a good solution to this yet, but in the meantime, I’ve mokneypatched Plucky to cache counts for large result sets. This means that my total counts for a large collection might desync over the course of an hour, but in my use cases, I only need ballpark numbers, so it works out well. This has a very noticeable effect on page times, effectively halving the amount of time I spend in the database for a given index page. Additionally, I can manually specify a count. So, for example, if I know a collection will have over 10k results, I can just pass 10k, and stop paginating after 10k results, drastically reducing my DB load at the expense of exposing older or long-tail content (which may be perfectly, appropriate, depending on the application context).

What I’m doing is caching any counts over some arbitrary limit (I chose 10k, at which point the counts would take ~10ms) for an hour via the Rails cache (memcached, in my case, leveraging the expires_in parameter). I brought the issue up in the #mongodb IRC channel, and the advice I was given was basically “cache your counts”, which is all well and good for simple data sets, but when I’m building pages per-user based on their preferences and myriad inputs (all indexed, mind you), it just doesn’t work, so I’ve resorted to this. It’s a hack, but it’s gotten my page times down substantially.

module Plucky
  class Query
    BIG_RESULT_SET = 10000

    def paginate(opts={})
      page          = opts.delete(:page)
      limit         = opts.delete(:per_page) || per_page
      query         = clone.update(opts)
      cache_key     = "count-cache-#{criteria.source.hash}"
      total         = opts.delete(:total) || Rails.cache.read(cache_key)
      if total.nil?
        total       = query.count
        if total > BIG_RESULT_SET
          Rails.cache.write(cache_key, total, :expires_in => 1.hour)
        end
      end
      paginator     = Pagination::Paginator.new(total, page, limit)
      query[:limit] = paginator.limit
      query[:skip]  = paginator.skip
      query.all.tap do |docs|
        docs.extend(Pagination::Decorator)
        docs.paginator(paginator)
      end
    end
  end
end

I’m not entirely happy with this solution, and would love input on ways to fix it.

I couldn’t find a quick and easy answer to this with some Googling, but it turns out that the answer is fortunately rather simple.

In your test_helper.rb:

 def setup
    Resque.redis.select 1
  end

  def teardown
    Resque.redis.keys("queue:*").each {|key| Resque.redis.del key }
  end

That’s all there is to it. The setup causes Resque to write to database #1 (#0 is default, and is what your development environment is likely using), and the teardown just deletes all your queues (which are really just lists of jobs to run). Test all you want and you won’t have to worry about tens of thousands of jobs junking up your redis DB.

Wart #1: Case sensitivity

All data in MongoDB is case-sensitive. This is in stark contrast to something like MySQL, where indexed text columns are case-insensitive. So, if you have a “username” field, “Chris” and “chris” could be two different users, and a user trying to log in as “chris” by tying “CHRIS” into the username field would fail their login attempt. You can solve this by either a) forcing a consistent casing (lower or upper) on the column, or b) by keeping a second shadow column with normalized (and indexed) data. So, for example, I might need to keep two columns, username and shadow_username, then index and do all queries against shadow_username, but display username. This isn’t a huge wart, but it’s going to bite you in the ass if you aren’t used to it.

Wart #2: Aggregate queries

SQL databases do aggregate queries really well. Consider a use case I had recently: I needed to find all accounts with duplicate emails.

In SQL, this is:

SELECT email, count(id) as ct FROM users HAVING ct > 1 GROUP BY email;

In MongoDB, you have to do this with a map/reduce/finalize:

var map = function() {
  if(this.email) {
    emit(this.email.toLowerCase(), {count: 1})
  }
}

var reduce = function(key, values) {
  var r = {count: 0};
  for(var i=0; i<values.length; i++) {
    r.count += values[i].count;
  }
  return r;
}

db.users.mapReduce(map, reduce, {out: 'users.group_by_email'})
db.users.group_by_email.find({"value.count": {$gt: 1}})

There are two warts here – the first is that as of MongoDB 1.7.4, you no longer have the option for temporary collections in MongoDB, and so you have to either A) write the whole result to a new collection, or B) return the whole result as a single document, but limited to 16MB of data. This means more maintenance – cleaning up old collections, notably. Secondly, it’s obviously a lot more code. The fact that M/R queries are written in Javascript is cool, but it’s kind of a pain in the ass to crank out a one-liner. Fortunately, because you are encouraged to store denormalized data in MongoDB, you don’t have to do as many of these sorts of queries, but when you do, it’s noticeably more painful than it would be with SQL.

Wart #3: Timestamp sorting

This is more of a bug than a design flaw, but it’s a nasty bug. MongoDB stores dates as unsigned long long, which is great, right? It gives you seriously far-future dates!

Until you want to sort a resultset by date that includes dates before 1970, that is (like, say, birthdays).

Since the storage type is unsigned, dates before the UNIX epoch get stored as very, very large numbers rather than as small negative numbers. You don’t notice this when you’re just querying data, but if you try to sort on a date column, any dates before 1970 will appear as being later than your other “normal” dates.

You might want to find all users who are more than 60 years old:

db.users.find({birthday: {$lt: [Date Object for 60 years ago]}})

You’ll get an empty result set, because 60 years ago, in UNIX time, is -582954786, but you won’t ever have any results with a birthday indexed with a value of less than 0.

Likewise, if you want to get all users younger than a certain date:

db.users.find({birthday: {$gt: [Date Object for 13 years ago]}})

You will get all users who were born before 1970 in this result set, since the index is simply looking at numeric long ranges, and pre-1970 dates will index very far future dates instead. This is fixed as of July 6th, but won’t make it into production until Mongodb 1.10.

Workarounds:

* Map/Reduce users’ ages into a separate collection periodically (once a month, perhaps?), then query off of that new collection.
* Add some arbitrary amount of time to all birthdays in your application logic. For example, for birthdays, store the given birthday + 120 years, and then subtract 120 years before doing any calculations app-side with the birthday. This is an ugly hack, but requires no m/r maintenance.

Wart #4: Arbitrary Javascript operations obtain a global lock

Yesterday, I needed to fix a data problem; emails needed to be unique in my DB, but to check them, I needed a normalized lowercase email to query for. To do this, I had a simple bit of Javascript to execute:

db.users.find().forEach(function(obj) {
  if(obj.email) {
    obj.email = obj.email.toLowerCase();
    db.users.save(obj);
  }
});

Easy enough. Iterate each record, lowercase the email, save the record. Except when you give it a few hundred thousand records, with a bunch of indexes on the collection, it’s slooooow. My first naive crack at this locked my MongoDB master for several minutes, and didn’t complete before I killed it. During that time, the app was completely unresponsive. Oops.

I was able to rewrite the migration to be a lot faster, but you have to take a lot of care when running arbitrary data migrations, since you really can shoot yourself in the face easily with it.

db.users.find().forEach(function(obj) {
  if(obj.email) {
    var email = obj.email.toLowerCase();
    if(email != obj.email) {
      db.users.update({_id: obj._id}, {$set: {email: email}});
    }
  }
});

By a) only updating the record if the email changed, and b) using the atomic $set rather than updating the whole record, my migration ran in less than a second, rather than locking the entire application for minutes on end.

Be wary of that global lock. The official docs warn you about it, but the implications can’t be understated. Improvements to it are coming, but you’re going to nutpunch yourself with it eventually if you aren’t careful.

Wart #5: $or queries and index hints

MongoDB supports the $or operator for easy query unions, which makes life nice a lot of respects. However, it completely jacks up the query optimizer if you introduce sorting. What happens is that the query optimizer decides to use the sort field for the index, and results in a full table scan for each of your $or queries! Consider the following:

db.videos.find({$or: [{tags: {$all: ['b', 'a']}}, {tags: {$all: ['c', 'd']}}]})

This will find the union of all documents that have tags “b” and “a” OR “c” and “d”. The tags index is used per subquery, resulting in a fast query.

But if you want to sort the results…

db.videos.find({$or: [{tags: {$all: ['b', 'a']}}, {tags: {$all: ['c', 'd']}}]}).sort({title: -1})

MongoDB ignores the tags index for each of your $or clauses, and instead chooses to use the title index. This means that each of your $or clauses invokes a full table scan to find the tag matches, resulting in an extremely slow query. There is no way, as of right now, to tell the query optimizer to use the tags index when also using a sort on the cursor. Oops.

The accepted solution, right now, is to perform an in-app sort of the result set, which is a giant pain in the ass if you can’t prune the unsorted resultset to a reasonable size before sending it to the app before sorting. If in-app sorting isn’t an option for whatever reason, you’ll have to restructure your data to avoid the $or clause.

A wobble: Document size

One of MongoDB’s strengths can also be a weakness, if you’re not careful. Because you can store so much denormalized data in a single record, you can drastically reduce your number of queries, and build pages faster. However, it’s easy to forget that when you store all that data, you have to move it over the wire. Consider something like the following document:

topic: {name: 'Topic', description: 'A bit about this topic', followers: [array of follower BSON IDs]}

This will work just fine in development. But what about in production, when that popular topic has 85,000 followers? All of a sudden, that’s 1,020,000 bytes that have to be sent over the wire every time you query that topic. What this really is is the old SELECT * problem, but magnified 100x. It’s ridiculously easy to accidentally end up with pages that are pulling tens of megabytes of data from the database server for every request, which does not scale well at all. Be judicious in your use of the field selection parameter when querying your database – your app will thank you.

To omit the followers array, I just omit the fields I don’t want in the query:

db.topics.find({name: 'Topic'}, {followers: false})

Just a few little tweaks like this realized massive performance gains in my app, once I wasn’t moving tens of megabytes and instantiating thousands of BSON::ObjectId objects per record.

Wrapping Up

Despite the warts, none of these are reasons to decide to not use MongoDB. They introduce more work for you, and will make you pull your hair out in frustration if you naively wander into one of them by mistake, but if you’re aware of them, you can avoid them, and get all the good parts without having to taste too much of the bad. Like any piece of software, MongoDB has quirks and irritations, but if you aren’t buying into the hype that it’s a magical web-scale fix-all that mysteriously makes database and query design a non-issue, they aren’t that big of a deal. After all, now you know, and…

After continually slimming down the code, I realized that even though it’s tiny, it’s danged useful to be able to just drop this into a Rails app and go. Thus, I’d like to present Tarot, my Rails configuration solution.

Tarot’s current form is heavily inspired by the Rails I18n usage, and is very quick and easy to use in your app. The generator installs a sample yaml file at config/tarot.yml, as well as an initializer to bootstrap your configuration and provide a handy helper method for quick access to those config values.

Assuming you have a config file like so:

---
base: &base
  foo: bar
  nested:
    tree: value
  array:
    - value 1
    - value 2

development: &development
  <<: *base

test: &test
  <<: *base

production: &production
  <<: *base
  foo: baz

You’ll notice that all the environments inherit from your base environment; this gives you an easy way to define common settings once, then override them per environment. Handy!

You could can access values by key, or by dot-delimited path:

config('foo') => 'bar'
config('nested.tree') => value

Default values are similarly easy.

config('foo.missing', 42) => 42

Finally, while Tarot will read your current application environment’s config, if you want to reach into another environment, that’s likewise easy:

config('foo', nil, 'production') => 'baz'

As of 0.1.2, Tarot also supports method_missing invocation:

Config = Tarot::Config.new('settings.yml', Rails.env)
Config.foo.bar.baz => "bin"

It also supports default values:

# Assuming foo has no subkey bar
Config.foo.bar("default") => "default"

But it’ll fail if you try to invoke method_missing on a non-leaf node

# Assuming that there is no `blaze` tree
Config.blaze.blarg => NameError

That’s about all there is to it — config isn’t (or shouldn’t be) a hard problem, so there’s not a whole lot to it, but it should get you up and running with easily-configured Rails apps in seconds.

If you’re just here for the code, here’s the quick and dirty. I grabbed this from Beautiful Pixels and adapted it to my needs.

::-webkit-scrollbar {
  width: 13px;
  height: 13px; }

::-webkit-scrollbar:hover {
  height: 18px; }

::-webkit-scrollbar-button:start:decrement,
::-webkit-scrollbar-button:end:increment {
  height: 15px;
  width: 13px;
  display: block;
  background: #101211;
  background-repeat: no-repeat; }

::-webkit-scrollbar-button:horizontal:decrement {
  background-image: url(scrollbar/horizontal-decrement-arrow.png);
  background-position: 4px 3px; }

::-webkit-scrollbar-button:horizontal:increment {
  background-image: url(scrollbar/horizontal-increment-arrow.png);
  background-position: 3px 3px; }

::-webkit-scrollbar-button:vertical:decrement {
  background-image: url(scrollbar/vertical-decrement-arrow.png);
  background-position: 3px 4px; }

::-webkit-scrollbar-button:vertical:increment {
  background-image: url(scrollbar/vertical-increment-arrow.png);
  background-position: 3px 4px; }

::-webkit-scrollbar-button:horizontal:decrement:active {
  background-image: url(scrollbar/horizontal-decrement-arrow-active.png); }

::-webkit-scrollbar-button:horizontal:increment:active {
  background-image: url(scrollbar/horizontal-increment-arrow-active.png); }

::-webkit-scrollbar-button:vertical:decrement:active {
  background-image: url(scrollbar/vertical-decrement-arrow-active.png); }

::-webkit-scrollbar-button:vertical:increment:active {
  background-image: url(scrollbar/vertical-increment-arrow-active.png); }

::-webkit-scrollbar-track-piece {
  background-color: #151716; }

::-webkit-scrollbar-thumb:vertical {
  height: 50px;
  background: -webkit-gradient(linear, left top, right top, color-stop(0%, #4d4d4d), color-stop(100%, #333333));
  border: 1px solid #0d0d0d;
  border-top: 1px solid #666666;
  border-left: 1px solid #666666; }

::-webkit-scrollbar-thumb:horizontal {
  width: 50px;
  background: -webkit-gradient(linear, left top, left bottom, color-stop(0%, #4d4d4d), color-stop(100%, #333333));
  border: 1px solid #1f1f1f;
  border-top: 1px solid #666666;
  border-left: 1px solid #666666; }

::-webkit-scrollbar defines the height and width of your scrollbars for horizontal and vertical scroll bars, respectively.

::-webkit-scrollbar-button:start:decrement and ::-webkit-scrollbar-button:end:increment style the arrows on either end of the scroll bar. I have mine set to 1px black bars, but you could add an image, or a gradient, or whatnot.

::-webkit-scrollbar-track-piece sets styles for the background of the scrollbar (called the “track” here)

::-webkit-scrollbar-thumb:vertical and ::-webkit-scrollbar-thumb:horizontal set styles for the draggable portion of the scroll bar. Background images, gradients, round corners, and even box shadows are all valid here. Go nuts.

With just those pieces, you should be able to crank out awesome-looking theme-fitting scroll bars. Have fun with it!