Turns out there’s a pretty easy fix in the form of a couple of module parameters:

In /etc/defaults/grub, add the following kernel parameters:

video.brightness_switch_enabled=1 video.use_bios_initial_backlight=0

(You may also want to add radeon.modeset=1 and acpi_osi=Linux for this particular machine, but they aren’t related to the brightness fix.)

Then update your grub2 config:

 grub2-mkconfig > /boot/grub2/grub.cfg 

Reboot, and your brightness controls should work as expected. The brightness slider in GNOME still doesn’t work, but I’m content with hardware brightness controls over no brightness controls.

However, PixelPerfect doesn’t work all that well (well, at all) anymore, and Firefox is no longer my browser of choice. There weren’t any good options for Chrome, so I wrote one.

You can grab it straight from the Chrome store if you want (it’s free!), or you might be interested in perusing the source code.

Comps is very lightweight, and gets straight to the point – click the button, drop your comp into the webpage, and position it to make direct comparisons. Mousewheel over the comp to change its opacity, or you can toggle it on and off using the thumbnail drawer or the Comps button. It’ll remember your settings between sessions (and even pageloads!) so that you can tweak and refresh to your heart’s content, and instantly have your comp right there for comparison.

Give it a shot, let me know if/how you like it.

check file resque-restart.txt with path /path/to/your/app/tmp/resque-restart.txt
  if changed timestamp then
    exec "/usr/bin/monit restart resque-worker"

Ties in nicely with deploy tasks, and you don’t have to end up leaving root access SSH keypairs laying around.

This place is cool. Very cool. The atmosphere is invigorating – it’s exactly the sort of place I’d love to really put in some serious time at. Smart people, lots of technology, and a laid-back, fun, productive atmosphere. Awesome.

It also highlights fairly starkly the fact that I don’t have much in the way of “hack” projects to work on. Most all my stuff these days is Real Work(TM), which, good as it is, isn’t the sort of thing that you can really unwind with. I need a project that I can work on and let it fail – something I can sink some serious creative juices into. I want to break out of what I’ve been working on for 90 hours per week for the last couple of years, and sink my teeth into something truly creative.

I’ve got a lot of directions I could go – I have a few webapps in mind, some node.js ideas, a few experiments with MongoDB. I want to do more Android development and get better with Java; I want to learn Clojure and maybe even poke at Haskell. I want to find an excuse to use Amazon’s compute clusters to solve some ridiculously huge-scale problem. More than that, I want to find out what I don’t know about yet, and put some effort towards learning it.

I don’t have a good answer for what I want to do tonight, but it’s a good starting point. I’m excited at the opportunity and sheer energy this place provides – a place to work, to be creative, to talk and play and laugh and hack and enjoy. I’ll find something interesting to do yet.

I recently did a Windows 7 x64 reinstall, and after doing so, Netflix wouldn’t play in any of my clients – Windows Media Center, Chrome, IE, you name it. After various solutions (DRM reset, security component upgrade, Silverlight uninstall and reinstall), it turns out the solution is stupidly easy:

Run your browser as an Administrator. Open Netflix, start watching a movie. Once it successfully starts playing, you can close your browser and then resume watching whatever you’d like to watch.

I can only guess that some initial setup doesn’t get done properly, and it fails if you’re trying to do the initial license acquisition in a non-elevated program. Hopefully this saves someone else a headache!

Mobile authenticators work by transforming a seed value (called the “token secret”) + the current time into your 8-digit authentication code. FlexAuth lets you set up multiple authenticators by providing the secret, or will let you have Blizzard generate one for you.

Why would you need this?

• You want to use a mobile authenticator, but don’t want to be locked out if you ever lose your phone (just setup a new token with your registered token secret).
• You want to use multiple mechanisms to log in – maybe you need token authentication in a script, or you want to have the same authenticator values on multiple mobile phones.
• You already have a token secret from another source and want to use it on your mobile phone.

Obviously, these won’t apply to most people, but some folks will definitely find it useful.

1. Menu -> Add Account
2. Enter a name for this token/account. It can be whatever you’d like.
3. Either enter a serial + secret, or you can use the already-provided one, or generate a new one.
4. Save the token. You’ll notice that auth codes start generating right away.
5. It is highly recommended that you back up your token secret. If you uninstall the app, wipe your phone, etc, then you will lose the secret, and consequently lose the ability to generate auth codes. To back up a code, click into the token’s details, and long press on the secret to copy it. You can then paste it into a note or email or whatnot. To restore a token, simply generate a new token and use your backed up secret. It will generate compatible auth codes.

All that said, a word of caution: Never ever ever run authenticator software on the same machine that you’re logging in on. It’s bad, it’s dumb, and you shouldn’t do it. Keep your authentication token generation on a separate device if you value your account.

If any particular same-machine authentication scheme gained any measure of popularity, it would be targeted by malware and your authenticator would be useless. Don’t do it.

Other than that, enjoy!

There are lots of benchmarks and numbers under the cut. Keep reading!

Right now, there are three options:

1. Use a “low-level” script handler, like a Rack script or Rails Metal handler to serve them out of the database
2. Use something like gridfs-fuse to mount the database as a filesystem, and read it with the Fileserver directly
3. Use something like nginx-gridfs to talk directly to MongoDB from your webserver.

I wasn’t able to get gridfs-fuse to build on my system, but I was able to build the nginx module. The question, of course, is how fast are you going be serving files with each solution?

Filesystem read through Apache

First, I’ll establish a baseline. I’m running Apache as my frontend server, and we’ll use ab to benchmark its throughput.

[chris@polaris conf]# ab -n 50000 -c 10 http://advice/images/embed/alliance-60.png

Server Software:        Apache/2.2.13
Server Hostname:        advice
Server Port:            80

Document Path:          /images/embed/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   1.904 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      159463760 bytes
HTML transferred:       158043192 bytes
Requests per second:    2625.37 [#/sec] (mean)
Time per request:       3.809 [ms] (mean)
Time per request:       0.381 [ms] (mean, across all concurrent requests)
Transfer rate:          81767.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.4      1       4
Processing:     1    3   0.5      3       6
Waiting:        0    1   0.4      1       4
Total:          2    4   0.4      4       8

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      5
  99%      5
 100%      8 (longest request)

Nice and fast, like like we’d expect.

Filesystem read through nginx

[chris@polaris conf]# ab -n 50000 -c 10 http://advice:81/images/embed/normal_alliance-60.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/embed/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   7.623 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      1590513618 bytes
HTML transferred:       1579863192 bytes
Requests per second:    6559.31 [#/sec] (mean)
Time per request:       1.525 [ms] (mean)
Time per request:       0.152 [ms] (mean, across all concurrent requests)
Transfer rate:          203763.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       9
Processing:     1    1   0.4      1      11
Waiting:        0    0   0.1      0       9
Total:          1    1   0.5      1      12

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      2
  90%      2
  95%      2
  98%      3
  99%      3
 100%     12 (longest request)

nginx screams. At 6500 requests/sec, it’s blisteringly fast.

GridFS read through nginx-gridfs

[chris@polaris conf]# ab -n 5000 -c 10 http://advice:81/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png

Server Software:        nginx/0.8.33
Server Hostname:        advice
Server Port:            81

Document Path:          /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      10
Time taken for tests:   4.613 seconds
Complete requests:      5000
Failed requests:        0
Write errors:           0
Total transferred:      158580000 bytes
HTML transferred:       157980000 bytes
Requests per second:    1083.88 [#/sec] (mean)
Time per request:       9.226 [ms] (mean)
Time per request:       0.923 [ms] (mean, across all concurrent requests)
Transfer rate:          33570.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       1
Processing:     1    9   4.7      9     103
Waiting:        1    9   4.7      9     102
Total:          2    9   4.7      9     103

Percentage of the requests served within a certain time (ms)
  50%      9
  66%      9
  75%      9
  80%      9
  90%      9
  95%      9
  98%      9
  99%     11
 100%    103 (longest request)

Definitely a lot slower, but still very respectable. 1051 requests/sec is going to be more than adequate for most purposes, particularly if fronted with a CDN.

And finally…

Rails Metal handler

The nice thing about the Rails metal handler solution is that it’s easy. No recompiling, just drop the handler into your project and you’re off to the races. That said…

[chris@polaris nginx-gridfs]$ ab -n 250 -c 4  http://advice/images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png

Server Software:        Apache/2.2.13
Server Hostname:        advice
Server Port:            80

Document Path:          /images/gfs/uploads/user/avatar/4b7b2c0e98db7475fc000003/normal_alliance-60.png
Document Length:        31596 bytes

Concurrency Level:      4
Time taken for tests:   4.646 seconds
Complete requests:      250
Failed requests:        0
Write errors:           0
Total transferred:      7960000 bytes
HTML transferred:       7899000 bytes
Requests per second:    53.81 [#/sec] (mean)
Time per request:       74.338 [ms] (mean)
Time per request:       18.585 [ms] (mean, across all concurrent requests)
Transfer rate:          1673.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:    15   74  75.6     34     287
Waiting:        0   72  75.8     30     276
Total:         15   74  75.6     34     288

Percentage of the requests served within a certain time (ms)
  50%     34
  66%     39
  75%    139
  80%    192
  90%    201
  95%    210
  98%    239
  99%    245
 100%    288 (longest request)

I obviously ran far fewer requests this go-round. The reason is pretty obvious – running 5000 requests through the Ruby stack would have taken approximately forever. At 53 requests per second, this is not an attractive solution, particularly if you consider the CPU overhead that it’s incurring.

Conclusions

If you’re looking to abstract away from storing files on a filesystem, GridFS is a feasable solution. It can really crank some mean output numbers, and though it’s not up to par with a raw filesystem read, also consider that in many production environments, such a raw filesystem read might be happening via an NFS or GFS share, which is going to massively degrade the performance of that request. Given the no-hassle store-and-forget-about-it solution that GridFS offers, even when faced with the challenge of multi-server replication, it seems that you can get enough performance out of it to justify it as a solution.

First, the boring documentation

All the juicy information is available in the documentation, but I’m going to just go over the key points real quick.

RUBY_HEAP_MIN_SLOTS: This is the number of “heap slots” that each Ruby instance starts up with. One heap slot can hold one Ruby object. By default, this is 10,000. By controlling this value, we can get our apps to stabilize very quickly. More on this later.

RUBY_HEAP_SLOTS_INCREMENT: Once Ruby has allocated RUBY_HEAP_MIN_SLOTS objects on its first heap, it will have to allocate a second heap to make room for more. This variable controls the size of this second heap, and sets the baseline for future heaps, as well.

RUBY_HEAP_SLOTS_GROWTH_FACTOR: For heaps #3 and onward, Ruby uses RUBY_HEAP_SLOTS_INCREMENT and this value to determine the size to allocate for the new heap. By default, this is 1.8, meaning that your third heap will end up with 10,000 * 1.8 = 18,000 slots in it.

RUBY_HEAP_FREE_MIN: After each garbage collection run, if the number of free slots is less than RUBY_HEAP_FREE_MIN, a new heap will be allocated. The default is 4096.

So, let’s look at this practically. Presume that we have a Rails process that is going to require 50,000 Ruby objects before it’s fully initialized. The allocation process, when at defaults, will look something like this:

Allocate 10,000 slots (10,000 total available)
Allocate 10,000 slots (20,000 total available)
Allocate 18,000 slots (38,000 total available)
Allocate 68,400 slots (106,400 total available)

So, we end up with about 53% more slots than we actually needed, and it took us four heap allocations to even boot the process. Surely we can do better.

Enter Scrap.

Scrap is a little Metal handler I wrote for tracking memory usage and garbage statistics over an instance’s lifetime. Installing it is trivial – just drop it into your vendor directory, restart your app, and navigate to http://yoururl.com/stats/scrap.

With this in hand, we can peek our memory usage and see what we can see.

There are some stats at the top, but for our purposes, we’re interested in the per-request garbage statistics. The newest request is near the top of the file, and the oldest request is at the bottom of the file. The last 50 requests are tracked. Each request looks something like this:

[71.92 MB] GET /apps/176568-WordPress

Number of objects    : 817571 (658305 AST nodes, 80.52%)
Heap slot size       : 20
GC cycles so far     : 503
Number of heaps      : 7
Total size of objects: 15968.18 KB
Total size of heaps  : 18036.81 KB (2068.63 KB = 11.47% unused)
Leading free slots   : 27104 (529.38 KB = 2.93%)
Trailing free slots  : 1 (0.02 KB = 0.00%)
Number of contiguous groups of 16 slots: 2829 (4.90%)
Number of terminal objects: 4307 (0.47%)

Key points here for the time being are Number of objects and Number of heaps. When we look at the number of objects – in this case, 817,000, it’s obvious that we’re going to have to allocate a number of heaps to handle all those objects. Rails’ boot-up cost is fairly significant, and the default Ruby settings just really don’t cut it here. As you can see, we’ve allocated 7 heaps, and we’re using 15.9 of 18.0 MB allocated to the heap. Once a heap is allocated, it’s never de-allocated, so we’re perma-stuck at 18 MB of heap usage. Note that this isn’t the size of all the data in the program – just the space allocated for objects. A string that contains 100MB of data will only consume 20 bytes (that’s the “heap slot size – the amount of memory each object on the heap consumes”) on the heap.

However, what if we could just allocate the whole startup cost in the initial heap, and save ourselves the problems of having to reallocate so often?

We note that we have 891k slots allocated, so we can guesstimate at a number to set our initial allocation to. In my production app, I set mine to 1,250,000 – I was observing peaks around the 1,100,000 mark, and just increased it by 10% and rounded up.

So, my first custom environment variable is

RUBY_HEAP_MIN_SLOTS=1250000

And it results in something like this on the app’s first boot:

[137.99 MB] GET /movies/7505-Star-Wars-Episode-V-The-Empire-Strikes-Back

Number of objects    : 933037 (664785 AST nodes, 71.25%)
Heap slot size       : 20
GC cycles so far     : 12
Number of heaps      : 1
Total size of objects: 18223.38 KB
Total size of heaps  : 24414.08 KB (6190.70 KB = 25.36% unused)
Leading free slots   : 316963 (6190.68 KB = 25.36%)
Trailing free slots  : 0 (0.00 KB = 0.00%)
Number of contiguous groups of 16 slots: 19810 (25.36%)
Number of terminal objects: 25941 (2.08%)

Yowza, a full 25% of my heap is unused after boot. But…well, that’s okay. We’ve only allocated 1 heap, and later on, my object allocation grows to around 1,100,000. This is still 15k under the heap size, and I’ve set RUBY_HEAP_FREE_MIN=12500 (1% of the initial size), so if I have less than 12,500 heap objects free after a GC cycle, a new heap will be allocated. Stabilizing there means that I end up with 1 heap for the lifetime of my app, and I end up sitting just under the threshold that’d cause a new heap to be born. If I have a leak, or a super heavy action or something, though, that might kick me over my limit and require a new heap. So, we come to…

RUBY_HEAP_SLOTS_INCREMENT=100000

This value says “Hey, if you have to allocate a second heap, start with this many slots”. If we go over our limit of 1.25 million slots, we’ll allocate a second heap that’s about 8% the size of the original. That seems awfully small, but consider that we’re hoping to never get to that heap.

Should we end up using that entire second heap, then we have to worry about our third setting, RUBY_HEAP_SLOTS_GROWTH_FACTOR=1. This says “Each new heap should be 1.0 as large as the previous heap.” In this case, it means I’ll keep allocating 100k-slot heaps until the cows come home. In an untuned environment, this could be bad – we would either end up having to do a ton of allocations to get to our target, or we would overallocate very badly. However, because we know our app’s memory requirements, and know about where we want it to end up, a relatively small, linear growth factor is just what the doctor ordered here.

Okay, now what?

So, we have a collection of settings with which to run our app. Great! Now, how do we use it?

Fortunately, it’s easy.

pushd `which ruby | xargs dirname`
sudo vim ruby-with-env

We’re going to create a little bash script with the following:

#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=1250000
export RUBY_HEAP_SLOTS_INCREMENT=100000
export RUBY_HEAP_SLOTS_GROWTH_FACTOR=1
export RUBY_GC_MALLOC_LIMIT=30000000
export RUBY_HEAP_FREE_MIN=12500
exec "/opt/ree/bin/ruby" "$@"

Note that last line – the path will have to match the path to your Ruby executable, which fortunately, should be in the directory that you’re in.

Save it, don’t forget to chmod a+x ruby-with-env, and then edit your Apache or nginx configuration.

Under nginx, you’ll have a line like this:

passenger_ruby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby;

Just change it to use your new wrapper script, like so:

passenger_ruby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby-with-env;

The process is similarly easy for Apache – the line you need is something like:

PassengerRuby /opt/ruby-enterprise-1.8.6-20090610/bin/ruby

It might be in either your httpd.conf or conf.d/passenger.conf.

Once you’re all edited up, restart your webserver, and congratulations, you’ve got a fine-tuned garbage collector humming along with your app.

Taking out the garbage

“But Chris!”, you say, “There’s a variable in there that you didn’t talk about! What gives?” You are indeed correct, astute reader. We’ve thus far avoided the RUBY_GC_MALLOC_LIMIT variable. This is a handle little setting that lets you tell Ruby how often to clean up after itself. Ruby is written in C, and C uses malloc to allocate memory. Ruby just keeps a little counter each time it allocates an object with malloc, and it runs its garbage collector after so many malloc calls have been made. I haven’t found a great way to tune this one yet, except via experimentation, but here’s what to know about it:

1. The lower this value is, the more often your garbage collector runs. Garbage collection is slow. Garbage collection is painfully slow. If a user is waiting on garbage collection, they are going to become impatient. You want as few users waiting on garbage collection as possible.
2. The higher this value is, the more memory Ruby will allocate before it tries to clean up after itself. If this value is too high, you’ll have dead objects hanging around eating up heap space, and possibly causing Ruby to crap itself and allocate a new heap. This is bad.
3. To tune this value, you want to find the happy medium, wherein you stabilize under your initial heap allocation value, but with as few garbage collection passes as possible. Read up on Evan Weaver’s blog for some more in-depth analysis of what garbage collection frequency tuning can do to your app’s performance.
4. If you have excess memory and want a faster app, err on the side of this being too high. If you are on a tight memory budget, and would prefer slower actions in exchange for not blowing your heap and allocating a whole new one, err on the side of this being too low.
5. Recommended values for this are all over the board. Evan recommends a setting of 50 million. I’m using a setting of 30 million. The Ruby default is 8 million. You’ll have to play around and find what works best for you. Just pay attention to how many requests there are in between that “GC cycles so far” number incrementing in Scrap, and you’ll be able to measure approximately how often you’re entering a GC cycle.

Good luck with it, and have fun!

For example, something like this might be used in a model validation:

require 'uri'

def validate_url(url)
	parsed_uri = URI::parse(url)
rescue URI::InvalidURIError
	errors.add :url, "Sorry, that doesn't look like a valid URL"
end

I noticed a bit ago that I started getting invalid URL errors where there shouldn’t be any. After far too long spent in the library’s code, I realized my error: the URLs were being pasted with a trailing space. Stripping the string before attempting to parse it fixed it right up.

I’d argue that URI::parse should likely strip any incoming strings, but in the meantime, remember to strip your user input before trying to determine whether it’s valid or not, or you may end up with frustrated users.

Switch the application entirely to LibXML for all its XML parsing needs

In config/environment.rb: Add the following

ActiveSupport::XmlMini.backend = 'LibXML'

This means that the faster_xml_simple monkeypatch is no longer needed. I don’t think we’re doing much else with XML on blippr, but it’ll be nice to have libxml-backed parsing all around. I must not use REXML. REXML is the app-killer. REXML is the little-death that brings total obliteration.

Fixes for will_paginate and SQL errors when counting records with a custom :select clause

* Upgrade will_paginate. Even after the upgrade, something about 2.3′s named scope handling was still breaking my app. I have a named scope like so:

  :select => "*, (blips.vote_score+2)/WEIGHT_FACTOR as weighted_score",
  :order => "weighted_score desc"

This was causing .paginate calls with this named scope to fail with an invalid SQL error. will_paginate should automatically clobber :select phrases before attempting to count records, but it wasn’t. The solution is to specify a :count condition to my .paginate calls with the right select clause.

Blip.best.paginate(:page => current_page, :per_page => 30, :count => {:select => "blips.id"})

In general, any paginate call with a :select specified seems to break. The :count clause fixes them.

Upgrade my libmemcached plugin

A lot of the internal session stuff has changed. We use Evan Weaver’s libmemcached client, and an upgraded copy of 37signals’ libmemcached store for Rails. The plugin’s been upgraded to work with 2.3, and provides a session store on top of the general Rails store.

Our caching config now looks something like this:

GENERAL_CACHE_SERVERS = ["localhost:11211"]
GENERAL_CACHE_OPTIONS = {:untaint => true}
SESSION_CACHE_SERVERS = ["localhost:11212"]
SESSION_CACHE_OPTIONS = { :prefix_key => "session:blippr" }
SESSION_MEMCACHE_CLIENT = Memcached.new(SESSION_CACHE_SERVERS, SESSION_CACHE_OPTIONS)

config.cache_store = :libmemcached_store, GENERAL_CACHE_SERVERS, GENERAL_CACHE_OPTIONS
config.action_controller.session_store = :libmemcached_store
config.action_controller.session = {
	:cache => SESSION_MEMCACHE_CLIENT,
	:expires_after => 86400
}

Works great with libmemcached, with separate memcached instances for fragments and sessions (so that an over-populated fragment store won’t start clobbering sessions).

Update query parsing

I parse query parameters for some funky filtering. In 2.2.2 I used:

ActionController::AbstractRequest.parse_query_parameters(query_string)

In 2.3, that becomes:

Rack::Utils.parse_query(query_string)

That’s about it for now, but as problems arise I’ll be sure to add them.