Comments on: Mass inserting data in Rails without killing your performance

By: Vid Luther

Vid Luther — Fri, 07 Aug 2009 14:19:09 +0000

Great post, just an FYI for all who are interested, the reason why the single mass insert in MySQL is so much faster, is because you’re doing what’s known as an “extended insert”. What this does is, tells MySQL to rebuild the index AFTER the data has been inserted, not in the middle, or after each insert.

By: Brian Armstrong

Brian Armstrong — Mon, 27 Jul 2009 03:52:48 +0000

Awesome, you da man! Thanks for sharing this.

By: Michael

Michael — Tue, 16 Jun 2009 12:01:13 +0000

Is it possible to tell Rails to *not* wrap Model.save in a transaction — we’re uploading large volumes of data and the BEGIN/INSERT/COMMIT is surely much slower than a simple INSERT. Anyone know how to tell Rails to not wrap a ’save’ in a transaction?

By: Kalivo.com - Rails Batch Inserts/updates benchmark

Kalivo.com - Rails Batch Inserts/updates benchmark — Wed, 20 May 2009 16:52:16 +0000

[...] Here is a basic benchmark of ar-extensions. I used a slightly modified version of this script. [...]

By: jfromm

jfromm — Mon, 09 Feb 2009 10:20:05 +0000

Good overview.

By: David Alison

David Alison — Sat, 07 Feb 2009 09:46:45 +0000

Excellent post Chris – exactly what I was looking for when I was trying to Google up solutions for bulk inserts in Rails. I appreciate you taking the time to write this down.

By: Daniel Haran

Daniel Haran — Mon, 26 Jan 2009 13:58:14 +0000

On MySQL, I used LOAD DATA LOCAL INFILE:
http://danielharan.wordpress.com/2007/06/12/ruby-activerecord-is-too-slow/

By: Chris Heald

Chris Heald — Sat, 24 Jan 2009 18:24:54 +0000

Thanks for the tip, Mike. The docs on ar-extensions seem to be a bit light!

I’ll give that a shot. It’s nice to stay in Ruby when at all possible!

By: BJ Clark

BJ Clark — Sat, 24 Jan 2009 14:01:59 +0000

AR-Extensions is great, except that it monkey patches lots of stuff in ActiveRecord which can create crazy weird bugs if you’re doing any other modifications of AR. We use Masochism (with mods) for DB load balancing, and AR-Extensions borked a bunch of stuff with it. Which ended up costing us more dev time than just doing a transaction with a big loop.

By: Mike Perham

Mike Perham — Sat, 24 Jan 2009 13:09:27 +0000

ar-extensions is great. You didn’t get the multi-insert capability because you didn’t require the mysql connector extensions:

require ‘ar-extensions/adapters/mysql’
require ‘ar-extensions/import/mysql’

Also, you can get a big performance increase from ar-extensions when saving a lot of models by disabling validations:

Model.import data, :validate => false

Yes, I’ve spent a lot of time on this too. :-)