Dan Munckton's Home Page

I live in Edinburgh, work as a software developer, and like to play guitar.

7 Suggestions for Pain Free Data Migrations with Rails

Posted: 2018-01-11

  1. Don’t try to do them in a Rails migration - it’s very, very likely once you are deploying to production you will discover your migration is broken or hideously inefficient. At which point you’ll have to abandon it, make a guess as to how to fix it, then go back through the entire edit, PR, QA, deploy cycle before you can try again.
  2. Don’t try to do them in a Rails migration! - if you want your code reviewed add a temporary script to the repo and get people to review that. This makes it possible to write tests too.
  3. Break down the work into functions that can be pasted into the Rails console - small functions can easily be tested by inspecting their output in the console. Once you’re sure you’re seeing the expected outputs you can then run through a small subset of the work manually to see the overall results are correct. Finally you can confidently run the full migration against all of your data.
  4. Always use batches - while it may not be necessary when running locally or even in your QA environment, it is very likely the sheer amount of data you have in production will mean your data migration will run for a long time and would use a huge amount of memory - imagine slurping all effected records out of your database, across the network and into RAM to manipulate them. ActiveRecord provides some convenient batching methods that allow you to chunk your work into reasonably sized lumps that won’t consume excessive amounts of memory, and can be updated fast enough that you won’t exceed query time-outs.
  5. If possible, write your migration so it can be restarted - regardless of how conscientious you are while writing your data migration, it’s very unlikely it will work first time without either crashing out or just being ridiculously slow. Being able to restart means you don’t need to wait for the whole thing to rerun, if you encounter something unexpected first time around.
  6. Create a snippet that outputs a summary of the work to-do and done - in the event that your migration bombs out having this will help you check your assumptions, give you confidence about what to do next. Or you can execute it in another terminal to check progress of your migration while it’s running.
  7. Find someone to pair with you - this is definitely one of those situations where having someone else watching your back as you run things is really useful.