Reducing the cost of mistakes can be just as valuable as preventing them

With system safety on my mind lately (thanks to the excellent Pre-Accident Investigations podcast), I want to reflect on one way we improved the safety of our deployments at work.

As our app and user-base have grown, we’ve had a few incidents where the typical Rails migration methods like add_column and add_index have caused outages (these are leaky abstractions) due to the fact that adding columns with defaults or indexing large tables in certain ways can lock the database in ways that prevent it from serving users. Our first answer was to attempt to prevent these kinds of migrations from running.

Preventing Mistakes with Software

For this we turned to the strong_migrations gem which will raise errors when running migrations if you perform certain “potentially unsafe” operations. This helped catch some issues for us, but had two major drawbacks:

• False positives: operations which we knew were safe given the circumstances but were still flagged by the tool trained us in the bad habit of frequently ignoring the tool’s warnings with its safety_assured escape hatch, even when we shouldn’t.
• False negatives: strong_migrations is a useful tool, but it can’t catch everything. Having a warning mechanism in place gave us confidence when shipping queries that were incidentally, rather than structurally, problematic.

Preventing Mistakes with Humans

Our lack of trust in strong_migrations then led us to institute a somewhat onerous process where all new migrations had to wait for approval by a developer familiar with all the ways they could go wrong. Not only did this slow us down, but this approval was itself no guarantee of safety.

Reducing the Cost of Mistakes

While we still use strong_migrations, we have since augmented it with activerecord-safer_migrations which takes a different approach: rather that trying to prevent unsafe migrations from being run, this tool applies an aggressive timeout (<2s) to all migrations so that if they attempt to perform an unsafe operation, they can lock the database for only a few seconds rather than indefinitely. The worst that happens is we’re notified and have to rewrite the migration in a safer way.

By reducing the impact of unsafe migrations, we felt comfortable removing the imperfect “human approval” step. strong_migrations now has the role of teaching tool, helping us write safer migrations most of the time but not needing to catch every possible issue (which it can’t).

The Lesson

Think of the systems and processes you have in place to prevent mistakes—especially if their cost is non-negligible and their benefits inconsistent—and consider whether they could be replaced or at least augmented by factors which lower the cost of inevitable mistakes.