The production bug that changed our QC system

We move fast at NAMA. A problem reported in the evening is often fixed in production before the customer wakes up. I used to be purely proud of that loop. Then one night it shipped a mistake.

The bug that got me wasn't clever. On 18 September we shipped a guard that stops a booking being marked confirmed until every hotel and transfer line has been confirmed by the supplier - the kind of rule an operator would nod at. The check compared each line's status to the word CONFIRMED. The status, though, was an enum, and in Python the text form of an enum member is BookingStatus.CONFIRMED, not CONFIRMED. Never equal. So from that afternoon, every booking with lines failed the check. Every one. The only path through was the manager override we had added for exceptions.

The unit test passed because it fed the function plain strings; the real rows never did. It surfaced the way these things do: an external tester on a QA tenant confirmed every line by hand, pressed Confirm, and got a polite 422 explaining that nothing was confirmed. Twenty-two hours on production. The fix was one line - compare the enum's value - and one test that uses the real model instead of a stand-in.

The lesson I keep: a test that passes with fake shapes is a test of your imagination, not your system.

But the lesson took longer. Speed without a quality loop is just a faster way to make mistakes. In travel, a wrong number on a quote is not a typo. It is someone's margin, or someone's holiday.

So we changed the system, not just the code. Every fix now ships with a test that would have caught it. Nothing merges without a checklist. And after every deploy, probes run real scenarios against real tenant data and report back.

The bug was embarrassing for a day. The QC system it forced is permanent. That is a trade I will take every time.