Some bugs announce themselves. This one didn’t — it took months, a suite that kept growing, and one very confused stray row in an unrelated test’s count before anyone could say for certain what was going on.

The problem: a job enqueue three conditionals deep

Somewhere inside converting a lead into a subscription plan, there’s a helper called enqueueInvoiceRequest. You don’t get to it directly — you get to it by going through convertProduct, which is itself dispatched to from a product-type switch one layer up, and even once you’re inside it you still have to clear a legacy-plan early return, a brand check, an invoice-count guard, and a null check on a webhook request ID before the enqueue itself ever runs:

// server/src/services/lead-conversion/plan/index.ts
if (attributes.brandId === BRAND_ID.acme) {
  await enqueueInvoiceRequest({ plan, trx });
}

// ...inside enqueueInvoiceRequest:
if (inboundWebhookRequestId) {
  // We need to wait for the transaction to commit before we can add the item to the queue
  attachToExecutionPromise(trx, "invoice-enqueue", async () => {
    await invoiceQueue.addItem({
      type: "create-invoice",
      inboundWebhookRequestId,
      planId: plan.id,
    });
  });
}

That comment — “we need to wait for the transaction to commit” — is the whole problem in one line. The enqueue can’t run inside the transaction, because the transaction might still roll back and there’s no un-enqueueing a job. It also can’t run synchronously right after trx.commit(), because nothing at this depth of the call stack has a reference to “right after commit” — it’s three function calls and four conditionals away from whoever actually owns the transaction and decides when it commits.

Why executionPromise, and what we didn’t do instead

Objection’s Transaction exposes executionPromise, which resolves once the transaction settles — commit or rollback, either way. Attaching to it with .then() gives you exactly the guarantee you need at exactly the point in the code where you need it, no matter how deep that point is:

trx.executionPromise.then(fn);

That’s the entire mechanism. No return value has to travel back up through enqueueInvoiceRequest → convertProduct → the product-dispatch switch → whatever called that, just to tell the transaction’s owner “also run this once you’re done.” Every intermediate function stays exactly as it is.

We did consider the alternatives, briefly:

  • Thread the deferred action back out through the return value, and let the code that actually owns the transaction run it after commit. This is the “correct” version in some abstract sense — no fire-and-forget, no ambient coupling to a transaction object passed four calls deep. It’s also a real cost: every function between the enqueue and the transaction’s owner would need its return type widened to carry “oh, and also run this afterwards,” purely to plumb one job dispatch through call sites that have nothing to do with it. Given that we had multiple use cases like this it just didn’t make sense.
  • Just delay it. Fire the job on a setTimeout a few hundred milliseconds after the write, on the assumption the transaction will have committed by then. This isn’t a guarantee, it’s a bet — one that gets worse under load, not better, since a slow commit and a fast timer are exactly the conditions you’d expect during the traffic spike you most need this to be correct for.

executionPromise won on both counts: it’s precise (it resolves on the actual event, not a guess about timing), and it’s local (the code that needs the deferral is the only code that has to know about it).

The CI failure that got worse, not better

The trouble is that “fire-and-forget” and “test suite” don’t get along. The first sign of it was a commit back in April with an admirably honest message: “Add a executionPromise callback has made test more flakey” — the fix at the time was a global Mocha hook that stubbed the relevant client, wrapped its deferred callback in a 2-second timeout, and collected any errors to re-throw in a global afterEach. It made the symptom quieter without doing anything about the cause: a callback that isn’t awaited by anything can still be running, or about to run, after the test that triggered it has already finished and moved on.

It didn’t go away. It got worse, and predictably so, once we started rolling out transaction-based test isolation more broadly — every test now ran inside its own transaction that got rolled back at the end. That’s a good change on its own merits, but it also changed where a callback firing “late” actually landed: instead of writing to an already-cleaned-up database, it wrote straight into whatever other test happened to have its own transaction open at that exact moment, silently adding a row to whatever that test was counting, a row that wasn’t part of a transaction, and wouldn’t get rolled back. Small suites rarely lined up that unluckily. Running the full suite did, as a handful of count assertions that failed for no reason anyone could reproduce in isolation.

Reframing executionPromise as a background job, not a callback

The thing that actually unlocked the fix was stepping back from “this is a callback attached to a transaction” and treating it as what it really is: background job execution, deferred past a boundary the test environment doesn’t otherwise care about. Once it’s framed that way, the fix looks exactly like how you’d handle any other job queue in tests — don’t run it for real, capture it, and let the test decide if and when to run it:

export const attachToExecutionPromise = (
  trx: Transaction,
  description: string,
  fn: () => Promise<void>,
): void => {
  if (ENV.NODE_ENV === "test") {
    capturedCallbacks.push({ description, fn });
    return;
  }

  trx.executionPromise.then(fn);
};

Under test, nothing fires unawaited and nothing can outlive the test that triggered it — the callback just sits in an array until something asks for it by name, or the next test’s beforeEach clears it out. A test that cares can pull its own callback out with getCapturedExecutionPromiseCallback('invoice-enqueue') and run it deliberately, on its own schedule, same as it would call any other job handler directly.

It wasn’t a total, clean victory, though — worth saying, since that’s the more honest version of this story. One integration test, after adopting this fix, still leaked a row into unrelated tests at full-suite scale, from somewhere in the same request flow that this fix didn’t reach. We had a custom patch for it until we figure out a proper fix, and a comment saying as much, rather than a triumphant “and then everything was fine.” Sometimes the honest state of a fix is “this part’s solved, that part’s still open,” and it’s better to write that down than to pretend otherwise.

The full source — attachToExecutionPromise(), the capture/flush test helpers, and the wiring into the test suite — is up in the complete code reference, if you want the whole thing rather than the excerpts above.