Slow requests should not always stay requests

We recently had production get slow enough that pods started restarting. The liveness probe was becoming part of the problem. Requests were taking too long, the probe would go stale, Kubernetes would restart the pod, and the cycle would repeat.

The first patch was a plaster: increase the liveness probe timing so the pods would stop killing themselves while requests were slow. Not the real fix, but useful enough to stop the bleeding. After that, the actual investigation started.

Some of the fixes were straightforward database work. There were slow queries missing the right indexes, and lookups that only needed one row but were not limiting themselves to one.

One fix was literally this:

const [something] = await knex("something")
  .where({ type, identifier })
  .limit(1);

Boring, but useful.

Once those obvious issues were out of the way, another problem became much clearer: some routes were doing work that should not happen inside the request path at all.

They were doing heavy CPU work during normal API requests. While that work was running, the API had less room to respond to everything else, including the liveness probe. Eventually the probe would go stale and Kubernetes would restart the pod.

So the next fix was not another query tweak. It was changing the shape of the work.

I drafted the architecture and started building a BullMQ worker setup:

the API validates the request
the API enqueues a job
Redis stores the job
a separate worker picks it up
the worker does the slow part outside the request lifecycle

That makes the request much faster because the web process no longer has to sit there doing all the heavy work before it can respond.

The work still has to happen somewhere. BullMQ does not magically remove the cost. But it moves the cost to a better place.

The API can stay responsive. The worker can have its own concurrency. Retries can be handled as jobs instead of random repeated HTTP requests. Slow work becomes something we can observe and control instead of something hidden inside an endpoint.

For small tasks this would be overkill. For routes doing enough CPU work to slow down production and interfere with health checks, it is a much better fit.

Sometimes the fix for a slow request is not making the request a little smarter. Sometimes it is to stop making the request do the work.