Notes

Choosing dull infrastructure for a two-product company

Postgres, a queue, and a deliberately short list of managed services. The reasoning behind each choice, the timeout bug that taught us the most, and the places where it would not survive ten times the load.

Rentisha's backend is a NestJS monorepo talking to Postgres through Prisma, with BullMQ on Redis for background work, running on Cloud Run. The interesting thing about that list is how short it is and how old every item on it is.

That is a deliberate position for a company with three founders and two products. Every additional managed service is a thing to configure, secure, monitor, pay for, and (the cost people underestimate) a thing that must be understood by whoever is on call at 22:00. With a small team, the binding constraint is not what the architecture can do. It is how much of it fits in one person's head during an incident.

Postgres does more jobs than it is given credit for

Postgres is the relational store. It is also the authorisation boundary, via row-level security. It is also the audit log, via append-only tables. It is also, through unique constraints, the idempotency mechanism that keeps invoices from being filed with the tax authority twice.

Each of those could be a separate system. A dedicated audit pipeline, a policy engine, a distributed lock service. Each would be better in isolation and would add an integration point, a failure mode, and a consistency question between it and the database.

Keeping them in Postgres means they participate in the same transaction. When an invoice is filed and its audit row is written and its uniqueness is enforced in one transaction, there is no window where two of the three are true.

The bug that taught us the most

The most instructive failure in the system was not an outage. It was a default.

Prisma's interactive transactions default to waiting two seconds to acquire a connection and five seconds to complete. Those are sensible numbers for a warm pool serving web requests. Background workers are not that:

A scheduled worker on an otherwise-idle instance finds a cold connection pool, and establishing a fresh connection routinely takes longer than the default is willing to wait. So every single scheduled scan failed, at the transaction layer, with an error that reads like a database problem and is really a defaults problem. The correction is one line and the reasoning behind it is the durable part: in a background worker, waiting longer is always safe and failing fast never is.

Three ordinary decisions; scale to zero, a small database instance, framework defaults; combined into a scheduled job that failed every five minutes without any of the three being wrong. Nothing was down. Every component was behaving as configured.

The fix raised the waits to ten and thirty seconds, and the last line is the part we kept: for a background job, waiting longer is always safe and failing fast is not. Fail-fast is a good default for a request with a human attached to it. For a cron job with nobody watching, a slow success beats a fast failure every time, and the correct timeout is a property of the caller, not of the library.

Where it would not survive ten times the load

Being specific about this is more useful than claiming the architecture scales.

  • The eTIMS submission worker runs at concurrency 1. That is imposed by KRA's per-device rate limiting rather than by us, so ten times the invoice volume needs a different filing topology, not a bigger worker.
  • The database instance is small. It is the right size for current load and it is the first thing that would need to change, and unlike most of this list, changing it is easy.
  • Redis is a single instance. It is a queue, not a store of record; every durable obligation is a database row, which is what makes losing Redis a delay rather than a data-loss event. But a single instance is still a single instance.
  • Suppression checks and frequency caps run counting queries per message. Fine at current volume, obviously wrong at a hundred times it.

None of these are urgent. All of them are known, which is the actual point; an architecture you cannot describe the limits of is one you have not understood.

Boring is a strategy, not a compromise

The temptation with a small team is to compensate for headcount with sophistication: the newer tool, the more clever architecture, the thing that removes an entire category of work. Sometimes that pays. More often it converts a problem you understand into one you do not, and the bill arrives when the person who chose it is busy.

Postgres has been debugged by more people than will ever use our software. When something goes wrong at 22:00, the error message has been seen before, by someone who wrote about it. That is not a small property. It is most of why the list is short.

Keep reading