Case study · New Zealand restaurant platform
Sales ingest for a 180-store fast-food franchise group in New Zealand
Every night each store's point-of-sale system sends its sales, and by morning every one of them has to be in the ledger. I moved that ingest off a synchronous request path onto S3, SQS FIFO and autoscaled Fargate workers, then load-tested it until it held at 180 stores.
| Client | A fast-food franchise group in New Zealand, rolling out to 180 stores |
|---|---|
| Employer | Previous company, as a contract software engineer |
| Product | An AWS and TypeScript platform replacing a legacy low-code back office for the franchises |
| How I built it | With Claude Code agents, working from a spec, a plan and tests. I reviewed every change before it merged. |
Problem
Store sales arrive as XML bundles. The old path parsed each bundle inside the HTTP request, in a single 930-line database transaction. It managed 3.75 bundles a minute, which was too slow for a 180-store rollout, and long requests were timing out at the load balancer. Two pushes for neighbouring days from the same store could also race each other, which left some sales days silently underived.
Constraints
- No store-day can go missing. Sales feed stock, costing and the reports sent back to the franchisor.
- Bundles from one store have to be processed in order, while different stores can run in parallel.
- The ledger is a shared multi-tenant Postgres database with row-level security, so heavy locking hurts every tenant.
- The rollout goes tenant by tenant behind a feature flag, and the new path had to be proven under a full-size load test before any of that.
Architecture
- Store point-of-saleNightly XML sales bundle, sent through a relay
- Accept endpointStore raw bytes in S3, write a pending row, reply fast
- SQS FIFO queueOne message group per store, so each store's days stay in order. Dead-letter queue behind it.
- ECS Fargate workersAutoscaled. Claim the pending row, parse the bundle, derive ledger events.
- Postgres event ledgerMulti-tenant, row-level security
-
Scheduled sweepEventBridge job that finds sales days which never completedAlarms and recoveryCloudWatch metrics and alarms, plus a support console to replay a failed ingest
- Self-healing check
- Operator path
What I did
I re-architected the accept path so it does very little. It writes the raw bundle to S3, records a pending row, and sends one SQS FIFO message keyed by store. A pool of autoscaled Fargate workers picks up the messages, claims the row, parses the XML and derives the ledger events. A scheduled EventBridge sweep catches any sales day that didn't complete, and support staff get a console to replay a failed ingest without engineering help.
The first 180-store load test failed. p99 accept latency was 13.3s. I worked through the CloudWatch metrics and found three root causes. S3 writes were happening inside a locked database transaction. A version check parsed the whole XML file just to read one attribute. And the autoscaler was scaling in workers that were still busy. I moved the S3 writes out of the lock, replaced the full parse with a root-tag scan (507ms to 10ms at p50), and stopped busy workers from being scaled in. Together that cut p99 accept latency 8x.
Afterwards I extracted the reusable CDK constructs and the async-ingest runtime into shared npm packages and moved consumers across 4 repositories onto them. Synth-diffing 14 stack and environment pairs showed zero infrastructure drift.
Results
Measured in a 180-store, 7-day k6 load test of 1,260 bundles at 5 MiB each.
On real traffic, accept latency went from 4.5-7s to 0.7-1.1s.
Stack
Timeline
- Designed the queue-and-worker ingest and the per-tenant feature flag.
- Built the accept path, FIFO queue, Fargate workers, sweep and alarms.
- First 180-store load test: p99 accept latency of 13.3s, a fail.
- Traced the three root causes and fixed them.
- Rerun of the 180-store, 7-day test: p99 of 1.63s, 0 dead letters, 0 dropped site-days.
- Extracted the pattern into shared packages used across 4 repositories.
The client is described by country and industry instead of by name.