Case study · AI automation agency
Talk to your ERP
A natural-language analytics assistant over a 64k-order ERP mirror. The LLM writes a JSON query plan, the server checks it against a whitelist and runs it through a read-only API, and an eval harness keeps it honest after every change.
| Client | A B2B workplace supplier in Ireland |
|---|---|
| Employer | Previous company (an AI automation agency), as a contract AI engineer |
| Context | Part of a €31k AI sales platform I built as sole engineer in about 6 weeks |
| Users | The supplier's sales team and management |
Problem
The sales team and management wanted to ask questions about customers, orders and products in plain English. The ERP tables belonged to another system and had to stay read-only. Generated SQL is hard to fence in, and LLM analytics can fail with a "confident zero", where the model filters on a value that doesn't exist in the data, gets no rows, and reports zero as the answer.
Constraints
- Writes to the ERP data had to be impossible, whatever the model produced.
- Filters had to use values that exist in the data, such as the actual category names.
- Relative dates like "last quarter" had to resolve the same way every time, in Irish time.
- Every prompt or schema change needed a repeatable check, so improvements on one question didn't quietly break others.
Architecture
- QuestionTyped into the chat screen
- GroundingReal distinct values of low-cardinality columns (cached), the closest worked examples, and date windows computed in code
- LLM call 1: query planJSON naming a dataset, columns, filters and an aggregate
- Plan validatorDataset and column whitelist. Anything else is rejected.
- Read-only APIThe plan becomes a GET on a PostgREST API. Aggregation runs in code.
-
LLM call 2: answerWritten from the returned rows onlyBack-translation checkThe executed plan is rendered back into English and an LLM judges whether it answers the question
- AnswerShown with where the numbers came from, plus a note if the check had doubts
- LLM step
- Plain code
- User
What I did
I rejected text-to-SQL early. The model writes a JSON query plan, and the server validates it against a per-dataset column whitelist before turning it into a GET request on a read-only PostgREST API. Because a plan can only describe a read, writes are impossible by construction.
To stop the confident-zero failure, the assistant samples the real distinct values of short columns before planning and puts them in the prompt, so the model filters on values that exist. It also retrieves the worked examples closest to the question and passes in date windows already computed in Irish time, so the model never does date arithmetic.
After a query runs, a second call writes the answer from the rows. In parallel, a back-translation verifier renders the executed plan back into English and asks an LLM whether that plan answers the question. The verifier only adds a disclosure to the answer and never rewrites it.
I built a 74-question eval harness from the client's own question catalogue. It runs against the live endpoint after every prompt or schema change and records the dataset and aggregate chosen, the row count and the latency. I also wrote the model-aware OpenAI client underneath, which handles GPT-5 reasoning-token budgets, retries, and per-call token and dollar cost metering.
Results
The assistant refused the out-of-scope and destructive requests in the sweep, and none of them could have changed data in any case.
Stack
Timeline
The assistant was one feature of the sales platform, built in about 6 weeks.
- Query-plan design, whitelist validator and the read-only execution path.
- Value grounding and retrieved few-shot examples, so plans filter on values that exist.
- The back-translation verifier.
- The 74-question eval harness, re-run after each prompt or schema change.
The client is described by country and industry instead of by name.