Case study · AI automation agency

Talk to your ERP

A natural-language analytics assistant over a 64k-order ERP mirror. The LLM writes a JSON query plan, the server checks it against a whitelist and runs it through a read-only API, and an eval harness keeps it honest after every change.

ClientA B2B workplace supplier in Ireland
EmployerPrevious company (an AI automation agency), as a contract AI engineer
ContextPart of a €31k AI sales platform I built as sole engineer in about 6 weeks
UsersThe supplier's sales team and management

Problem

The sales team and management wanted to ask questions about customers, orders and products in plain English. The ERP tables belonged to another system and had to stay read-only. Generated SQL is hard to fence in, and LLM analytics can fail with a "confident zero", where the model filters on a value that doesn't exist in the data, gets no rows, and reports zero as the answer.

Constraints

Architecture

  1. QuestionTyped into the chat screen
  2. GroundingReal distinct values of low-cardinality columns (cached), the closest worked examples, and date windows computed in code
  3. LLM call 1: query planJSON naming a dataset, columns, filters and an aggregate
  4. Plan validatorDataset and column whitelist. Anything else is rejected.
  5. Read-only APIThe plan becomes a GET on a PostgREST API. Aggregation runs in code.
  6. LLM call 2: answerWritten from the returned rows only
    Back-translation checkThe executed plan is rendered back into English and an LLM judges whether it answers the question
  7. AnswerShown with where the numbers came from, plus a note if the check had doubts
  • LLM step
  • Plain code
  • User
The model never sends SQL. The only thing it can produce is a plan, and a plan can only describe a read.

What I did

I rejected text-to-SQL early. The model writes a JSON query plan, and the server validates it against a per-dataset column whitelist before turning it into a GET request on a read-only PostgREST API. Because a plan can only describe a read, writes are impossible by construction.

To stop the confident-zero failure, the assistant samples the real distinct values of short columns before planning and puts them in the prompt, so the model filters on values that exist. It also retrieves the worked examples closest to the question and passes in date windows already computed in Irish time, so the model never does date arithmetic.

After a query runs, a second call writes the answer from the rows. In parallel, a back-translation verifier renders the executed plan back into English and asks an LLM whether that plan answers the question. The verifier only adds a disclosure to the answer and never rewrites it.

I built a 74-question eval harness from the client's own question catalogue. It runs against the live endpoint after every prompt or schema change and records the dataset and aggregate chosen, the row count and the latency. I also wrote the model-aware OpenAI client underneath, which handles GPT-5 reasoning-token budgets, retries, and per-call token and dollar cost metering.

Results

70 of 74
questions answered in a hand-checked live sweep
median 12s
latency per question in the sweep
64k
orders in the ERP mirror

The assistant refused the out-of-scope and destructive requests in the sweep, and none of them could have changed data in any case.

Stack

Timeline

The assistant was one feature of the sales platform, built in about 6 weeks.

  1. Query-plan design, whitelist validator and the read-only execution path.
  2. Value grounding and retrieved few-shot examples, so plans filter on values that exist.
  3. The back-translation verifier.
  4. The 74-question eval harness, re-run after each prompt or schema change.

The client is described by country and industry instead of by name.

Back to case studies