Getting Reliable JSON Out of an LLM

Hello, tech enthusiasts! 👋

You have 200 supplier emails, and you want a list of invoice numbers, dates and amounts. You ask a language model (LLM) to read each email and give you the data. Most of the time it works. Then one answer starts with “Sure! Here is your data:” and your script crashes.

An LLM is like a very fast assistant who sometimes adds friendly words you did not ask for. For a person, that is nice. For a script, it is a problem. Today you will learn how to get clean JSON every time, or at least know when you did not.

JSON is a simple text format for data, like a form with labels and values: {"amount": 120.5}.

What you need

  • Python 3.9 or newer (https://www.python.org)
  • An API key from any LLM provider (we use the placeholder YOUR_API_KEY)
  • Two packages: pip install requests pydantic

Step 1: Ask clearly, with a schema and an example

A schema is a description of the shape of your data: which fields, and what type each one is. Give the model the schema and one example. Also say “Return only JSON”. Clear rules give clear answers.

PROMPT = """Read the email below and extract the invoice details.
Return ONLY a JSON object. No extra text. Use this shape:
{"invoice_number": string, "invoice_date": "YYYY-MM-DD", "amount": number, "currency": string}

Example answer:
{"invoice_number": "INV-1001", "invoice_date": "2026-03-01", "amount": 250.0, "currency": "USD"}

Email:
"""

Step 2: Turn on JSON mode if your provider has it

Many providers offer a “JSON mode” or “structured output” option. It tells the model that the answer must be valid JSON. Some even accept your full schema. Check your provider’s documentation and use it when you can.

Many providers share one common request format. Here is a small function for it. Change the URL and model name to match your provider.

import os
import requests

API_URL = "https://api.example.com/v1/chat/completions"  # your provider's URL
API_KEY = os.environ.get("LLM_API_KEY", "YOUR_API_KEY")
MODEL = "your-model-name"

def call_llm(prompt):
    response = requests.post(
        API_URL,
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": MODEL,
            "messages": [{"role": "user", "content": prompt}],
            # JSON mode: remove this line if your provider does not support it
            "response_format": {"type": "json_object"},
        },
        timeout=60,
    )
    response.raise_for_status()
    return response.json()["choices"][0]["message"]["content"]

Step 3: Clean the answer

Models sometimes wrap the JSON in “code fences”: three backticks and the word json before it, and three backticks after it. A small function can remove them.

import re

def strip_fences(text):
    text = text.strip()
    # Remove ```json at the start and ``` at the end, if present
    match = re.match(r"^```(?:json)?\s*(.*?)\s*```$", text, re.DOTALL)
    return match.group(1) if match else text

Step 4: Validate with json and pydantic

Valid JSON is not enough. The amount could be the text “two hundred”. So we check the data with pydantic. Pydantic is like a strict receptionist: it checks every field on the form before it lets the data in.

import json
from datetime import date
from pydantic import BaseModel, ValidationError

class Invoice(BaseModel):
    invoice_number: str
    invoice_date: date
    amount: float
    currency: str

def parse_invoice(text):
    data = json.loads(strip_fences(text))  # step 1: is it JSON?
    return Invoice.model_validate(data)    # step 2: is it the right shape?

If something is wrong, json.loads raises json.JSONDecodeError, and pydantic raises ValidationError with a clear message.

Step 5: Retry, and tell the model what went wrong

When the check fails, ask again and include the error message. Models are good at fixing a mistake once they see it. Stop after a few tries.

def extract_invoice(email_text, max_tries=3):
    prompt = PROMPT + email_text
    for attempt in range(1, max_tries + 1):
        answer = call_llm(prompt)
        try:
            return parse_invoice(answer)
        except (json.JSONDecodeError, ValidationError) as error:
            print(f"Attempt {attempt} failed: {error}")
            prompt = (PROMPT + email_text +
                      f"\n\nYour last answer was not valid: {error}\n"
                      "Return ONLY the corrected JSON object.")
    raise RuntimeError("No valid JSON after several tries")

if __name__ == "__main__":
    email = "Hi, please find invoice INV-2044 dated 2026-03-10 for 480.75 EUR."
    invoice = extract_invoice(email)
    print(invoice.invoice_number, invoice.invoice_date, invoice.amount, invoice.currency)

Save all the parts in one file, extract.py, set your key with export LLM_API_KEY=YOUR_API_KEY, and run python extract.py.

Common problems

  • The script fails with “400 Bad Request”. Your provider may not support response_format. Remove that line and rely on the prompt and the checks.
  • Dates come back as “10/03/2026”. Repeat the format in the prompt (YYYY-MM-DD) and keep the example. Pydantic will catch the wrong ones.
  • Numbers come as text, like “1,200.00”. Say “amount must be a number without commas” in the prompt.
  • It always fails on one email. Print the raw answer. Maybe the email has no invoice at all. Allow null by making the field Optional[str].

Wrap-up

You now have four layers of safety: a clear prompt, JSON mode, a strict check, and a retry. When it still fails, you get a clear error instead of wrong data in your report.

Next idea: save every valid invoice into a CSV file, and every failed email into a “Check by hand” list. Then you only look at the few that need a human. 🙂

fsaeed

, , , ,