{"id":61,"date":"2026-03-15T19:40:00","date_gmt":"2026-03-15T16:40:00","guid":{"rendered":"https:\/\/fsaeed.blog\/?p=61"},"modified":"2026-10-31T00:18:50","modified_gmt":"2026-10-30T21:18:50","slug":"reliable-json-from-an-llm","status":"publish","type":"post","link":"https:\/\/fsaeed.blog\/index.php\/2026\/03\/15\/reliable-json-from-an-llm\/","title":{"rendered":"Getting Reliable JSON Out of an LLM"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Hello, tech enthusiasts! \ud83d\udc4b<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You have 200 supplier emails, and you want a list of invoice numbers, dates and amounts. You ask a language model (LLM) to read each email and give you the data. Most of the time it works. Then one answer starts with &#8220;Sure! Here is your data:&#8221; and your script crashes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An LLM is like a very fast assistant who sometimes adds friendly words you did not ask for. For a person, that is nice. For a script, it is a problem. Today you will learn how to get clean JSON every time, or at least know when you did not.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">JSON is a simple text format for data, like a form with labels and values: <code>{\"amount\": 120.5}<\/code>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What you need<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Python 3.9 or newer (<a href=\"https:\/\/www.python.org\">https:\/\/www.python.org<\/a>)<\/li>\n\n\n<li>An API key from any LLM provider (we use the placeholder <code>YOUR_API_KEY<\/code>)<\/li>\n\n\n<li>Two packages: <code>pip install requests pydantic<\/code><\/li>\n\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Ask clearly, with a schema and an example<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">A schema is a description of the shape of your data: which fields, and what type each one is. Give the model the schema and one example. Also say &#8220;Return only JSON&#8221;. Clear rules give clear answers.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>PROMPT = \"\"\"Read the email below and extract the invoice details.\nReturn ONLY a JSON object. No extra text. Use this shape:\n{\"invoice_number\": string, \"invoice_date\": \"YYYY-MM-DD\", \"amount\": number, \"currency\": string}\n\nExample answer:\n{\"invoice_number\": \"INV-1001\", \"invoice_date\": \"2026-03-01\", \"amount\": 250.0, \"currency\": \"USD\"}\n\nEmail:\n\"\"\"<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Turn on JSON mode if your provider has it<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Many providers offer a &#8220;JSON mode&#8221; or &#8220;structured output&#8221; option. It tells the model that the answer must be valid JSON. Some even accept your full schema. Check your provider&#8217;s documentation and use it when you can.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Many providers share one common request format. Here is a small function for it. Change the URL and model name to match your provider.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import os\nimport requests\n\nAPI_URL = \"https:\/\/api.example.com\/v1\/chat\/completions\"  # your provider's URL\nAPI_KEY = os.environ.get(\"LLM_API_KEY\", \"YOUR_API_KEY\")\nMODEL = \"your-model-name\"\n\ndef call_llm(prompt):\n    response = requests.post(\n        API_URL,\n        headers={\"Authorization\": f\"Bearer {API_KEY}\"},\n        json={\n            \"model\": MODEL,\n            \"messages\": [{\"role\": \"user\", \"content\": prompt}],\n            # JSON mode: remove this line if your provider does not support it\n            \"response_format\": {\"type\": \"json_object\"},\n        },\n        timeout=60,\n    )\n    response.raise_for_status()\n    return response.json()[\"choices\"][0][\"message\"][\"content\"]<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Clean the answer<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Models sometimes wrap the JSON in &#8220;code fences&#8221;: three backticks and the word <code>json<\/code> before it, and three backticks after it. A small function can remove them.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import re\n\ndef strip_fences(text):\n    text = text.strip()\n    # Remove ```json at the start and ``` at the end, if present\n    match = re.match(r\"^```(?:json)?\\s*(.*?)\\s*```$\", text, re.DOTALL)\n    return match.group(1) if match else text<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Validate with json and pydantic<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Valid JSON is not enough. The amount could be the text &#8220;two hundred&#8221;. So we check the data with <strong>pydantic<\/strong>. Pydantic is like a strict receptionist: it checks every field on the form before it lets the data in.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import json\nfrom datetime import date\nfrom pydantic import BaseModel, ValidationError\n\nclass Invoice(BaseModel):\n    invoice_number: str\n    invoice_date: date\n    amount: float\n    currency: str\n\ndef parse_invoice(text):\n    data = json.loads(strip_fences(text))  # step 1: is it JSON?\n    return Invoice.model_validate(data)    # step 2: is it the right shape?<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If something is wrong, <code>json.loads<\/code> raises <code>json.JSONDecodeError<\/code>, and pydantic raises <code>ValidationError<\/code> with a clear message.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Retry, and tell the model what went wrong<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When the check fails, ask again and include the error message. Models are good at fixing a mistake once they see it. Stop after a few tries.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def extract_invoice(email_text, max_tries=3):\n    prompt = PROMPT + email_text\n    for attempt in range(1, max_tries + 1):\n        answer = call_llm(prompt)\n        try:\n            return parse_invoice(answer)\n        except (json.JSONDecodeError, ValidationError) as error:\n            print(f\"Attempt {attempt} failed: {error}\")\n            prompt = (PROMPT + email_text +\n                      f\"\\n\\nYour last answer was not valid: {error}\\n\"\n                      \"Return ONLY the corrected JSON object.\")\n    raise RuntimeError(\"No valid JSON after several tries\")\n\nif __name__ == \"__main__\":\n    email = \"Hi, please find invoice INV-2044 dated 2026-03-10 for 480.75 EUR.\"\n    invoice = extract_invoice(email)\n    print(invoice.invoice_number, invoice.invoice_date, invoice.amount, invoice.currency)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Save all the parts in one file, <code>extract.py<\/code>, set your key with <code>export LLM_API_KEY=YOUR_API_KEY<\/code>, and run <code>python extract.py<\/code>.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Common problems<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>The script fails with &#8220;400 Bad Request&#8221;.<\/strong> Your provider may not support <code>response_format<\/code>. Remove that line and rely on the prompt and the checks.<\/li>\n\n\n<li><strong>Dates come back as &#8220;10\/03\/2026&#8221;.<\/strong> Repeat the format in the prompt (<code>YYYY-MM-DD<\/code>) and keep the example. Pydantic will catch the wrong ones.<\/li>\n\n\n<li><strong>Numbers come as text, like &#8220;1,200.00&#8221;.<\/strong> Say &#8220;amount must be a number without commas&#8221; in the prompt.<\/li>\n\n\n<li><strong>It always fails on one email.<\/strong> Print the raw answer. Maybe the email has no invoice at all. Allow <code>null<\/code> by making the field <code>Optional[str]<\/code>.<\/li>\n\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Wrap-up<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You now have four layers of safety: a clear prompt, JSON mode, a strict check, and a retry. When it still fails, you get a clear error instead of wrong data in your report.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Next idea: save every valid invoice into a CSV file, and every failed email into a &#8220;Check by hand&#8221; list. Then you only look at the few that need a human. \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to ask a language model for clean JSON and check it with Python before you use it.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[7,56,55,57,12],"class_list":["post-61","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-automation","tag-json","tag-llm","tag-pydantic","tag-python"],"_links":{"self":[{"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/posts\/61","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/comments?post=61"}],"version-history":[{"count":1,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/posts\/61\/revisions"}],"predecessor-version":[{"id":72,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/posts\/61\/revisions\/72"}],"wp:attachment":[{"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/media?parent=61"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/categories?post=61"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/tags?post=61"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}