AI Daddy › Prompting & Context
Structured Generation · Prompting & Context
Structured Generation is the process of forcing an LLM to produce output in a machine-readable format (JSON, YAML, CSV) with 100% reliability. The…
Structured Generation
Structured Generation is the process of forcing an LLM to produce output in a machine-readable format (JSON, YAML, CSV) with 100% reliability. The discipline has moved from "prompt-based requests" to "engine-level constraints."
Table of Contents
The JSON Mode Revolution
Historically, getting JSON was a struggle of "only return JSON, no other text."
Standard approach: Use native response_format: { type: "json_schema" } (OpenAI/Gemini) or tool-output schemas (Anthropic).
- Benefit: 100% syntactical validity. The model literally cannot output a string that is not a valid JSON.
- Behind the scenes: The serving engine masks the vocabulary at each step, ensuring only valid JSON characters (e.g.,
{, ", :, [) can be picked next.
Function calling is structured generation where the LLM "picks" a function and populates its arguments.
// Example Tool Call
{
"name": "get_stock_price",
"arguments": { "symbol": "AAPL", "interval": "1d" }
}
Nuance: Parallel Function Calling is now standard. A model can decide to call 5 different tools simultaneously (e.g., check account balance, check credit score, check loan rates) and aggregate the results.
Constrained Decoding (CFG & Regex)
For self-hosted models (Llama-cpp, vLLM via Outlines), we use Context-Free Grammars (CFG) or Regex.
# Outlines Pattern
model = outlines.models.transformers("meta-llama/Llama-4-8B")
generator = outlines.generate.regex(model, r"(\d{3})-\d{3}-\d{4}")
# Result: The model can ONLY output telephone numbers.
For complex data extraction (e.g., 50 fields from a medical record), don't do it in one pass.
- Stage 1 (Text-to-Text): Extract a "messy" but complete set of facts in natural language.
- Stage 2 (Text-to-JSON): Use a smaller, cheaper model to convert those natural language facts into a strict JSON schema.
- Benefit: Reduces "hallucination under pressure"—large models struggle when forced to reason AND follow strict syntax simultaneously.
Even with "JSON mode," the Logic inside the JSON might be wrong (e.g., a field is missing or a date is in the wrong format).
Recovery pattern:
- Validate output against Pydantic/Zod.
- If it fails, send the Traceback back to the model:
"Error: Field 'age' must be an integer, got 'twenty'. Fix and re-generate."
- Most models fix the error on the first retry.
Interview Questions
Q: Why is "JSON Mode" more reliable than prompt-based JSON requests?
Strong answer:
Prompt-based requests rely on the model's willingness to follow instructions; "JSON Mode" (or Constrained Decoding) relies on the serving engine's inability to do anything else. By applying a "Logit Bias" or a "Grammar Mask" at the inference level, the engine restricts the choice of the next token to only those that would be valid according to the schema. This eliminates the "preamble" (e.g., "Sure, here is your JSON...") and ensures that you never get a malformed string due to high temperature or randomness.
Q: What is the risk of asking an LLM for too many structured fields at once?
Strong answer:
There is a trade-off between Schema Complexity and Information Integrity. As the schema grows (e.g., 20+ hierarchical fields), the model's attention is consumed by maintaining the JSON structure (brackets, keys, quotes) rather than verifying the accuracy of the data. This often leads to "Omission Hallucinations" where the model skips fields or fills them with placeholder data. The mitigation is to use a "Chain-of-Density" extraction or split the extraction into multiple parallel sub-tasks.
References
- OpenAI. "Structured Outputs Documentation" (August 2024 update)
- Outlines Project. "Context-Free Grammar Guided Generation" (2024)
- Willard et al. "Efficient Guided Generation for LLMs" (2023)
Next: Prompt Optimization (DSPy)