Getting Structured Output from LLMs: Stop Parsing JSON by Hand

I remember the first time I tried to build a production feature around an LLM. I was at Lit Alerts, and we needed to extract structured data from user-submitted text. I thought it would be easy. I asked the model for JSON, and it gave me JSON. Sometimes. Other times, it gave me markdown backticks, or it decided to add a conversational preamble, or it just hallucinated a field that did not exist in my schema. My JSON.parse calls were wrapped in try/catch blocks that were more complex than the actual business logic. It was a mess.

The Old Way: Prompt and Pray

The "Prompt and Pray" approach is exactly what it sounds like. You write a prompt, you hope the model follows your instructions, and you cross your fingers that the output is valid JSON.

const prompt = "Extract the user's name and email from this text. Return only JSON.";
const response = await llm.generate(prompt);

try {
  const data = JSON.parse(response);
  // Do something with data
} catch (e) {
  // Handle the inevitable failure
}

This pattern is fragile. If the model decides to wrap the JSON in markdown code blocks, JSON.parse fails. If the model adds a friendly "Sure, here is the JSON you requested:" at the start, JSON.parse fails. If the model decides to use a different key name than what you expected, your application crashes. You end up writing complex regex to strip out markdown, or you spend hours tweaking the prompt to be more "strict," which only works until the model updates.

Structured Output with Zod Schemas

The modern way to handle this is to treat the LLM response as a contract. Instead of hoping the model gives you the right shape, you define the shape using Zod and force the model to adhere to it.

When you use Zod to define your schema, you are not just validating data. You are creating a single source of truth that your TypeScript code and your LLM prompt can both understand.

By passing a Zod schema to the LLM, you ensure that the output is validated before it ever reaches your business logic.

const UserSchema = z.object({
  name: z.string(),
  email: z.string().email(),
});

// The LLM call now uses the schema to enforce the structure
const result = await llm.generateStructured({
  prompt: "Extract user info",
  schema: UserSchema,
});

// result is now a type-safe object
console.log(result.name);

This approach eliminates the need for manual parsing and try/catch blocks. If the model fails to produce the correct structure, the library handling the structured output will throw an error or return a validation failure, which you can handle gracefully.

Provider-Specific Approaches

Different LLM providers have different ways of handling structured output. OpenAI has a dedicated json_schema parameter that is incredibly robust. Anthropic uses a tool_use pattern, where you define a tool that the model must call, and the arguments to that tool are your structured data.

Open-source models are catching up quickly. Many now support grammar-based sampling, which restricts the model's output tokens to only those that are valid according to a JSON schema. This is the gold standard for structured output because it makes it mathematically impossible for the model to generate invalid JSON.

Graceful Degradation

Even with structured output, things can go wrong. The model might be overloaded, or the input text might be too ambiguous. You need a strategy for when validation fails.

Never assume the LLM will succeed on the first try. Build your pipelines to handle failure as a first-class citizen.

The best pattern is to use safeParse from Zod. If the parsing fails, you can take the error message from Zod, feed it back to the LLM, and ask it to correct its mistake.

const parseResult = UserSchema.safeParse(llmOutput);

if (!parseResult.success) {
  // Feed the error back to the LLM for a retry
  const correctionPrompt = `The previous output was invalid: ${parseResult.error.message}. Please fix it.`;
  return await llm.generate(correctionPrompt);
}

This retry loop is often enough to fix minor formatting issues or missing fields.

Real-World Patterns

In production, I use structured output for everything from data extraction pipelines to form auto-fill features. Here is a schema for a simple classification task:

const ClassificationSchema = z.object({
  category: z.enum(["support", "sales", "feedback"]),
  priority: z.number().min(1).max(5),
  summary: z.string(),
});

This schema is simple, but it is powerful. It allows me to route incoming messages to the right team automatically, without a human ever looking at the raw text. By moving away from manual parsing and embracing Zod, I have made my LLM-powered features significantly more reliable and easier to maintain. Stop parsing JSON by hand. Let your schemas do the work for you.