Lesson 0005 · ~20 minutes

LLM integration engineering

Case AI Interview prep · Layer 4 — making AI production-grade · Lesson 0004 · Glossary
Why this lesson The job post asks for exactly this: "integrate OpenAI, Anthropic, open-source models; provider routing; fallback handling; response normalization" plus "tool-calling / multi-step AI workflows … with human-in-the-loop checkpoints." This is ordinary backend engineering applied to an unreliable, expensive, non-deterministic dependency — which means your existing instincts carry most of the weight.

0 · Warm-up — retrieval from lesson 0004

1 · One interface, many providers

Case AI calls multiple model providers for three reasons: fit (a cheap fast model for classification and chunk-context generation, a frontier model for drafting), cost (routing 80% of calls to models 10–20× cheaper is real money), and resilience (a provider outage must not be a product outage). The pattern you already know from payment gateways applies verbatim: define one internal interface — complete(request): NormalizedResponse — with an adapter per provider. Response normalization means every adapter maps its provider's shapes (finish reasons, token counts, tool-call formats, error types) into your one internal shape, so the rest of the codebase never knows which vendor answered.

2 · Routing, fallbacks, and failure

Routing is a policy on top of that interface: by task type ("summarize" → cheap tier; "draft response to opposing counsel" → frontier tier), by tenant plan, or by current provider health. Fallback handling is what happens when a call fails: retry with exponential backoff on transient errors (429/5xx), then fail over to the next provider in the chain — and mark degraded responses as such rather than silently swapping quality. Two backend classics to name: timeouts (never wait unboundedly on a model) and a circuit breaker (stop hammering a provider that's down; probe it periodically). Streaming (token-by-token to the portal) is the other production reality — one more reason the API tier is a long-lived Fargate service, tying back to Lesson 0002.

3 · Structured outputs — when the answer is data, not prose

Extracting parties, dates, and obligations from a contract must yield machine-usable JSON, not an essay. Structured outputs constrain generation to a JSON Schema you supply — OpenAI enforces schema adherence natively (docs); with Claude the same is achieved via tool definitions (tool use docs). Production rule regardless of provider: validate against the schema on receipt, retry on mismatch — the validation layer is yours, not the vendor's. (You met this pattern in Lesson 0004: chunk-context generation and citation IDs are structured outputs too.)

4 · Tool calling — the model asks, your code acts

Tool calling (= function calling) inverts the flow: you describe functions to the model (search_cases(query), get_document(id)), and the model responds with a request to invoke one; your code executes it and feeds the result back; the model continues. That loop is what "multi-step AI workflows" means mechanically (OpenAI guide, Claude guide).

The security sentence interviewers wait for Tool arguments are untrusted input — the model may have been steered by prompt injection hidden in a document (Lesson 0003). So tools execute under the app's tenant-scoped permissions with validated arguments — never raw. The model proposes; your code, inside the same RLS/IAM boundaries as everything else, disposes.

5 · Workflows vs agents — and where the human sits

Anthropic's taxonomy (Building Effective Agents) is the shared vocabulary: workflows are LLM calls orchestrated through your predefined code paths (prompt chaining, routing, orchestrator-workers, evaluator-optimizer); agents are loops where the model itself directs its next step. Legal work wants predictability, so the senior answer is: workflows by default, agentic loops only where open-endedness earns its risk — research, not filing.

Human-in-the-loop (HITL) is the non-negotiable for legal: every AI output is a proposal, persisted with a status (draft → under review → approved/edited → released), and a lawyer approves before anything leaves the system. Design implication worth saying aloud: HITL is not a popup — it's a state machine in your schema, with the review queue as a first-class feature, edit-diffs captured (they become eval labels — next lesson), and the audit table from Lesson 0003 recording who approved what, when.

Whiteboard drill — out loud, in English Close this page and narrate for 90 seconds: "Design the feature where a lawyer asks for a first draft of an NDA response." Hit: route to the right model tier → RAG context from the firm's corpus (Lesson 0004) → generation with structured citations → validation → the HITL state machine → what gets logged. Bonus: what happens when the primary provider is down mid-draft.

6 · Prove it — retrieval quiz

Primary source

Read Anthropic's "Building Effective Agents" (~20 min). Its workflow taxonomy (prompt chaining, routing, orchestrator-workers, evaluator-optimizer) is the exact vocabulary for the "agent workflows" line of the job post — and its closing advice (simplest pattern that works) is the senior posture for this interview.

Questions? Anything fuzzy — how a circuit breaker actually behaves, what a tool definition looks like on the wire, how edit-diffs become training signal — ask your teacher in the session. Next: Lesson 0006 — LLM evaluation, the most differentiating topic of the whole interview.