Ship One LLM Feature Without Rewriting Your App

Most teams add their first LLM feature the wrong way. They stand up a new service, pull in an orchestration framework, add a vector database because a blog post said they'd need one, and three weeks later they have a demo that works on the happy path and falls over on everything else. The feature itself, once you strip the ceremony away, was usually a single API call wrapped in a form.

The good news: adding a model-backed feature to an existing app is closer to integrating a flaky third-party API than to building a new subsystem. Treat it that way and most of the difficulty disappears.

Start from the seam, not the model

Before you write a prompt, find the exact place in your app where the feature lives. A support tool that drafts reply suggestions has a natural seam: the moment an agent opens a ticket. A CMS that generates alt text has one too: the image upload handler. Name the input you already have and the output you need. If you can't describe both in one sentence without saying "and then the user can also," you're scoping two features.

Write the function signature first, with no model in it:

async function suggestReply(ticket: Ticket): Promise<ReplyDraft> {
  // returns a draft the agent can accept, edit, or ignore
}

This forces two useful decisions early. What context does the feature actually need (just the ticket body, or the customer's order history too)? And what does the caller do with a bad result? If the honest answer is "the agent reads it and decides," you have a human in the loop and a much wider tolerance for imperfection. If the answer is "it gets emailed automatically," you've signed up for a far harder problem and should probably reconsider.

The whole feature is often 40 lines

Here is a complete, production-shaped version of that function against a chat-completions style API. No framework.

import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();

const SYSTEM = `You draft support replies for Acme's billing team.
Rules:
- Only use facts from the ticket and account summary provided.
- If the refund policy isn't in the context, say you need to check, don't guess.
- Keep it under 120 words. Plain, warm, no marketing.`;

export async function suggestReply(ticket, account) {
  const res = await client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 400,
    system: SYSTEM,
    messages: [{
      role: "user",
      content: `Ticket:\n${ticket.body}\n\nAccount summary:\n${account.summary}`
    }]
  });
  return { draft: res.content[0].text, model: "claude-sonnet-4-5" };
}

That is the feature. Everything else you add should earn its place by fixing a problem you actually observed, not one you imagined.

Pick the model per call, and make it swappable

You do not need the largest model for most features. Classification, extraction, short rewrites, and routing run fine and far cheaper on a mid-tier model. Save the frontier model for open-ended reasoning or long, messy inputs. The practical move is to keep the model id in config, not inlined, so you can A/B two models against real traffic without a deploy:

const MODEL = process.env.REPLY_MODEL ?? "claude-sonnet-4-5";

Because provider APIs differ in small ways, put the call behind a thin adapter of your own from day one. Not a framework, just a function you control. When you want to try a different provider or add a fallback, you change one file. Teams that skip this and scatter raw SDK calls across the codebase pay for it later.

Structured output when you consume the result in code

If a human reads the output, free text is fine. If your code branches on it, ask for JSON and validate it. Most current APIs support a schema or tool-call mode that constrains the output; use it rather than parsing prose with a regex.

const Schema = z.object({
  category: z.enum(["billing", "bug", "feature", "other"]),
  urgency: z.number().min(1).max(5),
  needs_human: z.boolean()
});

// validate, and on failure retry once with the error appended,
// then fall back to a safe default rather than throwing at the user.
const parsed = Schema.safeParse(JSON.parse(res.content[0].text));
if (!parsed.success) return { category: "other", urgency: 3, needs_human: true };

The fallback matters more than the retry. Models occasionally return malformed output no matter how you prompt. Decide in advance what a broken response degrades to, and make that path safe.

The three things that actually bite in production

Latency. A frontier-model call can take several seconds, sometimes longer on long inputs. Never block a request handler on it synchronously if you can avoid it. Stream the tokens if the user is watching, or push the work to a queue and update the UI when it lands. A spinner that hangs for eight seconds reads as broken.

Cost that scales with usage, not with users. A single call is cheap. Ten thousand calls a day on your biggest model is a real line item. Log token counts per call from the start. Add a per-user or per-tenant rate limit before launch, not after someone loops your endpoint. Cache aggressively where inputs repeat: if two users summarize the same document, you should pay once.

Failure modes that don't look like errors. The API returns 200 and confidently wrong content. Your monitoring won't catch it because nothing threw. This is why the human-in-the-loop framing is worth defending. Where you can't keep a human, add a cheap validator: check that generated alt text isn't empty and isn't 500 words, that an extracted date parses, that a classification is one of your enum values. Cheap checks catch the embarrassing failures.

What to skip on version one

You almost certainly do not need a vector database, an agent loop, a fine-tuned model, or a framework to ship your first feature. Those solve real problems, but problems you haven't hit yet. Retrieval matters when the model needs knowledge it doesn't have and you can't fit it in the prompt. Agents matter when a task needs multiple dependent steps with branching. Fine-tuning matters when prompting genuinely can't get the behavior and you have the data to prove it. Reach for each when the simpler version demonstrably fails, and you'll know because you'll have logs.

Make it easy to turn off

Put the feature behind a flag. Log every input and output (scrubbed of anything sensitive) so you can inspect real behavior and build an eval set from actual traffic. Keep a static fallback for when the provider has an outage, which will happen. A drafting feature that silently returns nothing during an incident is fine. One that takes down the ticket queue because the call was on the critical path is a postmortem.

The measure of a well-added LLM feature is not how sophisticated it is. It's whether you could delete it on Friday and have a calm weekend. Build the boring version, watch how people actually use it, and add complexity only where the logs tell you to.