What Your AI Coding Assistant Actually Sees: Context, Retrieval, and Why It Invents APIs

The most useful mental model for an AI coding assistant is not "a robot that knows your codebase." It's a next-token predictor that only knows what fits inside a prompt you never see. Almost every surprising behavior — the great refactor, the fabricated method, the sudden amnesia about a file you edited two minutes ago — falls out of that one fact. If you understand how the prompt gets built and why prediction fails the way it does, you stop being surprised and start steering.

The prompt is assembled, not remembered

When you type a request in Cursor, Copilot, or Claude Code, the model does not "open" your project. A client-side layer builds a single text prompt and sends it to the model. That prompt is roughly: a system message describing the assistant's role and tools, some retrieved chunks of your code, the currently open file (or a window around your cursor), recent chat turns, and your instruction. The model sees that blob and predicts a continuation. That's the whole transaction.

This is why two identical questions can get different answers. The retrieval step picked different chunks. It's also why "it forgot what we just discussed" happens: older turns got evicted to make room. The assistant has no persistent memory of your repo between requests beyond what the tool re-sends each time.

Context windows: budget, not knowledge

A context window is measured in tokens — sub-word units where a line of code is often 8 to 20 tokens. In 2026 most coding models expose windows in the hundreds of thousands of tokens, and a few advertise a million or more. Treat those numbers as a ceiling, not a promise of quality.

Two failure modes matter in practice. First, cost and latency scale with tokens, so tools aggressively trim what they send even when the window is large. Second, models attend unevenly across a long context. Information at the very start and very end of the prompt tends to get used; material buried in the middle of a 400k-token dump gets underweighted. This is the "lost in the middle" effect, and it's still real in current models even as it's improved.

The practical consequence: dumping your entire monorepo into context is usually worse than sending three relevant files. You pay more, you wait longer, and you dilute the signal. When an assistant gives a vague answer about a large project, the problem is often too much context, not too little.

Retrieval: how code gets chosen

Because you can't fit a real codebase in the window, assistants retrieve. The common approach is to index the repo, split files into chunks, embed each chunk into a vector, and at query time embed your request and pull the nearest chunks by cosine similarity. Some tools blend this with keyword search (BM25-style) and with structural signals like the call graph or the file you currently have open.

Retrieval quality is where a lot of "the AI is dumb" complaints actually live. Embedding search matches on semantic similarity, not on correctness or recency. If your PaymentService was renamed to BillingService last week but the index is stale, the model retrieves the old chunk and reasons about a class that no longer exists. Chunk boundaries hurt too: a function split across two chunks may surface with its signature in one and its early-return guard in another, and the model stitches a plausible-but-wrong version.

You can influence retrieval more than you think. Naming the exact file or symbol pulls it in directly. Keeping the relevant file open biases most tools toward it. And giving a concrete anchor ("the function that calls stripe.charges.create") beats a vague description, because it gives both embedding search and keyword search something specific to hit.

Why it hallucinates APIs

Here is the mechanism, stripped of mysticism. The model was trained to predict the most probable next token given everything so far. It learned the shape of APIs: that HTTP clients tend to have .get() and .post(), that ORMs expose .findOne() or .find_by(), that a config object usually takes a timeout. When you ask for a method that should exist by that pattern, the model produces the most statistically natural name — whether or not the library actually ships it.

Consider a request to "retry the request three times with backoff" against a client library. The model has seen thousands of libraries with a retries option, so it writes:

const client = new ApiClient({
  baseUrl: process.env.API_URL,
  retries: 3,          // plausible
  retryBackoff: 'exponential'  // invented
});

Maybe retries exists and retryBackoff doesn't. Maybe neither does and retry is handled by a separate wrapper. The model has no ground-truth list of that library's surface at generation time; it has a prior over what such APIs usually look like. Confident tone is not confidence about facts — the fluency and the correctness are produced by the same process, so wrong code reads exactly as smoothly as right code.

Hallucination gets worse in predictable situations:

• Newer or niche libraries that were sparse in training data. The model leans harder on generic patterns.
• Version skew. The library moved a method between major versions; the model averages across versions it saw.
• Under-specified prompts. With little to constrain it, the model fills gaps with priors, and priors are where invention lives.
• Long generations. Once it has committed to a fabricated symbol early, it stays internally consistent with its own mistake.

What actually reduces invented code

The single highest-leverage move is to put the real API surface into context. If the tool can read the library's types, the actual function signatures, or a doc page, prediction gets anchored to fact instead of prior. Concretely:

• Feed it the types. In a typed language, an assistant that can see the .d.ts or the package's type stubs hallucinates far less, because the valid surface is right there in the prompt. This is a real argument for TypeScript, typed Python, or Rust in AI-heavy workflows.
• Point at the source or docs. Tools that support fetching a URL or reading node_modules let you say "use the actual signatures from this file." That single instruction removes a whole class of fabricated arguments.
• Close the loop with execution. The strongest guardrail is letting the assistant run the code, tests, or a type-check and read the error. A model that sees AttributeError: 'Client' object has no attribute 'retryBackoff' will correct itself. Agentic tools like Claude Code or Aider that compile and test in a loop catch their own hallucinations that a chat-only tool never would.
• Keep the index fresh. After large renames or dependency bumps, re-index. A stale vector store is a hallucination generator.

The working model to keep in your head

Think of the assistant as a very well-read pair programmer with no memory, who can only see the sticky notes you hand them, and who will always give you a fluent answer even when they're guessing. That framing makes the good practices obvious. Hand over the right notes (relevant files, real signatures). Keep the stack small enough to reason about. And never merge on fluency alone — the review step is not optional overhead, it's the part of the pipeline that supplies ground truth the model structurally cannot. Once you internalize that the prompt is assembled fresh every time and that invention is the price of generalization, the tool stops feeling magical and starts feeling like something you can drive.