What is retrieval in AI? Definition and examples

Retrieval is the step where an AI system searches a document collection for the passages most likely to answer a question, before a model writes the answer.

How it works

Retrieval is the "R" in retrieval-augmented generation (RAG). The 2020 paper that named RAG paired a language model with a "non-parametric memory": an index of documents searched by a neural retriever, so the model could look facts up instead of relying only on what it learned in training (Lewis et al.).

A typical retrieval pipeline has two halves.

Before any question: documents are split into short passages by chunking. Each passage is indexed twice in many systems: as a vector embedding for search by meaning, and in a keyword index such as BM25 for exact words.

When a question arrives:

1. The question is embedded and compared with the stored vectors to find the closest passages.
2. A keyword search runs in parallel, which catches exact terms like product codes or error messages.
3. The two result lists are merged, and a reranking model may reorder them.
4. The top few passages go to the language model with the question.

Small improvements here matter. Anthropic reported that adding a short, generated context line to each chunk before indexing cut the rate of failed top-20 retrievals by 35% with embeddings alone, 49% when combined with keyword search, and 67% with reranking added, on its test sets. The same article notes that a knowledge base under about 200,000 tokens can sometimes skip retrieval and go into the prompt whole (Anthropic).

Why it matters for your website

An AI assistant on your site can only be as good as its retrieval. If the right passage is not found, the model either answers from the wrong one or, if it is well built, says it does not know. Many "wrong answers" are retrieval failures, not model failures.

You can help retrieval with your content:

• Give each section one topic and a heading that says what it is.
• Repeat the subject in the passage ("The Team plan includes SSO"), not just "It includes SSO".
• Put facts in text, not only in images, scanned PDFs or tabs that load late.
• Use the words visitors use alongside your product names.

Example

A visitor asks, "Do you support single sign-on on the Team plan?" Keyword search finds the security page passage that mentions "SAML". Search by meaning finds the pricing passage "Team: SSO with Okta and Azure AD". Both go to the model, which answers "Yes, on the Team plan" and links the pricing page. Without the second passage, the model would only know that SSO exists, not which plan includes it.

Related terms

• Retrieval-augmented generation (RAG): the pattern retrieval is part of.
• Semantic search: finding passages by meaning.
• Vector embedding: the representation that makes meaning searchable.
• Reranking: a second pass that reorders retrieved passages.
• Chunking: how documents are split before indexing.
• Knowledge base: the collection retrieval searches.