A retrieval system is useful when the answer shows the page it came from. Without that, you have a fluent guess wearing your documents as a costume.
Chunk for the way people ask
Policies, contracts, and manuals are not one embedding. Split them on headings, keep the title and the section name on each chunk, and store the page or clause number. A question about refunds should retrieve the refund section, not a similar sentence from the introduction. If you cannot point at a location, do not put that chunk in the index.
Refuse when the file is silent
The model should be allowed to say the documents do not contain the answer. That sentence is a feature. A system that always answers will eventually invent a clause, and the person who trusted it will send that clause to a customer. Log the retrieved passages next to the reply so a reviewer can see the gap.
Access control is part of retrieval
If two clients share a deployment, one client’s files must not answer the other’s question. Filter the index by the tenant before you measure similarity. A clever prompt is not an access boundary. The query should be incapable of seeing the other set of chunks.
Return the passage, not only the paragraph
The model is allowed to quote from the chunks you retrieved and from nowhere else. The response you store includes the answer and the list of locations. A location is the document id, the title, and the heading or page you kept on the chunk. If the list is empty, the answer is the refusal, and you still store the question.
Filter by tenant in the query that fetches chunks, before you rank them. A later instruction in the prompt cannot put that fence back if the other tenant’s text is already in the context.
Python · retrieval/answer.py
def answer_from_docs(tenant_id: int, question: str) -> dict:
chunks = (
Chunk.objects.filter(document__tenant_id=tenant_id)
.search(question)[:5]
)
if not chunks:
return {
"answer": "The documents do not contain that.",
"citations": [],
}
citations = [
{"document": c.document.title, "location": c.heading, "id": c.id}
for c in chunks
]
return {"answer": draft_from(question, chunks), "citations": citations}How you know a citation is real
Take questions you already know the answer to, from the documents, and questions you know the documents do not answer. The first set should cite the section you expected. The second set should refuse. A system that answers both with equal confidence is not ready for a customer.
Keep the heading and the page on the chunk at index time. A chunk that is only a bare sentence cannot be shown to a reviewer, and the model will happily quote it without a location. If you cannot point at the location in the product, do not retrieve that chunk.
When two organisations share a deployment, the test is not a prompt that says “ignore other tenants.” The test is a query as tenant A that would match a distinctive sentence you planted in tenant B’s file. Zero of those chunks may come back. If one does, the filter is in the wrong place, and no instruction in the prompt repairs it.
Show the citation in the product as a link to the document and the heading, not as a footnote the model invented. If the reviewer cannot open that heading and see the sentence, the answer does not ship. A citation that does not open is decoration.
Chunk on headings and keep the overlap small. A chunk that runs across two policies will cite the wrong title. A chunk that is a single short line will match too many questions. The heading you stored is the heading you show, so the split and the citation are the same decision.
On the next document set
- Attach a source location to every chunk you index.
- Add an explicit “not in the documents” reply.
- Filter retrieval by tenant before similarity.