Document Q&A is a search system with an answer layer. If the right evidence is not retrieved, a better prompt cannot recover it. If permissions are wrong, a fluent answer becomes a data incident.
Start with a narrow collection, explicit ownership, and questions whose answers you can verify. Build source visibility into the interface from the beginning.
Prepare documents as governed evidence
Preserve document title, owner, version, effective date, permissions, and a stable source location. Remove obsolete duplicates or label them clearly. Decide whether the system should answer from current policy only or expose historical versions.
Extract text without destroying structure. Keep headings with their paragraphs, captions with figures, and tables as coherent units. OCR scanned documents, then inspect a sample because silent OCR errors can change identifiers and amounts.
Retrieve before generating
Split content by meaning and document structure rather than a fixed character count alone. Index chunks with metadata. At question time, retrieve a broad candidate set, apply filters and permissions, rerank for relevance, and send only the strongest evidence to the model.
Require citations that link back to the exact document and location. Show the supporting passage beside the answer. Instruct the model to say when the retrieved evidence is insufficient or conflicting.
- Test keyword search and semantic search together.
- Preserve metadata filters for date, product, jurisdiction, and permission.
- Rerank before expanding the final context.
- Log the retrieved passage IDs with the generated answer.
Evaluate retrieval and answering separately
Create answerable questions, unanswerable questions, questions with conflicting sources, and questions that should be blocked by permissions. First measure whether the necessary passage appears in retrieval. Then measure whether the answer uses it correctly.
A safe refusal is a passing result when evidence is absent. A plausible uncited answer is a failure, even if it happens to be true.
The interface should make it easier to inspect the evidence than to trust the prose.
Primary sources