Built a source-grounded research assistant for asking questions across multiple PDFs, with every generated answer tied back to the supporting filename and page number.
Retrieval Architecture
CiteCraft partitions narrative text and tables with Unstructured while preserving source metadata. Each full element is stored as a parent document in Redis, while a compact summary is embedded and indexed in PostgreSQL with pgvector. At query time, cosine similarity over an HNSW index finds relevant summaries and their parent IDs, allowing the answer model to receive the complete original evidence instead of a shortened retrieval representation.
Grounding and Citations
- Preserves filenames, page numbers, element types, document hashes, and stable parent IDs
- Filters retrieval to the PDFs active in the current Streamlit workspace
- Builds numbered source labels into the model context for page-level citations
- Returns an explicit no-answer response when the indexed documents do not support a claim
- Deduplicates repeated uploads through SHA-256 content hashes
Evaluation
Evaluated the system on 32 human-annotated FinanceBench questions spanning 10 financial reports, 403 PDF pages, and 1,154 extracted elements. The recorded run measured 85.9% grounded responses, 84.4% citation completeness, 79.7% citation correctness, and 90.0% abstention accuracy when the source was withheld. Retrieval latency was 0.46 seconds at P95 and end-to-end latency was 7.09 seconds at P95.
Engineering and Operations
The application uses environment-backed configuration, concurrent summary generation, context-size guardrails, and configurable PDF partition strategies. Docker Compose provisions PostgreSQL with pgvector and Redis locally, while Pytest, Ruff, and GitHub Actions provide automated quality checks.