← Back to selected projectsTECHNICAL CASE STUDY · BREAK THROUGH TECH AI STUDIO
Financial Document Q&A:
Grounding answers in evidence.
A FinanceBench-based question-answering project combining retrieval, passage reranking, and Qwen2.5-3B-Instruct generation. My focus was understanding the dataset, integrating answer generation, and handling cases where retrieved text did not support a reliable answer.
PythonFinanceBenchTF-IDF + FAISSCross-EncoderQwen2.5-3B-InstructRAG
01 / THE PROBLEM
Finding a passage isn't the same as proving an answer.
Financial questions often depend on specific companies, fiscal years, document sections, and accounting line items. A retrieval system might surface a generally relevant passage without the exact fact needed to answer. The engineering challenge was to connect retrieved evidence to generation without automatically treating relevance as correctness.
1 · RetrieveSparse and dense search bring back candidate filing passages
2 · RerankA cross-encoder scores query–passage relevance
3 · Evaluate evidencePrototype threshold plus instructions to refuse unsupported answers
4 · GenerateQwen answers from supplied passages or abstains
Team vs. individual work: This diagram describes the shared pipeline. My individual implementation, analysis, and testing contributions are detailed below.
02 / ENGINEERING DECISIONS
Why these components?
Why sparse + dense retrieval?
Sparse search is useful for exact company names, dates, and accounting terms; dense search can also find semantically similar language. The team pipeline used both to widen the candidate pool before reranking. This is the design rationale, not a claim of a measured accuracy gain.
Why a cross-encoder?
It scores a question together with each candidate passage, helping prioritize contextual relevance after initial retrieval. My work used those scores in an evidence gate and investigated cases where high relevance still failed to indicate answer support.
When should the model abstain?
I implemented a configurable score threshold and strengthened Qwen's system prompt to decline answers that retrieved passages could not substantiate. Testing showed the threshold alone was insufficient, so the prompt-level evidence check mattered.
How did I avoid evaluation leakage?
I reviewed a 150-question FinanceBench sample and split it by source document, rather than splitting question rows randomly. That kept multiple questions from the same filing out of both partitions. This was a dataset split for development and held-out evaluation—not evidence that I trained or fine-tuned the pretrained Qwen or retrieval models.
03 / MY CONTRIBUTIONS
What I implemented and tested.
- Dataset investigation: reviewed 150 question–answer examples from 84 documents across 32 companies; examined question types, evidence counts, missing data, and duplicates.
- Leakage-aware split: created a document-level partition of 124 questions / 67 documents labeled “train” (for development) and 26 questions / 17 documents labeled “test” (held out for evaluation), with no documents shared across sets. These labels describe the dataset partition; I have not documented model training or fine-tuning on the 124 questions.
- Answer generation: integrated Qwen2.5-3B-Instruct into the existing hybrid retrieval and reranking workflow; strengthened the prompt to answer only from provided evidence.
- Abstention logic: implemented a configurable cross-encoder score cutoff and fallback for weakly retrieved evidence; explored the interaction between score gating and prompt-level sufficiency checks.
- End-to-end debugging: built a missing local dense index, resolved environment issues, and ran manual supported/unsupported test cases on a Mac using MPS acceleration.
04 / REAL TEST CASES
The surprising failure mode.
Supported question: 3M FY2018 capital expenditures
The retrieved cash-flow evidence included the “Purchases of property, plant and equipment” line, relevant to the benchmark answer of $1,577 million. However, the highest cross-encoder score was −2.548, so an initial threshold of 0.0 rejected even this supported example.
Takeaway: a strict score threshold caused a false abstention on a relevant example.
Unsupported question: Apple's 2025 iPhone revenue
This question was unsupported by the available documents in the test. The top reranker score was approximately −2.003—higher than the supported 3M example's score—even though the retrieved passages did not justify the requested answer.
Observed final response: “I can’t answer this question from the available documents.”
What changed: I tested a −3.0 prototype score threshold alongside a stricter Qwen instruction to refuse unsupported answers. Both cited questions would pass that score cutoff, so the successful Apple refusal depended on the additional evidence instruction. This was manual scenario testing, not a validated abstention-accuracy benchmark. The excerpt above identifies relevant evidence but is not a verbatim retrieval log.
05 / TAKEAWAYS
What this taught me about reliable AI systems.
Retrieval relevance and answer support are different problems. A model can receive passages that look related yet still lack the exact information needed. Combining ranking, explicit evidence constraints, and realistic negative tests made that difference visible.
There is more to evaluate: automatic benchmark coverage, calibration over a larger question set, and measured comparisons between sparse-only, dense-only, and hybrid retrieval. I do not present those as completed results.