Handwritten Document RAG

Figures & Results

Benchmark

Test-split CER against published results

Architecture

Document-to-answer pipeline

Progression

Bakeoff CER over rounds

Project Overview

The core system is an agent that structures PDFs for retrieval-augmented generation. Printed-text OCR handles typed pages, but handwriting it cannot, and there was no off-the-shelf HTR model that met the project's accuracy, deployment and integration requirements. So I treated the ingestion gap as a machine-learning problem in its own right: a CRNN-CTC recognizer trained from scratch on Brazilian-Portuguese handwriting, plugged in behind the same interface as the existing OCR and selectable at runtime. Nothing downstream needs to know whether a page was typed or handwritten.

Main Products / Deliverables

Key Questions Answered

• Can a competitive Portuguese handwriting reader be trained with no pre-existing model and no local GPU?
• Which architecture actually wins - decided by a metric on the official split, not by intuition?
• How does a research model become a swappable component of a cloud pipeline?

Methodology - how the model was selected

The evaluation protocol was fixed before model comparison. CER is aggregated across the corpus, with edits and lengths summed before dividing, following the ICDAR 2024 convention. The two numbers are kept apart on purpose. 9.57% is validation CER, used only for model selection; the benchmark figure is 10.35% CER on the official BRESSAY test split (5,916 lines), measured once after the model was frozen. Reporting a validation number against published test results would have flattered the model by roughly 0.8 points.

1. Fixed the ground rules. Official BRESSAY splits (19,565 train / 4,609 validation lines), never a custom partition. The alphabet is built from the training split only, keeping 95 characters that clear a minimum-frequency cutoff, plus the CTC blank, for 96 output classes. It travels with the checkpoint, because desynchronized indices would make the model read another language. A character outside this set cannot be emitted, so rarer symbols appearing elsewhere in the corpus are counted as errors rather than quietly excused.
2. Framed a bakeoff with a promotion gate. Each architecture trains on the same data and is ranked by validation CER; a challenger only replaces the baseline if it beats that CER. The baseline is CRNN + CTC with greedy decoding, which avoids an autoregressive language decoder and keeps the transcription tied to the visual signal. The challenger is TrOCR, a pretrained vision encoder-decoder. To date it has only been run as a one-epoch smoke test proving the training path works end to end; it has not been fine-tuned to convergence and has not cleared the gate.
3. Trained with stable, cheap choices. Adam (lr 1e-3), ReduceLROnPlateau on validation CER, gradient clipping at 5.0, batch 24, and - critically - the checkpoint saved is the best epoch, not the last. Resumes are anchored to the loaded model's CER so a noisy first epoch never overwrites a good checkpoint.
4. Used training diagnostics to separate optimization instability from overfitting. The CER oscillated in epochs 8 to 10, which pointed to a learning rate that was too high rather than to overfitting. The final configuration combined a larger CRNN with a lower learning rate (1e-3→5e-4), reducing validation CER from 15.40% to 9.57%. Both the capacity and the learning rate changed between those runs, so the gain is not attributable to the learning rate alone.
5. Evaluated at three levels. Line CER guides training; the line segmenter's count is checked against the ground truth; and page-level CER is measured end-to-end - the number the product actually feels.

How the Technologies Were Used

• PyTorch - the CRNN reader (CNN feature extractor → bidirectional LSTM) and a custom training loop with best-checkpoint saving, LR scheduling, and anchored resume.
• CTC loss - alignment-free line recognition with greedy decoding, so the transcription depends on the visual signal rather than on a learned language model over previously emitted tokens.
• Hugging Face Transformers / TrOCR - the challenger path, wired for GPU fine-tuning with AMP/fp16, LR warmup and cosine decay, and gradient accumulation. Exercised so far only as a one-epoch smoke run.
• PyMuPDF - PDF parsing and layout structuring into ordered text blocks.
• Tesseract - the printed-text OCR backend, exposed through the same contract as the HTR reader.
• Qdrant + FastAPI + Claude - vector store for semantic retrieval, service layer, and an agentic RAG loop that answers over retrieved blocks and cites the source.
• pytest & Colab - 105+ tests (including hermetic and smoke coverage) and a parametrized notebook for the managed GPU training run.

Technologies Used

Getting Started

git clone https://github.com/Calazansmacchiutti/Stratifile-Agent-PDF.git
cd Stratifile-Agent-PDF
pip install -e .

# validate the whole HTR pipeline end-to-end on CPU (seconds)
python scripts/train_trocr.py --smoke

# train the CRNN baseline locally on CPU
python scripts/train_bressay_local.py --model crnn-large --epochs 12

# fine-tune the TrOCR challenger on GPU (Colab/Kaggle)
python scripts/train_trocr.py --epochs 8 --batch 8

Repository

Results & Deployment

A compact 4.3M-parameter CRNN trained from scratch on Brazilian-Portuguese handwriting, reaching 10.35% line CER on the BRESSAY test split and delivered as a 17 MB versioned artifact that serves on CPU, with no GPU needed for inference. On the same split it comes in ahead of Bluche (18.02%), marginally behind Flor (10.26%), and behind Puigcerver (8.94%) and the ICDAR 2024 winner PERO (2.88%). The gap between validation and test is 0.78 points, which suggests the model generalizes rather than having been tuned into its validation set. It is a deliberate trade-off: small enough to ship inside the pipeline rather than built for a leaderboard. Because every stage is isolated by contract, the recognizer can be swapped - the moment the TrOCR challenger clears the gate - without touching the rest of the pipeline. The result is less a single model than a repeatable method for turning scanned documents, typed or handwritten, into retrievable and cited answers.