rollroyces/py-idp

What it claims

py-idp is a Python package (pip install py-idp) that turns PDFs, images and text into Pydantic-validated dicts with per-field confidence. Fields under 0.6 confidence go to a Streamlit review UI. The core install runs with no API key on an in-tree MockBackend, and the repo reports 654 tests across Python 3.10 to 3.12. Its distinctive piece is a "policy update" that reads human corrections and raises confidence penalties on fields reviewers keep fixing, so those fields reach review more often. The author states this is a deterministic rule update, not a learned reward model or a fine-tuned LLM.

How it works

Each stage is a pure function over a Document. Classification is rule-first with an LLM fallback. Routing then chooses one of two extraction paths: page images sent to a VLM, or parsed text (Docling or pdfplumber) sent to an LLM. Every provider implements one Backend protocol (backend.complete(CompletionRequest)). Retry and an SQLite extraction cache wrap that protocol, and the cache key is a sha256 of schema, backend and payload.

flowchart LR
    A[extract + assess] --> B{field conf < 0.6?}
    B -->|no| C[(result)]
    B -->|yes| D[Streamlit review]
    D --> E[review_edits table]
    E --> F[idp rl-update<br/>min_reviews=10]
    F --> G[policy.json<br/>field penalties]
    G --> A

Auto-chunking is triggered by size. Multimodal input over 4 pages uses PageChunker (4 pages, 1 overlap). Text over 4000 tiktoken tokens uses TokenChunker (4000 tokens, 200 overlap). Per-chunk results are merged, then validated as a whole. A failed chunk is logged as extract_chunk_failed[i] and the other chunks are still kept. Markdown templates in templates/*.md are prepended to every prompt to carry field guidance that JSON Schema cannot express. The template name and version are recorded on the result for audit.

Evidence

The README says it shows measured runs on 3 in-tree invoices with Ollama qwen2.5:0.5b, but the result tables do not appear in the fetched text. The author's own caveat on the policy result: the sample is 27 (field, doc) pairs, and the reviews are synthetic, derived from gold labels and so "biased optimistic". The 0.5B model's confidence was so pessimistic that the gain "looks dramatic", and a better-calibrated model would gain less. The eval fixtures are small: 3 invoices, 2 contracts and 5 CORD-style receipts.

How to use it

pip install py-idp[docling,anthropic,eval,dev]
idp run invoice.pdf --schema Invoice --backend openai
idp batch /mnt/inbox/ --backend ollama --schema Invoice \
          --output out.jsonl --report report.json --dlq dlq.jsonl \
          --checkpoint ledger.jsonl
idp eval --dataset src/idp/eval/datasets/invoices \
         --strategy mock,mock-omits,ollama --output results.json
idp rl-update --storage idp_data/results.jsonl --output policy.json

Any Pydantic model works as schema=. The package ships Invoice, Contract, BankStatement, W2, IDCard and MedicalClaim. check_regression() fails CI on metric thresholds. AsyncPipeline and process_batch_async(concurrency=N) serve asyncio contexts, but acomplete defaults to asyncio.to_thread, so no backend is natively async yet. Retry defaults to 4 attempts (1s, 2s, 4s, 8s, ±20% jitter, capped at 30s). Auth and bad-request errors are not retried.

The self-hosted Nanonets-OCR2-3B backend needs py-idp[hf-vlm], a download of about 7 GB, and IDP_ENABLE_NANONETS=1. On an Apple M4 with 16 GB it uses about 8.3 GB and takes 5 to 15 s per page. The unlimited-ocr backend handles 32K-token multi-page context in one pass. discover_schema() asks a VLM to propose a schema from a hint and reports a grounding_score that shows which hint fields the model ignored.

Limits

• AGPL-3.0-or-later: a modified version deployed as a network service must be published. Closed or SaaS use needs the commercial licence, and its pricing is not in the fetched text.
• Third-party model licences: the Nanonets fine-tune licence must be verified before commercial use. unlimited-ocr loads with trust_remote_code=True.
• Version labels conflict: the README mentions v0.5 and v0.6 features, but idp.__version__ reports 0.4.0.
• Screenshots are mockups: the author labels the HITL screenshots as illustrative.
• Error classification is fragile: retry decisions rely on matching error message text.
• Single-tenant only: multi-tenant isolation, SSO/SAML/RBAC and audit-grade storage are not provided.
• One process for live updates: only one process may hold the online PolicyCache.