Primary question: Does your pipeline waste OCR budget on text-based PDFs that could be parsed directly?
RepoDaily adoption score
RepoDaily rates this as 94/100 (strong) for adoption: evidence, installation path, production risk, differentiation, license clarity, and AI/agent fit are scored from the article sources and adoption notes.
5 source(s) across 3 source category/categories, plus a RepoDaily-specific evidence module when available.
5 workflow step(s), 4 next-action step(s), and 5 command/install signal(s) were detected.
Trending momentum is +1,769 stars, with maintenance/release/issue signals counted when present.
Risk is marked low, with 4 security note(s) and 3 explicit skip condition(s).
3 opportunity lens item(s), 4 alternative(s), and 3 type-specific section(s) support differentiation.
License source or license wording is present.
4 AI/agent-related signal(s) were detected in the article text and metadata.
Project overview
pdf-inspector is a Rust library from Firecrawl that inspects PDF files and classifies them as either scanned image documents or native text-based documents. That classification matters because the two document types require fundamentally different extraction strategies: text-based PDFs yield their content through direct parsing, while scanned PDFs demand OCR or vision-language models that are orders of magnitude more expensive. By making the routing decision cheap and predictable, pdf-inspector sits at the front of ingestion pipelines and prevents wasted compute.
The project ships as three distinct artifacts from a single codebase. The Rust crate (pdf-inspector 0.1.7 on Cargo) exposes the core library API. A Python wheel (pdf-inspector 0.2.6 on PyPI) is built through maturin with pyo3 0.25 bindings and supports CPython 3.8+. Three command-line binaries — pdf2md, detect-pdf, and dump_ops — cover batch extraction, classification checks, and low-level operator inspection respectively.
Under the hood, pdf-inspector relies on lopdf 0.42.0 for PDF structure parsing. Native builds enable lopdf's rayon-based parallel parser for speed, while WASM builds disable rayon and use lopdf's wasm_js feature so the library runs in browsers without cross-origin isolation. The crate bundles external CMap data (external/bcmaps) for CID font decoding, including Identity-H cmap extraction through ttf-parser 0.25, which means it can handle CJK and other CID-keyed fonts that often break naive extractors.
Why it is trending now
- 1769 stars in the trending window, ranking #3, driven by demand for cheaper document routing ahead of vision-model OCR.
- Multi-target build from one codebase: Rust crate, Python wheel via maturin, and browser-compatible WASM artifact.
- Three practical CLI binaries (pdf2md, detect-pdf, dump_ops) make the library usable without writing any Rust code.
- MIT license and a clearly scoped SECURITY.md lower the barrier for enterprise adoption reviews.
Problem it solves
- OCR and vision-language model calls are expensive; sending a text-based PDF through them wastes budget and adds latency.
- Many PDF extractors silently produce empty or garbled output when they encounter scanned documents, causing silent pipeline failures.
- CID-keyed fonts (Identity-H) and CJK CMaps break naive text extraction tools, producing mojibake or missing characters.
- Browser-based PDF tooling often cannot rely on filesystem access or multi-threading, limiting where extraction logic can run.
How it works
- A PDF is passed to pdf-inspector either as a file path or as an in-memory buffer.
- lopdf parses the cross-reference table and page tree; on native targets, rayon parallelizes structure traversal for throughput.
- The classifier inspects each page for embedded text operators and font resources versus image-only content streams, producing a scanned-vs-text verdict.
- For text extraction, unicode-normalization and the bundled bcmaps CMap data resolve CID-keyed fonts, while ttf-parser handles Identity-H cmap tables from embedded TrueType fonts.
- On Python targets, pyo3 0.25 with abi3-py38 exposes the same functions as native extension methods; on WASM targets, lopdf's wasm_js feature provides randomness for encrypted PDFs and include_dir bundles CMaps.
Integration Surface: Three Targets, One Crate
pdf-inspector compiles to a Rust library crate (pdf_inspector, crate-type lib + cdylib), a Python wheel built by maturin, and a WASM-compatible build. The Cargo.toml conditionally selects dependencies: native targets get lopdf with the rayon feature and env_logger, while wasm32 targets get lopdf with wasm_js and include_dir for embedded CMaps. This means you can embed the same classification logic in a Rust service, call it from a Python data pipeline, or run it in a browser without changing the core code.
Command Surface: pdf2md, detect-pdf, dump_ops
- pdf2md (src/bin/pdf2md.rs): Converts PDF content to Markdown text, intended for extraction pipelines.
- detect-pdf (src/bin/detect_pdf.rs): Runs classification and reports whether a PDF is scanned or text-based.
- dump_ops (src/bin/dump_ops.rs): Dumps low-level PDF content stream operators, useful for debugging extraction failures.
Maintenance and Dependency Risk
The project pins lopdf 0.42.0, pyo3 0.25, ttf-parser 0.25, and thiserror 2.0 — all actively maintained Rust ecosystem crates. The SECURITY.md explicitly excludes bugs in upstream dependencies like lopdf from its own scope, directing reporters upstream. The crate version (0.1.7) and Python version (0.2.6) diverge, which is normal for maturin projects but worth tracking when pinning. The Cargo.toml include allowlist notes that test fixtures exceed crates.io's 10 MiB upload cap, so the published crate is intentionally lean.
Who should pay attention?
Good fit if
- Document ingestion pipelines that currently send every PDF through OCR regardless of content type.
- RAG and search systems that need clean text extraction before embedding.
- Edge or browser-based PDF tooling where filesystem access is unavailable and WASM is required.
- Teams that want a fast pre-filter before invoking GPT-4V or Claude for scanned document understanding.
Skip for now if
- Projects that need full layout reconstruction, table structure recognition, or reading-order detection — pdf-inspector focuses on text extraction and classification, not layout.
- Teams that require a pure-Python dependency tree with no Rust build step; the wheel is pre-built but building from source needs a Rust toolchain.
- Use cases where PDF forms field extraction (AcroForm/XFA) is the primary requirement.
Risks and cautions
MIT-licensed, narrowly scoped, and built on established Rust crates. The main integration cost is the Rust toolchain for source builds or pulling the pre-built Python wheel.
- MIT license permits commercial use, modification, and redistribution with minimal restriction.
- Security policy clearly defines in-scope (memory safety, DoS, binary bugs) and out-of-scope items (upstream lopdf bugs, extraction quality).
- Python wheel targets abi3-py38, covering Python 3.8 through 3.13+.
- WASM build path is explicitly maintained, not accidental.
- Crate is pre-0.2 and Python package is pre-0.3, so breaking API changes between minor versions are possible.
- SECURITY.md defines in-scope vulnerabilities: memory-safety issues (panics, OOB reads, UB), DoS vectors (unbounded allocation, infinite loops), and bugs in pdf2md/detect-pdf/dump_ops or the pdf-inspector crate.
- Vulnerability reporting is via private email (help@firecrawl.dev) or GitHub's private vulnerability reporting under the Security tab.
- Out of scope: upstream dependency bugs (lopdf) and extraction quality issues (wrong text, missing tables) — these should be regular GitHub issues.
- The request for a minimal triggering PDF or version/commit hash in reports indicates a serious triage process.
Alternatives to compare
| Approach | When to use | Trade-off |
|---|---|---|
PyMuPDF (fitz) | You need a mature, full-featured Python PDF library with rendering, annotation, and layout support. | AGPL license or commercial license; larger footprint than pdf-inspector. |
pdfplumber | Your priority is table extraction and precise layout information in Python. | MIT; pure Python but slower on large corpora. |
pdfminer.six | You need detailed text extraction with font and position metadata in Python. | MIT; pure Python, no Rust dependency. |
Apache Tika | You need a JVM-based server that handles hundreds of document formats beyond PDF. | Apache 2.0; heavier operational footprint. |
What this trend reveals
Pre-OCR Cost Gate for Vision Pipelines
Running pdf-inspector's detect-pdf before invoking GPT-4V or Claude on a document batch can cut vision API spend by filtering out text-based PDFs that cost cents to parse but dollars to OCR.
Run detect-pdf on a representative 1,000-PDF corpus, measure the scanned ratio, and multiply the text-based portion by your per-document vision API price.
Browser-Native PDF Classification
The WASM build path with lopdf wasm_js and embedded CMaps enables client-side PDF triage in edge or browser environments without a backend round-trip.
Build the wasm32 target from the Cargo.toml configuration and load it in a browser test harness with a sample mixed corpus.
Debugging Extraction Failures with dump_ops
The dump_ops binary exposes raw PDF content stream operators, which is valuable for diagnosing why a specific PDF produces empty or garbled text.
Run dump_ops on a known-problematic PDF and compare the operator stream against expected text-showing operators.
RepoDaily verdict
pdf-inspector solves a specific, expensive problem — knowing whether a PDF needs OCR — with a fast, multi-target Rust core, clean security boundaries, and an MIT license. For any team routing documents through vision models, it is worth a one-hour evaluation.