Skip to content

feat(indexer): extract Office documents to Markdown with downmark - #225

Open
giraffesyo wants to merge 2 commits into
mainfrom
feat/downmark-office-extraction
Open

feat(indexer): extract Office documents to Markdown with downmark#225
giraffesyo wants to merge 2 commits into
mainfrom
feat/downmark-office-extraction

Conversation

@giraffesyo

@giraffesyo giraffesyo commented Aug 20, 2026

Copy link
Copy Markdown
Member

What

Office documents (.docx, .pptx, .xlsx, and now .doc) extract to Markdown through @giraffesyo/downmark (pure-Go converter compiled to wasm, bundled as a server npm dependency). The Python Office readers are gone. PDFs are unchanged (pdftotext → OCR).

  • server/src/preextract.ts — walks the scope, converts stale/missing entries into $INDEX_BASE/extract/<rel>.txt, runs as a forked child (runPreExtract) from indexing.ts before each enrich.py pass, and as a CLI from reindex.sh. Entries refresh once per downmark version (extract/.downmark-version).
  • enrich.py reuses the cache as before. python-docx / python-pptx / openpyxl are removed; the standard-library OOXML pass remains as the last resort for a document downmark rejects. .doc added to the extracted suffixes.
  • Deploy: the pip/apt lines leave app.def (proof step now checks pdftotext, tesseract, soffice and the downmark wasm); the per-host venv step and its PYTHON_BIN exports leave workflow.yaml (reindex.sh and the server already locate a Python 3.10+); requirements.txt keeps only the optional pypdf.
  • Viewer renders the extracted Office text with Streamdown; extractor report lists downmark.

Why

  • Office extraction no longer depends on anything on the host — no pip, no venv, no "the apt line silently installed nothing" class of failure.
  • Materially better output: pptx tables/charts/notes, docx headings/tables/equations, xlsx as GFM tables — verified against the markitdown fixtures; the old readers dropped all of those.
  • Markdown is the better shape for the chat context, FTS snippets and the viewer.

Verified

  • pnpm build (server + web) clean; workflow.yaml still parses.
  • CLI: full pass converts 4/4; rerun reuses 4/4; touch + --subdir reconverts 1; enrich.py over a fake GUFI tree indexes the Markdown (tables/headings present in words), PDF still via pdftotext.
  • runPreExtract fork works under tsx (dev) and from dist/ (prod, incl. pnpm deploy --legacy layout with the wasm).
  • Unreadable input (random.bin as .docx) is logged, counted as failed, exit 0; with no cache entry and no python-docx installed, enrich.py's stdlib reader still indexes the document.

Notes

Word, PowerPoint, Excel and legacy .doc files now convert through
@giraffesyo/downmark (pure-Go converter shipped as wasm in the server's
npm deps) into the extract cache before enrich.py runs. enrich.py reuses
those entries unchanged and keeps python-docx/pptx/openpyxl and the stdlib
OOXML pass as the fallback for whatever downmark cannot read. PDFs stay
with pdftotext and OCR.

The conversion runs as a forked child (server/src/preextract.ts), never on
the event loop, and as a CLI from reindex.sh. Cache entries refresh once per
downmark version via extract/.downmark-version. The viewer renders the
extracted Office text as Markdown, and .doc is now an extracted format.
downmark covers .docx/.pptx/.xlsx/.doc, so python-docx, python-pptx and
openpyxl no longer earn their pip installs: remove them from enrich.py
(the standard-library OOXML pass stays as the last resort), from the
container recipe, from the per-host venv step in the workflow (gone
entirely; reindex.sh and the server already find a Python 3.10+ on their
own), from requirements.txt, and from the extractor report.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant