╭────────────────────────────────────────────────────────────╮
│      ███████╗██████╗ ██╗  ██╗ █████╗ ██╗     ██╗           │
│      ██╔════╝██╔══██╗██║  ██║██╔══██╗██║     ██║           │
│      █████╗  ██║  ██║███████║███████║██║     ██║           │
│      ██╔══╝  ██║  ██║╚════██║██╔══██║██║     ██║           │
│      ███████╗██████╔╝     ██║██║  ██║███████╗███████╗      │
│      ╚══════╝╚═════╝      ╚═╝╚═╝  ╚═╝╚══════╝╚══════╝      │
╰────────────────────────────────────────────────────────────╯

Ed4All

Turn learning materials into accessible courses—and course-grounded AI

Ed4All transforms books, PDFs, HTML, and documentation into structured, accessible HTML, modular digital course content, and LMS-ready IMS Common Cartridge packages.

From the same source-grounded content, Ed4All can build a searchable course library, generate supervised fine-tuning (SFT) and preference (DPO) pairs, and optionally train a course-specific LoRA adapter.

Its custom retrieval layer keeps lexical and semantic scores in their proper domains, fuses ranked evidence with Reciprocal Rank Fusion (RRF), refuses weak queries before generation, and verifies citations before returning an answer.

One source. Four useful outcomes.

Accessible HTML · Digital course + IMSCC · Grounded training data · Hybrid retrieval

Get started · See the pipeline · Explore the components · Read the documentation

---

What Ed4All does

• Converts source material into semantic, accessibility-oriented HTML with source provenance and automated validation.
• Builds digital courses with modules, learning objectives, activities, assessments, and machine-readable educational metadata.
• Packages courses for an LMS using IMS Common Cartridge (IMSCC).
• Indexes course content in a reusable local library with lexical, semantic, and hybrid-RRF retrieval.
• Generates grounded training data as SFT instruction pairs and DPO preference pairs derived from course content.
• Optionally trains a LoRA adapter and evaluates it alongside the course's retrieval system.

Token Tracking
13,294,430,457 🧠 DEVELOPMENT TOKENS	53 🧭 SESSIONS	6,444 💬 USER TURNS OBSERVED	1,201,191 🧱 TRACKED TEXT LOC
🤝 COLLABORATOR	TOKENS	SESSIONS	USER TURNS
Claude	12,016,436,779	38	5,961
Codex	1,277,993,678	15	483
↔️ TOKEN FLOW	READ	WRITTEN	AVG / SESSION
All sessions	13,247,419,164	47,011,293	250,838,310
🔎 TOKEN DETAIL	COUNT	TOKEN DETAIL	COUNT
Fresh input	30,359,926	Cache writes	285,091,038
Cache reads	12,931,968,200	Model output	47,011,293
Reasoning output subset	476,721	Counted again in total	No
⏱️ SESSION DURATION	CLAUDE AVG	CODEX AVG	COMBINED AVG
First-to-last observed event	18h 20m	13h 53m	17h 5m
📚 TRACKED TEXT	LINES	TRACKED TEXT	LINES
Application source	552,257	Tests	504,882
Documentation	33,226	Tooling / configuration	109,577
Other text	1,249	Total physical lines	1,201,191

_{[How these privacy-safe project metrics are counted →](docs/operations/development-token-stats.md)}

From source to course-grounded AI

flowchart LR
    materials["Books, PDFs, HTML,<br/>and learning materials"]
    semantik["SemantiK<br/>Accessible, structured HTML<br/>with source provenance"]
    courseforge["Courseforge<br/>Modular course content<br/>and an LMS-ready IMSCC"]
    trainforge["Trainforge<br/>Retrieval corpus plus<br/>SFT and DPO pairs"]
    libv2["LibV2<br/>Private course archive<br/>and optional LoRA training"]
    retrieval["Hybrid retrieval<br/>BM25 + dense search<br/>with rank-domain RRF"]
    answers["Course-grounded AI<br/>Evidence thresholds,<br/>refusal, and cited answers"]

    materials --> semantik --> courseforge --> trainforge --> libv2 --> retrieval --> answers

    classDef sourceNode fill:#eef6ff,stroke:#2563eb,color:#172554,stroke-width:2px;
    classDef buildNode fill:#f0fdf4,stroke:#16a34a,color:#14532d;
    classDef deliveryNode fill:#fff7ed,stroke:#ea580c,color:#7c2d12;
    classDef intelligenceNode fill:#faf5ff,stroke:#9333ea,color:#581c87;

    class materials sourceNode;
    class semantik,courseforge buildNode;
    class libv2 deliveryNode;
    class trainforge,retrieval,answers intelligenceNode;

Loading

One continuous workflow carries the source through accessible HTML, course design, LMS packaging, grounded-data synthesis, archival, retrieval, and citation validation. Optional LoRA training uses the generated training pairs as a separate operator opt-in; the course package, archive, and retrieval system remain complete deliverables without an adapter.

Retrieval that earns trust

Ed4All does more than make course content searchable. It builds a private, course-scoped evidence system designed to show its work:

• Lexical precision + semantic reach. BM25 catches exact terminology while dense retrieval finds conceptually related passages.
• Custom rank-domain RRF. Ed4All fuses rank positions instead of adding incompatible BM25 and cosine scores, rewarding evidence found by both arms without discarding strong single-arm results.
• Refusal before generation. Weak retrieval stops before the answer model can turn uncertainty into fluent guesswork.
• Citation-grounded answers. Returned citations must resolve to passages visible to the composer; unsupported answers are withheld.
• Useful with or without training. The retrieval system is a complete deliverable on its own and can also evaluate whether a LoRA adapter adds value beyond the indexed course.

The result is a local course intelligence layer that stays attached to the source: private indexes, reproducible ranking, inspectable evidence, and answers that can be traced back to the material. See the retrieval architecture.

Quick start

Ed4All requires Python 3.10 or newer. Tesseract OCR and Poppler improve extraction from scanned or image-heavy PDFs.

See the installation guide for platform dependencies, capability extras, Playwright, and the required third-party IMS Common Cartridge schemas.

git clone https://github.com/mdmurphy822/Ed4All.git
cd Ed4All
pip install -e ".[full]"

ed4all run textbook-to-course \
  --corpus <path-to-source> \
  --course-name <course-name>

Authoring and synthesis phases require a configured model provider. See the pipeline invocation guide for local and hosted OpenAI-compatible endpoint setup.

The default install leaves out the largest machine-learning dependencies. Add only the capabilities you need:

Extra	Adds	Use it for
embedding	Sentence Transformers and PyTorch	Dense retrieval, hybrid RRF, and embedding-backed validators
training	Transformers, TRL, PEFT, and training dependencies	Optional SFT/DPO LoRA training on a supported GPU

pip install -e '.[full,embedding]'
pip install -e '.[full,training]'

Before a production build, review the full-run playbook and licensing posture.

Choose your outcome

Convert documents to accessible HTML

Create remediated HTML without generating a course:

ed4all convert <source> --output <output-directory>

See the conversion guide.

Build and package a digital course

ed4all run textbook-to-course \
  --corpus <path-to-source> \
  --course-name <course-name>

This orchestrates conversion, course planning and generation, validation, IMSCC packaging, archival, and indexing according to the selected workflow and configuration.

Query existing course content

Query an archived and indexed course through the retrieval layer:

libv2 retrieve "<question>" --course <course-name> --engine hybrid-rrf

Hybrid RRF combines BM25 term matching with dense vector similarity. Results retain course and chunk provenance so downstream answer systems can cite the retrieved material. See retrieval and serving.

Generate training data or train an adapter

Training-pair synthesis produces SFT instructions and DPO preferences from the course's chunks and assessments. Adapter training is a separate, opt-in, GPU-bound stage:

# Build a course and explicitly include its training stages.
ed4all run textbook-to-course \
  --corpus <path-to-source> \
  --course-name <course-name> \
  --with-training

# Or train from an already archived course.
ed4all run trainforge_train \
  --course-name <course-name> \
  --base-model <supported-model>

Model licenses and provider terms determine whether generated pairs and trained derivatives are distributable. Read Licensing and ToS posture first. Full training runs and promotion decisions remain operator-driven.

Generated SFT and DPO pairs, retrieval indexes, and course artifacts always remain private local data; they are not published from this repository.

Why Ed4All

• One grounded content lineage. Every stage works from the same source material.
• Accessibility is part of the pipeline. Semantic structure and automated checks are designed to support WCAG 2.2 AA targets; final conformance still depends on the source, configuration, generated content, and human review.
• Standards-based packaging. Courseforge emits IMS Common Cartridge.
• Retrieval is a first-class deliverable. A course can be searched and queried without training an adapter.
• Training is explicit. LoRA stages do not attach to a default build unless the operator opts in.
• Providers are configurable. Authoring can use configured local or hosted OpenAI-compatible endpoints.

Components

• SemantiK converts documents into accessibility-oriented HTML with source provenance.
• Courseforge creates modular course content, learning activities, and IMS Common Cartridge packages.
• Trainforge creates tagged chunks, assessments, knowledge structures, SFT/DPO pairs, and optional LoRA training inputs.
• LibV2 archives and queries course content through lexical, semantic, and hybrid reciprocal-rank-fusion retrieval.

MCP orchestrates workflows; cli, gui, and lib provide the user and shared service surfaces.

Ways to run it

• Command line: ed4all run --help and the invocation guide.
• Browser: ed4all gui and the GUI guide.
• Containers: the provided Compose deployment and Docker guide.
• Long-running workflows: checkpoints, resume, and graceful stop are covered by the operations runbook.

Documentation

• Installation and local dependencies
• Architecture overview
• Pipeline flow
• Validation gates
• Licensing and ToS posture
• Component guides: SemantiK, Courseforge, Trainforge, and LibV2

License

Ed4All is available under the Apache License 2.0. See LICENSE.