EFGDL annotation system, edit-distance matching, sample-barcode specifier, native single-end support#17
Draft
elanfisher wants to merge 67 commits into
Draft
EFGDL annotation system, edit-distance matching, sample-barcode specifier, native single-end support#17elanfisher wants to merge 67 commits into
elanfisher wants to merge 67 commits into
Conversation
…from the docs. Currently these only test length correctness and in my next push ill check content and some error handling paths. Lastly I added a benchmark so that we can compare runtime opts via criterion.
…nchor appears in the right place for sci-RNA-seq3 data. Added test which check error handling for anchor mismatches, exact geometry for no tollerence checks, and thresholded via hamming distance checks. Lastly added multi-thread consistency checks for 1v4 threads for 10x and sci-rna-seq3.
…ding of the inner workings before I make my next optimization commit.
… recent sucessful ANTISEQUENCE batch optimization.
…tion - Update SearchWhitelist from tuple to struct for cleaner API - Add followed_by parameter for chained barcode+linker validation - Implement linker validation in interpreter after barcode match - Add search_whitelist_followed_by test with expected output - All existing tests continue to pass
The anchor_relative code was adding a SetOp to redirect seq2.* to seq2._r for subsequent geometry processing. However, read.set() modifies the underlying string buffer and adjusts all intersecting mappings, which corrupted the umi/bc3 mappings that were created at correct positions. The fix removes this SetOp since the label vector update (label.push(_r)) at the end of each iteration already handles redirection for subsequent pieces without modifying the shared string buffer. This fixes the anchor_relative extraction bug where UMI and BC3 were being extracted from wrong positions (after-anchor region instead of before-anchor). Results: - Before: 110K reads matched - After: 361K reads matched (matching splitcode)
…ch, search_whitelist
…ntation New Features: - anchor_relative(): Search for anchor from current position and extract preceding elements relative to anchor position. Handles variable-length regions (indels). - --unassigned1/--unassigned2 CLI flags: Output reads that fail geometry matching to separate files (uses TryOp routing). - filter_within_dist(): Whitelist filtering with hamming distance tolerance. Documentation: - docs/anchor_relative.md: Full documentation for anchor_relative function - docs/unassigned_output.md: Documentation for --unassigned CLI flags - docs/GEOMETRY_EXTENSIONS.md: Updated with all new geometry functions Tests: - Added 3 tests for anchor_relative in compile_tests.rs - All existing tests pass Bug Fixes: - Fixed SetOp corruption in anchor_relative that was modifying shared string buffer - Proper 3-way split for anchor matching (before, anchor, after) Performance validated on SPLiT-seq 2018 500k subset: - 2.4x faster than splitcode (1.5s vs 1.7s) - 3.7x less memory than splitcode (36MB vs 141MB) - 100% barcode validity with whitelist filtering
- Updated chumsky from 0.10.1 to 0.12.0 - Updated lexer to use new map_with API while preserving comment support - Updated parser with macro-based function definitions - Preserved dev features: Search, Anchor, SearchWhitelist variants - Fixed lexer_tests to use chumsky 0.12 API (parse().into_output_errors()) - All tests passing
- Made --file2 optional in CLI args - Updated execution logic to dynamically construct input graphs for 1 or 2 files - Relaxed geometry parser to accept single-read definitions - Verified performance with new single-end vs paired-end benchmark (~2x speedup for SE) - Added regression tests for single-end processing
Add inline #[cfg(test)] modules to 10 source files covering the full EFGDL pipeline: lexing, parsing, compilation, interpretation, and processor utilities. Coverage increased from 2.6% to 80.20% line coverage. Files with tests added: - geometry/mod.rs: Nucleotide, S type tests - geometry/lexer.rs: all token types, Display impls - geometry/parser.rs: IntervalShape, IntervalKind, Expr, Function Display - geometry/compile/utils.rs: ReturnType, validate_composition, IntervalShape methods - geometry/compile/functions.rs: compile_fn for all 18 function types - geometry/compile/definitions.rs: compile_definitions - geometry/compile/transformation.rs: compile_transformation, label_transformation - geometry/interpret.rs: interpret, get_match_type, anchor_relative paths - processors/mod.rs: to_expr, map, match_node, cut/set/retain/trim nodes - execute.rs: compile_geom end-to-end tests, stats serialization Also fixes pre-existing clippy warnings across the codebase: - clone_on_copy in parser.rs - too_many_arguments allow attributes in execute.rs - unnecessary_to_owned in demux.rs and bench_regression.rs - items_after_test_module ordering - PathBuf to Path in bench_regression.rs - explicit_counter_loop to enumerate in bench_regression.rs - same_item_push allow in benches - unused imports in bin.rs and tests Updated pre-commit hook to include --lib tests in coverage measurement.
Add tests/paper_chemistry_tests.rs with 14 tests covering each chemistry
from the paper's Table 2 benchmarks:
10x Chromium v2 (3 tests):
- 100% recovery, R1=26bp (16bp CB + 10bp UMI), R2 passthrough
- R2 cDNA bit-identical to input
- Idiomatic EFGDL with definitions
sci-RNA-seq3 (4 tests):
- Anchor-based parsing with variable-length barcode b[9-10]
- CAGAGC anchor at expected offset in output
- Hamming tolerance 1 accepts 1-nt mismatched anchor
- R2 cDNA passthrough
SPLiT-seq PE (4 tests):
- Exact paper geometry compiles (dual anchor_relative + hamming + map
+ transformation with linker sequences from paper)
- Simplified pipeline without map()
- Full pipeline with barcode mapping files: post-transform
R2=34bp (UMI+3xBC), R1=80bp cDNA
- R1 cDNA verification after transformation
LR-SPLiT-seq (3 tests):
- Single-end long-read geometry compiles
- Full pipeline with synthetic long reads (200-300bp)
- Transformation to extract cDNA + barcodes + UMI
All linker sequences and hamming tolerances match the paper exactly.
Sprint 1 complete: 26 tests in tests/paper_chemistry_tests.rs Edit distance variants (12 new tests): - 10x Chromium v2: control test (no anchor, unchanged) - sci-RNA-seq3: basic, substitution tolerance, insertion tolerance (key edit-over-hamming advantage: 1-nt insertion in anchor) - SPLiT-seq PE: compile, simplified, full map+transformation - LR-SPLiT-seq: compile, pipeline, transformation Head-to-head comparison tests (2 new): - sci-RNA-seq3: hamming vs edit on exact synthetic data - SPLiT-seq PE: hamming vs edit on exact synthetic data Also adds dev_notes/2026-02-18_edit-distance-evaluation-plan.md documenting 8 flaws found in the current paper evaluation: 1. Apples-to-oranges: seqproc(hamming) vs matchbox(edit distance) 2. Config-vs-paper discrepancies (UMI 8bp vs 10bp, L2 16bp vs 30bp) 3. splitcode missing from benchmark script 4. Dataset size mismatch (10M in paper vs 1M in script) 5. Incomplete concordance analysis 6. No statistical rigor (no std dev) 7. Undocumented linker variants (GCT vs GAT) 8. Outdated conclusions section
ANTISEQUENCE moved Expr methods (add, sub, mul, div, not) to std::ops trait implementations. Import the required traits in seqproc: - src/geometry/interpret.rs: import Not - src/processors/mod.rs: import Not, Sub
…xes + 8-test matrix
Annotation System (Sprint 2):
- Lexer: 6 new tokens (HashBracket, Match, FatArrow, Dot, Fw, Rc)
- Parser: Annotation struct, TransformOutput enum (Direct/Match), match block
- Compiler: ReadAnnotations, CompiledMatchBlock, annotation extraction
- Interpreter: match_ori(either) detection, TryOrientationOp wrapping
Bug Fixes (test-first):
- BUG 1: Reject divergent match block arms at compile time (rc_transformation
was compiled but silently ignored by interpreter)
- BUG 3: Compile-time validation that read indices fit in u8 + checked
u8::try_from() in interpreter (was unsafe 'as u8' cast)
- compile_geom() now returns Err instead of process::exit(1)
8-Test Matrix (LR-SPLiT-seq):
- {annotation, no-annotation} x {synthetic, real-like} x {hamming, edit}
- Noisy data generator: 40% perfect, 30% substitution, 30% deletion errors
- Consistency test verifies ordering: ann+edit >= ann+hamming >= no-ann+edit
>= no-ann+hamming on both data types
- Results (n=500):
no-ann+hamming: 50.0%/40.0%, no-ann+edit: 50.0%/50.0%
ann+hamming: 100.0%/70.0%, ann+edit: 100.0%/100.0%
Also: head-to-head recovery, transformation correctness, runtime regression
tests. 48 E2E + 23 parser + 221 lib tests pass, 0 failures.
New tests named to match paper artifacts:
test_table2_10x_v2 -- verifies 100% recovery on 10x Chromium v2
using the exact 10x_v2.geom geometry (Table 2 claim)
test_table2_sciseq3 -- verifies sci-RNA-seq3 recovery using the
exact sciseq3_edit.geom structure with edit(f[CAGAGC], 1) anchor
and transformation output (Table 2 claim)
test_table2_splitseq_pe_edit -- verifies SPLiT-seq PE with the exact
paper geometry: 30bp L1, 30bp L2, 6bp BC1, anchor_relative + edit
distance (Table 2 claim). Includes a new FASTQ generator matching
the actual 94bp R2 structure from the paper.
test_table2_lr_splitseq_ann_edit -- verifies LR-SPLiT-seq with
match_ori(either) + edit distance on noisy mixed-orientation data.
Confirms ann+edit beats no-ann+hamming baseline and recovers from
both orientations (Table 2 claim).
test_concordance_10x_perfect -- verifies pipeline determinism by
running the 10x geometry twice and confirming identical read ID
sets and sequences (Figure 4 Jaccard=1.0 claim).
test_edit_vs_hamming_recovery -- verifies edit distance is a strict
superset of hamming on SPLiT-seq PE data with 1bp deletions and
insertions in linker sequences. Confirms the superset property
using read ID set comparison (Figure 5 claim).
Total: 54 tests in paper_chemistry_tests.rs (48 existing + 6 new).
Full cargo test suite passes with zero regressions.
New CLI flag: --preserve-order
When set, output reads are guaranteed to appear in the same order as
the input FASTQ files. This is useful when downstream tools expect
paired files to be in lock-step without re-sorting.
Implementation: forces single-threaded execution (threads=1) when the
flag is set. With a single thread, the ANTISEQUENCE graph processes
batches sequentially through InputFastqOp's atomic counter, so output
order matches input order by construction. If --threads N>1 is also
specified, the flag overrides it with an info-level log message.
New tests (4):
preserve_order_10x_single_thread -- 500 reads, 100% recovery, exact
input/output ID order match
preserve_order_sci_rna_seq3_single_thread -- 300 reads, 100% recovery,
exact order match
preserve_order_splitseq_pe_partial_recovery -- 200 reads, partial
recovery, output is a subsequence of input order
preserve_order_r1_and_r2_lockstep -- 300 reads, R1 and R2 output IDs
are identical and in the same order
Paper: added --preserve-order mention in methods.tex Performance
Considerations section.
Total: 58 tests in paper_chemistry_tests.rs (54 prior + 4 new).
Full cargo test suite passes with zero regressions.
Milestone 2 Phase B: Hierarchical attribute scoping and annotation-based matching modifier migration. LANG-SCOPE: - Parser: Definition struct now has annotations field; annotation parser accepts keyword tokens (edit, hamming, match, etc.) as names and args - Compiler: ReadAnnotations renamed to ElementAnnotations with ElementId enum (Read/Definition); resolve_annotations() merges parent+child with child-overrides-parent and last-wins dedup - Interpreter: element_annotations threaded through interpret_geometry LANG-MIGRATE-HAMMING/EDIT/SEARCH: - annotations_to_compiled_functions() converts #[hamming(N)], #[edit(N)], #[search(relative)] annotations to CompiledFunction entries injected into definition stacks - New syntax: #[search(relative)] #[edit(6)] l1 = f[SEQ] compiles identically to l1 = anchor_relative(edit(f[SEQ], 6)) - Old function-call syntax still works (both paths produce identical compiled output, verified by byte-identical E2E tests) LANG-UPDATE-CONFIGS: - edit_match.geom migrated to new annotation syntax Validation: - Malformed annotations (#[hamming(abc)], #[edit()], #[search(absolute)]) now produce compile errors instead of being silently ignored - Conflicting old+new syntax on same definition (#[hamming(3)] on edit(f[SEQ], 1)) detected and rejected at compile time - 399 tests pass (244 unit + 61 paper_chemistry + 28 parser + others)
Allow match block arms to produce different transformations based on runtime attribute values (e.g., ori=fw vs ori=rc). Previously, different arms were rejected; now they are conditionally applied via SelectOp. Compiler: - Removed identical-arms restriction from match block compilation - Extended CompiledMatchBlock with fw_map, rc_map, base_map to carry per-arm geometry maps with arm-specific function stacks - Added compile-time validation: match block on 'ori' requires #[match_ori(either)] on the referenced read Interpreter: - build_arm_graph closure diffs arm map vs base map to find labels with extra functions (e.g., ReverseComp for rc arm) - Applies arm-specific functions via set_node in SelectOp subgraph before label rearrangement - Two SelectOps added: one for ori==fw, one for ori==rc Tests: - lang_cond_output_different_arms_compile - lang_cond_output_fw_reads_get_fw_barcodes - lang_cond_output_rc_reads_get_rc_barcodes - lang_cond_output_trunc_in_match_arm_does_not_panic - lang_cond_output_match_block_without_match_ori_rejected - Updated bug1_divergent_match_block_arms -> _accepted 404 tests pass, zero failures.
Emit deprecation warnings when old function-call syntax (hamming(), edit(), anchor_relative()) is used in definitions. The old syntax continues to work identically; warnings suggest the new annotation syntax. Changes: - compile_definitions() now returns (HashMap, Vec<String>) with warnings - CompiledData has new 'warnings' field threaded through all compile paths - compile_geom() prints warnings to stderr after successful compilation - Warnings emitted for Hamming(N), Edit(N), Anchor in definition stacks New tests: - test_deprecation_warning_for_old_hamming_syntax - test_deprecation_warning_for_old_edit_syntax - test_deprecation_warning_for_old_anchor_relative_syntax - test_no_deprecation_warning_for_new_annotation_syntax - test_no_deprecation_warning_for_new_stacked_syntax Updated compile_tests.rs for new compile_definitions return type. 410 tests pass, zero failures.
New test file: tests/annotation_tests.rs with 24 tests covering: 1. Annotation parsing (4 tests): - Definition-level, read-level, stacked, both combined 2. Matching modifier migrations (4 tests): - hamming, edit, search(relative), stacked search+edit - Each verifies old and new syntax produce byte-identical output 3. Conditional output branching (2 tests): - Different fw/rc arms with revcomp - Identical arms produce same output as direct transform 4. Backward compatibility (3 tests): - Old hamming(), edit(), anchor_relative() still compile and run 5. Deprecation warnings (2 tests): - Old syntax emits warnings, new syntax does not 6. Error cases (7 tests): - Malformed annotations, empty args, bad search mode - Conflicting old+new syntax, match without match_ori 7. No-regression (2 tests): - Plain definitions without annotations - Geometry with no definitions at all 434 tests pass, zero failures.
Bulk migration of all paper_chemistry_tests.rs and compile_tests.rs geometry strings from old function-call syntax to new annotation syntax: - anchor_relative(hamming(f[SEQ], N)) -> #[search(relative)] #[hamming(N)] def = f[SEQ] - anchor_relative(edit(f[SEQ], N)) -> #[search(relative)] #[edit(N)] def = f[SEQ] - anchor_relative(f[SEQ]) -> #[search(relative)] def = f[SEQ] - hamming(f[SEQ], N) -> #[hamming(N)] def = f[SEQ] - edit(f[SEQ], N) -> #[edit(N)] def = f[SEQ] Kept old syntax ONLY in: - annotation_tests.rs backward-compat tests (explicitly test old syntax) - definitions.rs equivalence tests (compare old vs new compilation) - compile_tests.rs error case (edit(b[8], 2) rejection) - compile_tests.rs no_err test (inline hamming in read expression) Fixed paper_lr_splitseq_match_ori_compile assertion: element_annotations now has 3 entries (read + 2 definitions) instead of 1. 434 tests pass, zero failures. Zero deprecation warnings from new syntax.
Introduces a new EFGDL interval kind `SampleBarcode` with the key letter `s`, sibling to `b` / `u` / `x` / `r` / `f`. Motivated by multiplexed single-cell protocols such as 10x Genomics Chromium Flex, where up to 384 samples are pooled into one GEM well and the sample of origin is encoded by a probe barcode in R2 that is distinct from the GEM cell barcode in R1. Mechanically, `s` behaves identically to `b` and supports the same whitelist / Hamming / edit / padding / truncation transformations. Semantically, the `s` label makes cell identity and sample identity distinguishable in the output so downstream demultiplexers can route each barcode to its correct use without additional metadata bookkeeping. Changes - lexer.rs: `Token::SampleBarcode`, display as `s`, identifier match for the bare letter `s`, new unit tests. - parser.rs: `IntervalKind::SampleBarcode`, Display, wired through the `piece_type` select! macro, new TDD-style parser tests (bare, labeled, fixed-length, bounded, unbounded, combined with `b`/`u`). - compile/utils.rs: emit `s` in simplified descriptions; validation treats SampleBarcode as a length-bearing non-void kind (like Barcode and Umi), confirmed by direct validate_expr tests including a Reverse composition (which rejects Void). - Cargo.toml / Cargo.lock: add proptest 1.4 as dev-dependency for the new property-based round-trip tests. Tests: lexer + parser + compile unit tests plus proptest round-trip (lex -> parse -> Display) for geometries containing `s`. Full suite green (451 tests) and confirmed on real 10x Chromium v2 data: the same geometry run with `b[16]` vs `s[16]` in the cell slot produces byte-identical R1/R2 MD5s under --preserve-order.
elanfisher
force-pushed
the
edit_distance_map
branch
from
4ee1314 to
3152df6
Compare
demux: load_sample_map silently dropped any non-comment line lacking a TAB (e.g. space-separated), routing that barcode's reads to unassigned with no error. Return InvalidData on a malformed line instead. bin: a missing geometry file (read_to_string(...).unwrap()) and a missing input FASTQ (opened deep in the engine) both aborted with a panic + backtrace. Handle the geometry read explicitly and validate input FASTQ paths up front, exiting cleanly with a message. Tests: demux::malformed_sample_map_line_is_not_silently_dropped and tests/error_handling_tests.rs (missing geom / FASTQ do not panic).
Resolve conflicts by keeping the branch's versions, which supersede main's older 'argument parsing' commit: - Cargo.toml: antisequence -> COMBINE-lab/ANTISEQUENCE main - Cargo.lock: regenerated for that dependency - src/bin/bin.rs: branch version (already has geom: PathBuf and the input-validation fixes)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The EFGDL language + processor work behind the seqproc paper (64 commits ahead of
main).Language (EFGDL)
#[edit(n)],#[hamming(n)],#[search(relative)],#[match_ori(either)]..geomsyntax.Matching
New specifiers / modes
ssample-barcode interval specifier (10x Genomics Flex).Tests
Dependency
Cargo.tomlnow pointsantisequenceatCOMBINE-lab/ANTISEQUENCEmain(was a localpath = "../ANTISEQUENCE"dev override;mainpreviously pointed at a personal fork). Depends on the ANTISEQUENCE PR — merge that one first, then this builds against ANTISEQUENCEmain.