Add anonymizer tool by sfraczek · Pull Request #166 · RustInFinance/etradeTaxReturnHelper

sfraczek · 2025-11-24T22:23:38Z

Introduces etradeAnonymizer — a Rust binary and anonymizer module for anonymizing E*TRADE / Morgan Stanley PDF statements while preserving data needed for tax calculations.

Subcommands

list <input.pdf> — lists all text tokens from FlateDecode/uncompressed streams (debugging/inspection).
anonymize <input.pdf> [output.pdf] — replaces PII with stable numeric tokens; output defaults to anonymous_<input>.pdf.

How it works

Operates directly on PDF streams (FlateDecode and uncompressed with an explicit /Length, PDF 1.3).
Preserves calculation-relevant text: strings containing CLIENT STATEMENT / For the Period, and everything between CASH FLOW ACTIVITY BY DATE and NET CREDITS/(DEBITS) (inclusive).
In-place replacement that keeps the exact original stream size (recompression + null-byte padding), so the PDF XREF table stays valid without full re-parsing.

Module layout

list.rs, replace.rs (engine: replace_pii_smart), path.rs, pdf.rs. No separate detect module/mode.

Testing

Unit tests run in CI (PDF token scanning, escape/unescape, output path handling).
5 integration tests are marked #[ignore] — they require local plain PDF fixtures in anonymizer_data/ (git-ignored) and do not run in CI.

Notes

No openssl dependency, no bundled encrypted fixtures, no ANONYMIZER_TEST_KEY in CI.
Not a guarantee of complete PII removal — output should be reviewed manually before sharing.

jczaja · 2025-12-08T07:56:49Z

+
+/// Parse arguments and dispatch to detect / replace logic. Returns Ok even
+/// for usage errors (prints help) to keep CLI simple.
+pub fn run(args: Vec<String>) -> Result<(), Box<dyn Error>> {


For consistency with rest of project and ease of maintainance I would suggest using clap (ideally wich derive feature) to handle commandline. I have a branch where clap is bumped up to most recent version. So this can be used as an example

Copilot

Pull request overview

This PR introduces a new Rust-based “etradeAnonymizer” tool within the existing crate, aimed at detecting and replacing PII in (a constrained subset of) PDF statement streams while preserving original stream byte lengths to avoid rebuilding XREF offsets.

Changes:

Adds a new etradeAnonymizer binary with list and anonymize subcommands.
Implements PDF stream scanning/text token extraction plus “smart” token-preserving replacement with size-fitting recompression and padding.
Updates project metadata and dependencies to support the new functionality (e.g., flate2, sha2) and ignores local PDF fixtures.

Reviewed changes

Copilot reviewed 10 out of 12 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/lib.rs	Exports the new `anonymizer` module from the crate root for library/tests/binary access.
src/anonymizer/mod.rs	Defines the anonymizer module structure and re-exports submodules.
src/anonymizer/anonymizer.rs	Adds the new CLI binary entry point and subcommand routing.
src/anonymizer/pdf.rs	Implements strict PDF header validation, stream scanning, and token extraction/unescaping utilities.
src/anonymizer/replace.rs	Implements smart anonymization by token replacement with recompression-to-fit and padding.
src/anonymizer/list.rs	Adds a “list tokens” mode for inspecting extracted text tokens per stream.
src/anonymizer/path.rs	Adds helper for generating default anonymized output filenames.
src/anonymizer/README.md	Documents usage, strategy, limitations, and local-only testing workflow.
README.md	Updates SPDX copyright year range.
Cargo.toml	Registers the new binary and adds required dependencies.
Cargo.lock	Updates the lockfile to include new/updated dependency versions.
.gitignore	Ignores local PDF fixtures for offline integration testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            Ok((new_data, modified)) => {
+                if modified {
+                    streams_modified += 1;
+                }
+
+                // Check size and warn if too large
+                if new_data.len() > stream_data.len() {
+                    warn!(
+                        "Cannot fit modified data: {} > {} bytes",
+                        new_data.len(),
+                        stream_data.len()
+                    );
+                    warn!("Skipping modifications for this stream");
+                    continue;
+                }


+    // Extract texts along with their byte positions in the decompressed stream
+    let texts = extract_texts_from_stream(stream_data, is_compressed)?;


+                // Pad remaining space with NULL bytes to maintain exact original stream length.
+                // This preserves object offsets and avoids recalculating the PDF XREF table.
+                // Standard PDF viewers ignore bytes following the end of valid Zlib streams.


+//! This module provides functionality to extract and list all text tokens from
+//! FlateDecode streams in a PDF. Each token is printed with a global index,
+//! useful for understanding the structure and content of the PDF before detection/replacement.


+/// Regex matching an object with uncompressed stream and explicit /Length (no /Filter).
+pub(crate) const OBJ_UNCOMPRESSED_STREAM_RE: &str =
+    r"(?s)\d+\s+\d+\s+obj\s*<<[^>]*?/Length\s+(\d+)[^>]*?>>\s*stream\r?\n";


+/// Process a single stream with smart anonymization.
+/// Returns (output_data, was_modified).
+/// For compressed streams, returns compressed data. For uncompressed, returns raw data.
+fn process_stream_smart(
+    stream_data: &[u8],
+    is_compressed: bool,
+    letter_counter: &mut usize,
+    in_cash_flow_section: &mut bool,
+) -> Result<(Vec<u8>, bool), Box<dyn std::error::Error>> {


Copilot

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated 3 comments.

sfraczek · 2026-07-10T13:01:55Z

+            let is_compressed = if bytes_contain(dict, b"/FlateDecode") {
+                true
+            } else if bytes_contain(dict, b"/Filter") {
+                // Unsupported filter (e.g. /DCTDecode image): don't decode/scan.
+                continue;
+            } else if bytes_contain(dict, b"/Length1") {
+                // Embedded font program (FontFile/FontFile2): binary font data
+                // with no PDF text tokens; scanning it risks corrupting glyphs.
+                continue;
+            } else {
+                false
+            };


Good catch — fixed in ad139f3. The classifier now checks /Length1 before /FlateDecode, so an embedded font program (FontFile/FontFile2) is skipped regardless of compression: even a Flate-compressed font is never decoded or scanned. Added test_scanner_skips_compressed_font_program_stream to lock this in.

sfraczek · 2026-07-10T13:01:59Z

+    if stream_end + EXPECTED_END.len() > pdf_data.len() {
+        error!(
+            "End marker out of bounds after stream_end {} for object starting at {} (pdf_len={})",
+            stream_end,
+            data_start,
+            pdf_data.len()
+        );
+        return None;
+    }
+
+    // Validate exact end marker (requirements are strict)
+    let debug_slice = &pdf_data[stream_end..stream_end + EXPECTED_END.len()];
+    if debug_slice != EXPECTED_END {
+        warn!(
+            "End marker mismatch for object starting at {}: found {:?}, expected {:?}",
+            data_start, debug_slice, EXPECTED_END
+        );
+        // Return decompressed candidate but indicate end marker mismatch for caller decision
+        return Some((&pdf_data[data_start..stream_end], data_start, false));
+    }
+
+    Some((&pdf_data[data_start..stream_end], data_start, true))


Acknowledged as a deliberate limitation for now. The target E*TRADE %PDF-1.3 statements use the exact LF marker (\nendstream\nendobj), which the ignored integration tests validate. A CRLF stream would be flagged invalid and skipped, but this is no longer silent: the recent fail-loud change counts skipped streams and warns in the run summary, so such a case is visible rather than a hidden PII leak. We kept the end-marker strict to keep the parser narrow, and can revisit if a CRLF statement variant shows up.

sfraczek · 2026-07-10T13:01:57Z

+    // Fail loud: a cash-flow section that opened but never closed means every
+    // token after it was preserved and may still contain PII.


Fixed in ad139f3. The warning-only behavior is intentional (we still produce output), so this was a comment/behavior mismatch rather than a code bug. Updated the comment to "Warn loudly (output is still produced): ..." to match the actual behavior.

Register the etradeAnonymizer binary, add its dependencies (flate2, clap derive), refresh the lockfile (incl. ethnum 1.5.3 for newer rustc), ignore local PDF fixtures, and bump the SPDX copyright year.

Strict %PDF-1.3 reader, a single-regex stream scanner that classifies each stream by its dictionary (font programs via /Length1 and unsupported filters skipped, FlateDecode decoded, plain content scanned), byte-exact stream slicing with end-marker validation, and a linear text-token scanner that only commits tokens inside closed BT/ET text objects. Wires the module into the crate root.

replace_pii_smart decompresses each content stream, replaces PII tokens with positional indices while preserving marker phrases and the CASH FLOW section, recompresses to fit, and NUL-pads to keep the original stream byte length so the XREF table stays valid. Warns loudly on skipped streams and on a CASH FLOW section that never closes.

The list subcommand prints extracted tokens per stream; anonymous_output_path builds the default 'anonymous_<input>.pdf' name; the clap-based binary wires the list and anonymize subcommands.

$@sfraczek$

sfraczek requested a review from jczaja November 24, 2025 22:24

$@sfraczek$ sfraczek added the enhancement New feature or request label Nov 24, 2025

$@sfraczek$

sfraczek marked this pull request as draft November 24, 2025 22:25

jczaja reviewed Dec 8, 2025

View reviewed changes

$@sfraczek$

sfraczek force-pushed the sfraczek/anonymizer branch 2 times, most recently from c7fdef8 to 816af4d Compare December 13, 2025 18:12

$@sfraczek$

sfraczek force-pushed the sfraczek/anonymizer branch 2 times, most recently from ff026ea to 296708b Compare July 7, 2026 17:50

$@sfraczek$

sfraczek marked this pull request as ready for review July 7, 2026 17:50

$@sfraczek$

sfraczek requested a review from Copilot July 7, 2026 17:50

Copilot started reviewing on behalf of sfraczek July 7, 2026 17:50 View session

Copilot AI reviewed Jul 7, 2026

View reviewed changes

$@sfraczek$

sfraczek force-pushed the sfraczek/anonymizer branch 2 times, most recently from d45fb82 to c666272 Compare July 7, 2026 18:15

$@sfraczek$

sfraczek requested a review from Copilot July 9, 2026 22:16

Copilot started reviewing on behalf of sfraczek July 9, 2026 22:16 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

sfraczek added 5 commits July 10, 2026 13:13

$@sfraczek$

Add build setup for the anonymizer tool

2ff2d45

Register the etradeAnonymizer binary, add its dependencies (flate2, clap derive), refresh the lockfile (incl. ethnum 1.5.3 for newer rustc), ignore local PDF fixtures, and bump the SPDX copyright year.

$@sfraczek$

Add token listing, output-path helper, and CLI

9543895

The list subcommand prints extracted tokens per stream; anonymous_output_path builds the default 'anonymous_<input>.pdf' name; the clap-based binary wires the list and anonymize subcommands.

$@sfraczek$

Add anonymizer module documentation

5289612

$@sfraczek$

sfraczek force-pushed the sfraczek/anonymizer branch from 4b9e581 to 5289612 Compare July 10, 2026 13:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add anonymizer tool - #166

Add anonymizer tool#166
sfraczek wants to merge 5 commits into
mainfrom
sfraczek/anonymizer

$@sfraczek$ sfraczek commented Nov 24, 2025 •

edited

Loading

Uh oh!

jczaja Dec 8, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

$@sfraczek$ sfraczek Jul 10, 2026

Uh oh!

$@sfraczek$ sfraczek Jul 10, 2026

Uh oh!

$@sfraczek$ sfraczek Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// Extract texts along with their byte positions in the decompressed stream
		let texts = extract_texts_from_stream(stream_data, is_compressed)?;

		// Fail loud: a cash-flow section that opened but never closed means every
		// token after it was preserved and may still contain PII.

Uh oh!

Uh oh!

Conversation

sfraczek commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Subcommands

How it works

Module layout

Testing

Notes

Uh oh!

jczaja Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

sfraczek Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

sfraczek Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

sfraczek Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

$@sfraczek$ sfraczek commented Nov 24, 2025 •

edited

Loading

$@sfraczek$ sfraczek Jul 10, 2026

$@sfraczek$ sfraczek Jul 10, 2026

$@sfraczek$ sfraczek Jul 10, 2026