dlp-patterns

Fast, zero-dependency DLP pattern scanner for Python.

Detects PII, secrets, and sensitive data in any text — documents, logs, source code, emails. Built from the scanning engine that powers Spidercob, an enterprise DLP platform.

import dlp_patterns

result = dlp_patterns.scan("My SSN is 432-78-9012 and CC 4111 1111 1111 1111")
print(result.highest_severity)   # CRITICAL
print(result.critical[0].type)   # credit_card

clean = dlp_patterns.redact("Send to alice@corp.com with CC 4111 1111 1111 1111")
# "Send to [REDACTED: Email Address] with [REDACTED: Credit Card Number]"

Install

pip install dlp-patterns

No external dependencies. Python 3.9+.

What it detects

Category	Patterns
Financial	Credit cards (Luhn + BIN), SSN, IBAN, bank account, routing number
PII	Email, US phone, passport, driver's license, date of birth
Healthcare	Medical record numbers, ICD-10 codes, NPI, DEA numbers, NDC codes
Secrets	AWS keys, GitHub PATs, Slack tokens, Google API keys, Bearer tokens, JWTs
Cloud / SaaS	Stripe, SendGrid, Mailgun, Twilio, HuggingFace, NPM, Cloudflare, Azure
Infrastructure	DB connection strings, hardcoded passwords, Docker registry auth
Crypto	RSA/EC/SSH/PGP private keys, X.509 certs
Webhooks	Slack webhooks, Discord webhooks, Telegram bot tokens
Cryptocurrency	Bitcoin addresses, Ethereum addresses

50+ pattern categories total.

Features

• Validators — Luhn check for credit cards, FICA rules for SSNs, JSON decode for JWTs. Reduces false positives before they reach you.
• Entropy gating — Shannon entropy + sliding-window analysis rejects low-entropy matches (e.g. aaaaaaa...) from generic secret patterns.
• Context scoring — Each finding gets a context_score (0–1) based on surrounding words. Proximity to production, secret, deploy boosts the score; proximity to example, placeholder, test lowers it.
• Required context keywords — Patterns like ICD-10 codes and Telegram tokens only fire when relevant keywords appear nearby.
• secrets_only mode — Scan just for API keys and credentials, skipping PII. Faster for CI/CD secret scanning.
• redact() — Replace all findings with [REDACTED: <type>].
• fuzz() — Replace findings with realistic fake values (requires faker). Useful for building safe test datasets from production data.
• Git history scanning — dlp-scan --history walks every commit, not just the working tree, so a secret that was committed and later deleted still gets caught. See Git history scanning.
• Live secret verification — dlp-scan --verify checks whether a found secret is still active by making a real, read-only call to its own provider's API. See Live secret verification.
• CLI — dlp-scan command for shell pipelines and CI.

Usage

Python API

import dlp_patterns

# Scan
result = dlp_patterns.scan(text)

result.has_findings          # bool
result.highest_severity      # "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | None
result.critical              # list[Finding]
result.all                   # all findings across severities
result.elapsed_ms            # scan time in milliseconds

# Each Finding:
f = result.critical[0]
f.type                       # "credit_card"
f.description                # "Credit Card Number"
f.value                      # masked: "4111...1111"
f.severity                   # "CRITICAL"
f.position                   # "char 10-29"
f.context                    # surrounding text (±100 chars)
f.context_score              # float 0.0–1.0
f.context_keywords_found     # ["payment", "billing"]
f.verification                # None until dlp_patterns.verify() is called

# Secrets only (faster for source code scanning)
result = dlp_patterns.scan(code, secrets_only=True)

# Redact
clean = dlp_patterns.redact(text)

# Fuzz (pip install dlp-patterns[fuzz])
safe = dlp_patterns.fuzz(text)

# JSON output
result.to_dict()

CLI

# Scan a string
dlp-scan "My SSN is 432-78-9012"

# Scan a file
dlp-scan path/to/document.txt

# Scan a directory recursively
dlp-scan path/to/project/

# Pipe from stdin
cat logfile.txt | dlp-scan

# JSON output
dlp-scan --json document.txt

# Redact in place (files only, not directories)
dlp-scan --redact document.txt > clean.txt

# Secrets only (for source code)
dlp-scan --secrets-only src/config.py

# Exit code: 1 if CRITICAL findings, 0 otherwise — useful in CI
dlp-scan --secrets-only . && echo "clean"

# Only fail the exit code for high-confidence findings (context_score >= 0.5) —
# reduces false-positive CI failures from doc/test fixtures without hiding them
# from the output. Findings are always reported regardless of this flag.
dlp-scan --secrets-only --min-confidence 0.5 .

Exit code and --min-confidence

By default, exit code 1 means "at least one CRITICAL finding" — full stop, regardless of how confident the match is. The engine already computes a context_score per finding (0.0-1.0: proximity to words like production/deploy raises it, proximity to example/test/mock/a code fence lowers it — see Features), but by default the exit code ignores it entirely, same as it always has.

--min-confidence <float> changes what the exit code reacts to: a CRITICAL finding only fails the build if its context_score is >= the given value. This matters most for scanning a codebase whose job description includes containing realistic-looking fake secrets — a secret-scanner's own test suite, security training data, a docs site with credential examples — where --secrets-only alone will always find something. Findings are still fully reported either way; the flag only changes what fails the build.

dlp-scan --secrets-only --min-confidence 0.5 src/     # a reasonable CI default
dlp-scan --secrets-only --min-confidence 0.5 --history .   # combine with history scanning

Directory scanning

dlp-scan <directory> walks recursively and scans every text file it finds. It automatically skips:

• Version control and vendor directories: .git, node_modules, __pycache__, .venv, venv, dist, build, .mypy_cache, .pytest_cache, .tox
• Lockfiles (package-lock.json, yarn.lock, pnpm-lock.yaml, Cargo.lock, poetry.lock, Pipfile.lock, go.sum, composer.lock, npm-shrinkwrap.json) — these are auto-generated and full of base64 integrity hashes that false-positive against secret regexes
• Binary files (detected by a null-byte sniff on the first 8KB)

--json output for a directory scan has a different shape than a single-file scan — findings are grouped by file:

{
  "mode": "directory",
  "path": "src/",
  "files_scanned": 42,
  "files_with_findings": 2,
  "highest_severity": "CRITICAL",
  "findings_by_file": {
    "config.py": { "CRITICAL": [...], "HIGH": [], "MEDIUM": [], "LOW": [], "INFO": [], "elapsed_ms": 1.2 }
  },
  "elapsed_ms": 38.4
}

Exit code is still 1 if any file has a CRITICAL finding, 0 otherwise.

Git history scanning

Deleting a leaked key from HEAD does not un-leak it — anyone with a clone of the repo (including one taken before the deletion) can still read it out of git log -p. This is the single most common way real secrets leak, and plain dlp-scan <directory> mode can't see it: it walks the working tree and explicitly skips .git. --history walks every commit on every branch instead:

# Scan full history of the repo at (or containing) the given path
dlp-scan --history path/to/repo

# Defaults to the current directory
dlp-scan --history

# Limit to the N most recent commits
dlp-scan --history --max-commits 500 .

# --history defaults to secrets only (PII noise across full history is
# high — every email/phone number ever committed, including test fixtures).
# Opt into PII scanning too:
dlp-scan --history --full-scan .

dlp-scan --history --json . | tee history-report.json

Only lines added in a commit's diff are scanned, once per commit that introduced them — sufficient and non-redundant, since a line later removed was necessarily added by some earlier commit. Requires the git binary on PATH (an external tool dependency, not a pip package — this library still ships with zero pip-installable dependencies).

Each finding carries the commit it was introduced in:

{
  "type": "aws_access_key",
  "severity": "CRITICAL",
  "value": "AKIAIOSFOD...",
  "commit": "<full 40-char SHA>",
  "short_commit": "b874554",
  "author": "Jane Doe <jane@example.com>",
  "date": "2026-08-06T09:46:52+05:30",
  "subject": "add key",
  "file": "config.py"
}

This only detects — it intentionally does not try to rewrite history or force-push a fix. Purging a secret from history for real (git filter-repo

• a coordinated force-push + rotating the credential) is a separate, higher-stakes operation you should do deliberately, not something a scanner should do for you.

Live secret verification

Regex matching alone can't tell a live production key from one that's already been rotated. --verify closes that gap for a curated set of secret types by making one real, read-only API call per distinct secret — "who am I" / "list my scopes", never an action — to confirm whether it still authenticates:

dlp-scan --verify --secrets-only src/config.py
dlp-scan --verify --history .          # combine with history scanning
dlp-scan --verify --verify-timeout 8 . # per-secret network timeout (default 4s)

[CRITICAL]
  github_token                   GitHub Personal Access Token
                                  value=ghp_9a...***  pos=char 15-55
                                  verify=INVALID (GitHub API rejected the token (401))

Opt-in only — never runs from a plain dlp-scan/scan() call, and the CLI prints a warning the first time --verify fires. It's off by default because it makes real network requests using the extracted secret value.

What's checked: GitHub PATs, Slack tokens, Stripe (secret/restricted) keys, SendGrid, HuggingFace, npm, Google API keys, Telegram bot tokens, Mailgun, and Cloudflare API tokens — plus two paired credential types:

• AWS access/secret keys — an access key ID alone can't authenticate anything; it needs its secret key signed alongside it. --verify finds an aws_access_key and aws_secret_key near each other in the same file (within ~2000 characters — comfortably a few lines of a config/env file) and verifies the pair with a single AWS Signature Version 4 -signed call to STS GetCallerIdentity — implemented with stdlib hmac/hashlib, no boto3 dependency. Temporary (ASIA-prefixed) credentials additionally look for a paired aws_session_token nearby; without one, signing would fail even for a genuinely live credential, so it's reported unverifiable rather than a wrong invalid.
• Twilio Account SID + Auth Token — paired the same way, verified with a Basic-auth GET against the account's own Twilio API endpoint.

An access key (or SID/token) with no partner found nearby is reported unverifiable, not silently skipped — pairing never guesses across file or commit boundaries (an access key in one file is never checked against a secret key from another).

What's deliberately not checked, and why:

• Slack/Discord webhook URLs — "verifying" a webhook means POSTing to it, which sends a real message to someone's channel. That's a side effect, not a check, so it's never attempted.
• A network error (timeout, DNS failure) is always reported as error, never collapsed into invalid — not knowing is not the same as revoked.

Same verify= line appears in directory and --history output. In directory/history mode, an identical secret found in multiple files or commits is checked against its provider once, not once per occurrence.

Python API:

import dlp_patterns

result = dlp_patterns.scan(code, secrets_only=True)
dlp_patterns.verify(result)  # mutates findings in place

for f in result.all:
    if f.verification:
        print(f.type, f.verification.status, f.verification.detail)

Pre-commit hook

Block commits containing secrets automatically using dlp-pre-commit:

repos:
  - repo: https://github.com/SpiderCob/dlp-pre-commit
    rev: v1.0.0
    hooks:
      - id: dlp-scan-secrets-only

pip install pre-commit
pre-commit install

Use in CI (GitHub Actions)

The easiest way is the official dlp-scan-action:

- name: DLP Secret Scan
  uses: spidercob/dlp-scan-action@v1
  with:
    secrets-only: 'true'
    fail-on: 'critical'

Or run the CLI directly:

- name: DLP secret scan
  run: |
    pip install dlp-patterns
    dlp-scan --secrets-only --json src/ | tee dlp-report.json

Advanced — use Scanner directly

from dlp_patterns import Scanner

scanner = Scanner()

# Reuse the same instance (compiled patterns cached)
for text in documents:
    result = scanner.scan(text)
    if result.has_findings:
        print(result.to_dict())

Enterprise

Need a full DLP platform with dashboards, audit logs, ICAP proxy integration, Gmail/Slack scanning, compliance reports, and AI-powered analysis?

→ Spidercob — the enterprise DLP platform this library is extracted from.

License

Apache 2.0 — free for commercial use.