Modular FASTQ preprocessing pipeline for quality control (QC) and adapter trimming of NGS sequencing data using FastQC, MultiQC, and Cutadapt.
FASTQ-preprocess is a lightweight, modular shell pipeline for standardized preprocessing of raw NGS sequencing data. It automates per-sample quality assessment and adapter/quality trimming as a first step before any downstream bioinformatics analysis (alignment, variant calling, metagenomics, etc.).
Key features:
- Parallel quality control with FastQC
- Consolidated multi-sample QC summary with MultiQC
- Adapter and quality trimming with Cutadapt
- Supports both single-end and paired-end FASTQ files
- Designed for batch processing of large sample sets on HPC environments
FASTQ-preprocess/
├── scripts/
│ └── preprocessing.sh # Main preprocessing pipeline script
├── data/ # Input FASTQ files (not tracked in git)
├── LICENSE
└── README.md
Raw FASTQ Files (.fastq.gz)
│
▼
1. Quality Control (FastQC)
└─ Per-sample QC report: sequence quality, GC content,
adapter content, duplication rates
│
▼
2. Aggregate QC Report (MultiQC)
└─ Multi-sample summary HTML dashboard
│
▼
3. Adapter & Quality Trim (Cutadapt)
└─ Removes Illumina adapters, quality trims (Phred <20),
filters reads below minimum length threshold
│
▼
4. Post-Trim QC (FastQC + MultiQC)
└─ Verify trimming quality
│
▼
Trimmed FASTQ files ready for downstream analysis
| Tool | Version | Install |
|---|---|---|
| FastQC | ≥ 0.11.9 | conda install -c bioconda fastqc |
| MultiQC | ≥ 1.14 | pip install multiqc |
| Cutadapt | ≥ 4.0 | pip install cutadapt |
| Bash | ≥ 4.0 | Pre-installed on Linux |
# Clone the repository
git clone https://github.com/rvrane/FASTQ-preprocess.git
cd FASTQ-preprocess
# Make the script executable
chmod +x scripts/preprocessing.sh
# Install tools via conda (recommended)
conda install -c bioconda fastqc cutadapt multiqcbash scripts/preprocessing.sh \
--input /path/to/raw_fastq/ \
--output /path/to/output/ \
--threads 8| Parameter | Description | Default |
|---|---|---|
--input |
Directory containing raw .fastq.gz files |
Required |
--output |
Output directory for trimmed files and QC reports | Required |
--threads |
Number of parallel threads | 4 |
--min_length |
Minimum read length after trimming (bp) | 50 |
--quality |
Phred quality score cutoff for trimming | 20 |
output/
├── fastqc_raw/ # FastQC reports (pre-trim)
├── multiqc_raw/ # MultiQC summary (pre-trim)
├── trimmed/ # Trimmed FASTQ files
├── fastqc_trimmed/ # FastQC reports (post-trim)
└── multiqc_trimmed/ # MultiQC summary (post-trim)
# Run preprocessing on a batch of samples
bash scripts/preprocessing.sh \
--input /data/raw_fastq/ \
--output /data/preprocessed/ \
--threads 16 \
--min_length 50 \
--quality 20After the run, check multiqc_trimmed/multiqc_report.html for a QC summary of all samples.
- Adapter sequences are auto-detected by Cutadapt when not specified (recommended for Illumina data)
- For paired-end data, both R1 and R2 files are automatically processed together
- MultiQC aggregates all FastQC reports into a single browsable HTML report
- Samples failing minimum read length or quality thresholds will be flagged in the MultiQC report
- Andrews S. FastQC: A quality control tool for high throughput sequence data. https://www.bioinformatics.babraham.ac.uk/projects/fastqc/
- Ewels P, et al. MultiQC: Summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016;32(19):3047-3048.
- Martin M. Cutadapt removes adapter sequences from high-throughput sequencing reads. EMBnet.journal. 2011;17(1):10-12.
Rugved Rane
Senior Bioinformatician | Clinical Genomics
GitHub
MIT License — see LICENSE for details.