Building a Reproducible Nextflow Pipeline for Metagenomics

A minimal, worked example of structuring a metagenomics workflow in Nextflow so it is portable and reproducible.

This is a worked example tutorial demonstrating the site’s tutorial template. Commands are shown but not executed in this build (no Nextflow runtime in the build environment) — copy them into your own environment to run.

Problem

Metagenomics pipelines often start as a chain of manually-run scripts on one person’s laptop or HPC account. That works until someone else — a collaborator, a reviewer, or future-you — needs to rerun the analysis on new data. This tutorial shows a minimal but complete Nextflow structure that avoids that trap.

Learning objectives

By the end of this tutorial you will be able to:

• Structure a two-step metagenomics workflow (QC → assembly) as a Nextflow pipeline
• Parameterize inputs so the pipeline runs on any sample sheet
• Containerize each process so results don’t depend on locally installed software versions

Prerequisites

• Nextflow ≥ 23.10 installed (curl -s https://get.nextflow.io | bash)
• Docker or Apptainer/Singularity available
• Basic familiarity with the command line

Conceptual explanation

A Nextflow pipeline is a directed graph of processes connected by channels. Each process declares its own inputs, outputs, and (ideally) its own container — so a process that runs fastp for quality control doesn’t care what’s installed on the host machine, only what’s in its container image. This is what makes the same pipeline reproducible on a laptop, a lab server, or an HPC cluster managed by Slurm.

Reproducible example

We use a small simulated paired-end FASTQ dataset (no real or confidential sample data) to demonstrate the two core processes: quality control and assembly.

Code

main.nf:

#!/usr/bin/env nextflow
nextflow.enable.dsl=2

params.reads = "data/*_R{1,2}.fastq.gz"
params.outdir = "results"

process FASTP {
    container 'quay.io/biocontainers/fastp:0.23.4--h5f740d0_0'
    publishDir "${params.outdir}/qc", mode: 'copy'

    input:
    tuple val(sample_id), path(reads)

    output:
    tuple val(sample_id), path("${sample_id}_clean_{1,2}.fastq.gz")

    script:
    """
    fastp -i ${reads[0]} -I ${reads[1]} \\
          -o ${sample_id}_clean_1.fastq.gz -O ${sample_id}_clean_2.fastq.gz
    """
}

process MEGAHIT {
    container 'quay.io/biocontainers/megahit:1.2.9--h5b5514e_3'
    publishDir "${params.outdir}/assembly", mode: 'copy'

    input:
    tuple val(sample_id), path(reads)

    output:
    path("${sample_id}_assembly")

    script:
    """
    megahit -1 ${reads[0]} -2 ${reads[1]} -o ${sample_id}_assembly
    """
}

workflow {
    reads_ch = Channel.fromFilePairs(params.reads)
    qc_ch = FASTP(reads_ch)
    MEGAHIT(qc_ch)
}

Running it on a Slurm cluster with Apptainer instead of Docker just changes the executor/engine in nextflow.config:

process.executor = 'slurm'
apptainer.enabled = true
process.container = 'library://...'

Output

results/
├── qc/
│   ├── sample1_clean_1.fastq.gz
│   └── sample1_clean_2.fastq.gz
└── assembly/
    └── sample1_assembly/
        └── final.contigs.fa

Interpretation

Each process’s output is published to a predictable path under results/, independent of where or how the pipeline was executed. The final.contigs.fa file is your assembled contigs, ready for downstream binning.

Common errors

• “No such container” errors on HPC: usually means Apptainer isn’t enabled in nextflow.config, or the image needs to be pulled/cached first.
• Channel mismatch (“Invalid method invocation”): almost always a mismatch between how fromFilePairs globs your files and your actual filenames — check the glob pattern against ls data/.
• Silent resume failures: Nextflow’s -resume relies on unchanged inputs/hash; editing a script and expecting cached results for the same step is a common surprise.

Limitations

This example omits real-world necessities you’d add for production use: host-read depletion, multiple assemblers for comparison, and resource directives (cpus, memory) tuned to your cluster’s queue policy.

References

Di Tommaso, P. et al. (2017). Nextflow enables reproducible computational workflows. Nature Biotechnology.

Environment

nextflow.config specifies container engine (docker/apptainer) and executor (local/slurm).
Container images pinned by tag (see `container` directives above) for exact reproducibility.