1000 Genomes Full Pipeline Example

This example demonstrates Stage 3 of the GRiD workflow: dynamically generating config.yaml and running the complete GRiD pipeline on the 1000 Genomes Project dataset for the LPA KIV-2 locus (GRCh38).

The full workflow comprises three sequential modular scripts:

  1. Data Preparation Example: Downloads inputs, streams CRAM slices, and writes locus coordinates.

  2. IBS/IBD Neighbor Computation Example: (Optional) Computes IBS neighbors for haploid copy number estimation.

  3. GRiD Execution: Generates configuration and runs GRiD steps (indexing, read counting, depth normalization, and genotype calling).

Stage 3 Execution Script

03-run-grid.sh checks for the presence of data/ibs_neighbors_chr6.tsv.gz from Stage 2. If present, haploid estimation (compute_haploid_genotypes.run) is set to True. Otherwise, it falls back to diploid-only mode.

03-run-grid.sh — Stage 3: Config generation and pipeline execution
  1#!/bin/bash
  2#SBATCH --job-name=grid_run
  3#SBATCH --output=slurm/%x.out
  4#SBATCH --error=slurm/%x.err
  5#SBATCH --time=4-00:00:00
  6#SBATCH --cpus-per-task=8
  7#SBATCH --mem=200G
  8
  9# Stage 3 of 3 — generate config.yaml and run the GRiD pipeline.
 10# Submit after 01-prep-data.sh (and, if you want haploid CN, 02-run-ibs.sh)
 11# have both completed.
 12
 13set -euo pipefail
 14
 15module load anaconda mosdepth samtools
 16
 17WORK_DIR="$(pwd)"
 18CRAM_DIR="$WORK_DIR/crams"
 19LOG_DIR="$WORK_DIR/logs"
 20DATA_DIR="$WORK_DIR/data"
 21OUTPUT_DIR="$WORK_DIR/output"
 22MOSDEPTH_WORK="$WORK_DIR/mosdepth_work"
 23mkdir -p "$OUTPUT_DIR" "$MOSDEPTH_WORK" "$LOG_DIR" "$WORK_DIR/slurm"
 24
 25THREADS="${SLURM_CPUS_PER_TASK:-$(nproc)}"
 26
 27timestamp() { date '+%Y-%m-%d %H:%M:%S'; }
 28log() { echo "[$(timestamp)] $*" | tee -a "$LOG_DIR/pipeline.log"; }
 29
 30# --- Load locus coordinates written by 01-prep-data.sh ---
 31VARS_FILE="$DATA_DIR/pipeline_vars.sh"
 32if [[ ! -f "$VARS_FILE" ]]; then
 33    echo "ERROR: $VARS_FILE not found — run 01-prep-data.sh first."
 34    exit 1
 35fi
 36source "$VARS_FILE"
 37
 38REF_FA="$DATA_DIR/GRCh38_no_alt.fa"
 39REPEAT_MASK="$DATA_DIR/repeat_mask.bed"
 40GRID_SAMPLES_FILE="$DATA_DIR/grid_samples.txt"
 41NORM_REGIONS_FILE="$OUTPUT_DIR/norm_regions.gc.tsv"
 42
 43for f in "$REF_FA" "$GRID_SAMPLES_FILE"; do
 44    if [[ ! -f "$f" ]]; then
 45        echo "ERROR: expected input $f not found — run 01-prep-data.sh first."
 46        exit 1
 47    fi
 48done
 49
 50module load anaconda
 51eval "$(micromamba shell hook --shell bash)"
 52micromamba activate grid-env
 53
 54if ! command -v grid &>/dev/null; then
 55    echo "ERROR: 'grid' command not found after activating grid-test."
 56    echo "Check that 'pip install -e .' completed successfully in that env."
 57    exit 1
 58fi
 59
 60# --- Check for IBS output from stage 2 (optional — diploid-only if absent) ---
 61IBS_OUTPUT="$DATA_DIR/ibs_neighbors_chr${CHR}.tsv.gz"
 62HAPLOID_RUN="False"
 63if [[ -f "$IBS_OUTPUT" ]]; then
 64    log "IBS neighbors file found: $IBS_OUTPUT"
 65    log "Haploid CN estimation will run."
 66    HAPLOID_RUN="True"
 67else
 68    log "IBS neighbors file not found at $IBS_OUTPUT."
 69    log "Run 02-run-ibs.sh first if you want haploid CN estimation."
 70    log "Proceeding with diploid-only for this run."
 71fi
 72
 73# Empty mask: the downloaded repeat_mask.bed spans nearly the entire LPA
 74# window (KIV-2 is itself a massive tandem repeat), which zeroes out every
 75# normalization bin if applied here. Using an empty mask for this locus is
 76# the correct choice, not a workaround being skipped.
 77EMPTY_MASK="$DATA_DIR/repeat_mask_empty.bed"
 78touch "$EMPTY_MASK"
 79
 80CONFIG="$WORK_DIR/config.yaml"
 81
 82cat > "$CONFIG" << YAML
 83# Auto-generated GRiD config — 1000 Genomes LPA KIV-2 example
 84samples_file: "$GRID_SAMPLES_FILE"
 85directory_loc: "$CRAM_DIR"
 86reference_genome: "$REF_FA"
 87output_dir: "$OUTPUT_DIR"
 88threads: $THREADS
 89file_type: "cram"
 90chrom: "chr${CHR}"
 91start_bp: $START
 92end_bp: $END
 93output_file_type: "txt"
 94
 95index:
 96  run: True
 97  output_file_prefix: "index_file_results"
 98
 99count_reads:
100  run: True
101  output_file_prefix: "read_counts"
102  min_mapq: 1
103  flags:
104    - 83    # proper pair, read reverse strand
105    - 147   # proper pair, mate reverse strand
106    - 81    # read reverse strand
107    - 145   # mate reverse strand
108
109mosdepth:
110  run: True
111  output_file_prefix: "mosdepth_results"
112  bin_size: 1000
113  mode: "fast"
114  region_name: "LPA"
115  work_dir: "$MOSDEPTH_WORK"
116  # Keep intermediate files; gc_normalize (phase 2) reads .regions.bed.gz.
117  remove_intermediate: False
118
119  normalize:
120    run: True
121    min_depth: 1
122    max_depth: 30
123    top_frac: 0.9
124    output_file_prefix: "mosdepth_normalized"
125    repeat_mask_file: "$EMPTY_MASK"
126
127  neighbors:
128    run: True
129    output_file_prefix: "neighbors"
130    num_neighbors: 5
131    zmax: 2.0
132    sigma2_max: 1000
133
134  gc_normalize:
135    run: False
136
137compute_diploid_genotypes:
138  run: True
139  output_file_prefix: "diploid_genotypes"
140
141compute_haploid_genotypes:
142  run: $HAPLOID_RUN
143  output_file_prefix: "haploid_genotypes"
144  ibs_output: "$IBS_OUTPUT"
145  min_neighbors: 1
146  max_neighbors: 10
147  n_iters: 20
148  method: "ibs"
149YAML
150
151log "Running GRiD pipeline"
152grid wgs "$CONFIG"
153log "Phase 1 complete."

Full Pipeline Usage

Run all three stages sequentially from your working directory:

# Step 1: Prepare data and stream CRAMs
sbatch examples/01-prep-data.sh

# Step 2: (Optional) Compute IBS neighbors for haploid CN
sbatch examples/02-run-ibs.sh

# Step 3: Generate config and run GRiD
sbatch examples/03-run-grid.sh