1000 Genomes Full Pipeline Example¶
This example demonstrates Stage 3 of the GRiD workflow: dynamically generating config.yaml and running the complete GRiD pipeline on the 1000 Genomes Project dataset for the LPA KIV-2 locus (GRCh38).
The full workflow comprises three sequential modular scripts:
Data Preparation Example: Downloads inputs, streams CRAM slices, and writes locus coordinates.
IBS/IBD Neighbor Computation Example: (Optional) Computes IBS neighbors for haploid copy number estimation.
GRiD Execution: Generates configuration and runs GRiD steps (indexing, read counting, depth normalization, and genotype calling).
Stage 3 Execution Script¶
03-run-grid.sh checks for the presence of data/ibs_neighbors_chr6.tsv.gz from Stage 2. If present, haploid estimation (compute_haploid_genotypes.run) is set to True. Otherwise, it falls back to diploid-only mode.
1#!/bin/bash
2#SBATCH --job-name=grid_run
3#SBATCH --output=slurm/%x.out
4#SBATCH --error=slurm/%x.err
5#SBATCH --time=4-00:00:00
6#SBATCH --cpus-per-task=8
7#SBATCH --mem=200G
8
9# Stage 3 of 3 — generate config.yaml and run the GRiD pipeline.
10# Submit after 01-prep-data.sh (and, if you want haploid CN, 02-run-ibs.sh)
11# have both completed.
12
13set -euo pipefail
14
15module load anaconda mosdepth samtools
16
17WORK_DIR="$(pwd)"
18CRAM_DIR="$WORK_DIR/crams"
19LOG_DIR="$WORK_DIR/logs"
20DATA_DIR="$WORK_DIR/data"
21OUTPUT_DIR="$WORK_DIR/output"
22MOSDEPTH_WORK="$WORK_DIR/mosdepth_work"
23mkdir -p "$OUTPUT_DIR" "$MOSDEPTH_WORK" "$LOG_DIR" "$WORK_DIR/slurm"
24
25THREADS="${SLURM_CPUS_PER_TASK:-$(nproc)}"
26
27timestamp() { date '+%Y-%m-%d %H:%M:%S'; }
28log() { echo "[$(timestamp)] $*" | tee -a "$LOG_DIR/pipeline.log"; }
29
30# --- Load locus coordinates written by 01-prep-data.sh ---
31VARS_FILE="$DATA_DIR/pipeline_vars.sh"
32if [[ ! -f "$VARS_FILE" ]]; then
33 echo "ERROR: $VARS_FILE not found — run 01-prep-data.sh first."
34 exit 1
35fi
36source "$VARS_FILE"
37
38REF_FA="$DATA_DIR/GRCh38_no_alt.fa"
39REPEAT_MASK="$DATA_DIR/repeat_mask.bed"
40GRID_SAMPLES_FILE="$DATA_DIR/grid_samples.txt"
41NORM_REGIONS_FILE="$OUTPUT_DIR/norm_regions.gc.tsv"
42
43for f in "$REF_FA" "$GRID_SAMPLES_FILE"; do
44 if [[ ! -f "$f" ]]; then
45 echo "ERROR: expected input $f not found — run 01-prep-data.sh first."
46 exit 1
47 fi
48done
49
50module load anaconda
51eval "$(micromamba shell hook --shell bash)"
52micromamba activate grid-env
53
54if ! command -v grid &>/dev/null; then
55 echo "ERROR: 'grid' command not found after activating grid-test."
56 echo "Check that 'pip install -e .' completed successfully in that env."
57 exit 1
58fi
59
60# --- Check for IBS output from stage 2 (optional — diploid-only if absent) ---
61IBS_OUTPUT="$DATA_DIR/ibs_neighbors_chr${CHR}.tsv.gz"
62HAPLOID_RUN="False"
63if [[ -f "$IBS_OUTPUT" ]]; then
64 log "IBS neighbors file found: $IBS_OUTPUT"
65 log "Haploid CN estimation will run."
66 HAPLOID_RUN="True"
67else
68 log "IBS neighbors file not found at $IBS_OUTPUT."
69 log "Run 02-run-ibs.sh first if you want haploid CN estimation."
70 log "Proceeding with diploid-only for this run."
71fi
72
73# Empty mask: the downloaded repeat_mask.bed spans nearly the entire LPA
74# window (KIV-2 is itself a massive tandem repeat), which zeroes out every
75# normalization bin if applied here. Using an empty mask for this locus is
76# the correct choice, not a workaround being skipped.
77EMPTY_MASK="$DATA_DIR/repeat_mask_empty.bed"
78touch "$EMPTY_MASK"
79
80CONFIG="$WORK_DIR/config.yaml"
81
82cat > "$CONFIG" << YAML
83# Auto-generated GRiD config — 1000 Genomes LPA KIV-2 example
84samples_file: "$GRID_SAMPLES_FILE"
85directory_loc: "$CRAM_DIR"
86reference_genome: "$REF_FA"
87output_dir: "$OUTPUT_DIR"
88threads: $THREADS
89file_type: "cram"
90chrom: "chr${CHR}"
91start_bp: $START
92end_bp: $END
93output_file_type: "txt"
94
95index:
96 run: True
97 output_file_prefix: "index_file_results"
98
99count_reads:
100 run: True
101 output_file_prefix: "read_counts"
102 min_mapq: 1
103 flags:
104 - 83 # proper pair, read reverse strand
105 - 147 # proper pair, mate reverse strand
106 - 81 # read reverse strand
107 - 145 # mate reverse strand
108
109mosdepth:
110 run: True
111 output_file_prefix: "mosdepth_results"
112 bin_size: 1000
113 mode: "fast"
114 region_name: "LPA"
115 work_dir: "$MOSDEPTH_WORK"
116 # Keep intermediate files; gc_normalize (phase 2) reads .regions.bed.gz.
117 remove_intermediate: False
118
119 normalize:
120 run: True
121 min_depth: 1
122 max_depth: 30
123 top_frac: 0.9
124 output_file_prefix: "mosdepth_normalized"
125 repeat_mask_file: "$EMPTY_MASK"
126
127 neighbors:
128 run: True
129 output_file_prefix: "neighbors"
130 num_neighbors: 5
131 zmax: 2.0
132 sigma2_max: 1000
133
134 gc_normalize:
135 run: False
136
137compute_diploid_genotypes:
138 run: True
139 output_file_prefix: "diploid_genotypes"
140
141compute_haploid_genotypes:
142 run: $HAPLOID_RUN
143 output_file_prefix: "haploid_genotypes"
144 ibs_output: "$IBS_OUTPUT"
145 min_neighbors: 1
146 max_neighbors: 10
147 n_iters: 20
148 method: "ibs"
149YAML
150
151log "Running GRiD pipeline"
152grid wgs "$CONFIG"
153log "Phase 1 complete."
Full Pipeline Usage¶
Run all three stages sequentially from your working directory:
# Step 1: Prepare data and stream CRAMs
sbatch examples/01-prep-data.sh
# Step 2: (Optional) Compute IBS neighbors for haploid CN
sbatch examples/02-run-ibs.sh
# Step 3: Generate config and run GRiD
sbatch examples/03-run-grid.sh