Getting Started

GRiD is driven by a single YAML configuration file. This page walks through every section — what each parameter does, why it is needed, and how to obtain or prepare the required input files.

Copy the bundled example and edit it for your cohort:

cp grid/example_config.yaml my_config.yaml
# edit my_config.yaml, then:
grid wgs my_config.yaml

Global Settings

These fields apply to every pipeline step.

samples_file: "path/to/sample_ids.txt"
directory_loc: "path/to/crams/"
reference_genome: "path/to/hg38.fa"
output_dir: "path/to/output/"
threads: 8
file_type: "cram"
chrom: "chr6"
start_bp: 160605062
end_bp: 160647661
output_file_type: "tsv"

Parameter

Type

Description

samples_file

path

Plain-text file with one sample ID per line. IDs must match the file stems of the CRAM/BAM files in directory_loc.

directory_loc

path

Directory containing the CRAM (or BAM) files for your cohort.

reference_genome

path

Reference genome FASTA (hg19 or hg38). Must be indexed (samtools faidx). Download from UCSC, the Broad Institute, or 1000 Genomes.

output_dir

path

Directory where all pipeline outputs are written. Created if it does not exist.

threads

int

Number of CPU threads shared across steps. Match to your SLURM --cpus-per-task allocation.

file_type

str

Sequencing file format: "cram" or "bam".

chrom

str

Target chromosome in the same notation as your CRAM headers (e.g. "chr6" for hg38, "6" for hg19).

start_bp

int

Start coordinate (0-based) of the VNTR region of interest. For LPA KIV-2 in hg38: 160605062.

end_bp

int

End coordinate of the VNTR region of interest. For LPA KIV-2 in hg38: 160647661.

output_file_type

str

Output table format: "tsv" or "csv".

Note

Supported VNTR loci. Mukamel et al. (2021) Science (doi:10.1126/science.abg8289) catalogued 734 coding VNTRs with elevated IBD2 rates across the human genome, identifying loci where copy number is likely to vary and be heritable. Their full code is available at Zenodo 4776804.

The same file is bundled with GRiD at 734_possible_coding_vntr_regions.IBD2R_gt_0.25.uniq.txt. To run GRiD on a different locus, simply update the chrom, start_bp, and end_bp fields in your config file to match the coordinates of your target region.


Step 1 — index

Verifies or creates .crai / .bai index files for all CRAM/BAM files in directory_loc. Only needed if your files are not already indexed.

index:
  run: False
  output_file_prefix: "index_file_results"

Parameter

Description

run

Set True to run this step. Skip if files are already indexed.

output_file_prefix

Prefix for the log output file.

Step 2 — count_reads

Counts properly paired reads overlapping the target VNTR region from each CRAM/BAM file. The resulting per-sample read counts are used downstream as a proxy for locus-specific sequencing depth.

count_reads:
  run: True
  output_file_prefix: "read_counts"
  min_mapq: 1
  flags:
    - 83    # proper pair, read reverse strand
    - 147   # proper pair, mate reverse strand
    - 81    # read reverse strand
    - 145   # mate reverse strand

Parameter

Description

run

Set True to run this step.

output_file_prefix

Prefix for the output read-count file.

min_mapq

Minimum mapping quality. Reads below this threshold are ignored. 1 excludes unmapped reads while retaining multi-mappers; higher values (e.g. 20) restrict to uniquely mapped reads.

flags

List of SAM bitwise flags to include. Only reads whose flag exactly matches one of the listed values are counted. The defaults capture properly paired reads on both strands. See SAM flag documentation for flag definitions.

Step 3 — mosdepth

Runs mosdepth in windowed mode to compute binned read depth across the full genome for each sample. The resulting depth profiles are used for normalization and neighbor finding.

mosdepth must be available on your PATH:

conda install -c bioconda mosdepth
# or: module load mosdepth  (HPC)
mosdepth:
  run: True
  output_file_prefix: "mosdepth_results"
  bin_size: 1000
  mode: "fast"
  region_name: "LPA"
  work_dir: "path/to/mosdepth_work/"
  remove_intermediate: True

Parameter

Description

run

Set True to run mosdepth.

output_file_prefix

Prefix for the aggregated depth output file.

bin_size

Genomic bin size in base pairs. 1000 (1 kb) is recommended — smaller bins increase resolution but memory and runtime.

mode

"fast" skips per-base depth (faster, less memory) or "full" for per-base output. "fast" is sufficient for normalization.

region_name

Label applied to the VNTR region in output files (e.g. "LPA").

work_dir

Scratch directory for per-sample mosdepth intermediate files. Should be on fast local or scratch storage.

remove_intermediate

If True, per-sample mosdepth files are deleted after aggregation to save disk space.

Step 3a — mosdepth.normalize

Normalizes the depth matrix: filters low/high-coverage bins, applies a repeat-mask, and z-scores across individuals. See Coverage Normalization for the full method.

mosdepth:
  normalize:
    run: True
    min_depth: 20
    max_depth: 100
    top_frac: 0.1
    output_file_prefix: "mosdepth_results_normalized"
    repeat_mask_file: "path/to/repeat_mask.bed"

Parameter

Description

run

Set True to normalize after mosdepth.

min_depth

Bins with median depth below this value are excluded.

max_depth

Bins with median depth above this value are excluded.

top_frac

Fraction of bins with the highest variance to retain for normalization (e.g. 0.1 keeps the top 10 % most variable bins).

output_file_prefix

Prefix for the normalized depth matrix output.

repeat_mask_file

BED file of repetitive regions to exclude before normalization. Download UCSC RepeatMasker tracks from the UCSC Table Browser (group: Repeats, track: RepeatMasker).

Step 3b — mosdepth.neighbors

Finds read depth-matched nearest neighbors in the normalized depth space using scikit-learn NearestNeighbors. See Nearest-Neighbor Identification.

mosdepth:
  neighbors:
    run: True
    output_file_prefix: "neighbor_coverage"
    num_neighbors: 5
    zmax: 2.0
    sigma2_max: 1000

Parameter

Description

run

Set True to run neighbor finding.

output_file_prefix

Prefix for the neighbor assignment output file.

num_neighbors

Number of nearest neighbors to retain per sample. 5 is a conservative default; increase for small or heterogeneous cohorts.

zmax

Clip normalized depth values to \(\pm\) zmax before distance computation. Reduces the influence of outlier bins. Typical range: 1.5–3.0.

sigma2_max

Maximum allowed neighbor variance. Samples whose nearest-neighbor set has variance above this threshold are flagged. Increase if many samples are being excluded in diverse cohorts.

Step 4 — compute_diploid_genotypes

Estimates diploid copy number for each sample using the neighbor-normalized depth ratio. For each sample, its locus depth is scaled by the mean depth of its nearest neighbors at the same locus. See Diploid Copy Number Estimation.

compute_diploid_genotypes:
  run: True
  output_file_prefix: "diploid_genotypes"
  n_nbr: 300

Parameter

Description

run

Set True to estimate diploid copy numbers.

output_file_prefix

Prefix for the diploid CN output file.

n_nbr

Number of nearest neighbors to use when computing each sample’s diploid CN estimate. Default: 300. Increase for small cohorts; decrease to restrict to the closest matches.

Step 5 — compute_haploid_genotypes

Decomposes diploid CN estimates into haplotype-specific copy numbers using a neighbor-based phasing algorithm. Two neighbor sources (IBS or IBD) and five phasing algorithms are supported. See Haplotype Inference for full algorithm details and Computing IBS/IBD Neighbors for how to generate the neighbors file.

IBS (original method):

compute_haploid_genotypes:
  run: True
  output_file_prefix: "haploid_genotypes"
  method: "ibs"
  phasing_method: "iterative"
  ibs_output: "path/to/ibs_neighbors_chr6.tsv.gz"
  min_neighbors: 1
  max_neighbors: 10
  n_iters: 20

IBD (recommended):

compute_haploid_genotypes:
  run: True
  output_file_prefix: "haploid_genotypes"
  method: "ibd"
  phasing_method: "graph_prop"
  ibd_output: "path/to/output.match"
  alpha: 0.7
  n_iters: 20
  max_neighbors: 20
  min_length: 2.0
  min_match: 0.95

Parameter

Description

run

Set True to run haplotype inference.

output_file_prefix

Prefix for the haploid CN output file.

method

Neighbor source: "ibs" (computeIBSpbwt output) or "ibd" (iLASH output).

phasing_method

Phasing algorithm. Options: "iterative" (original Hujoel method), "graph_prop" (recommended), "graph_prop_weighted", "oneshot", "rank_based". See Haplotype Inference for descriptions of each.

alpha

(graph_prop / graph_prop_weighted) Mixing weight between the individual’s own diploid signal and the neighbor aggregate. Range (0, 1); higher values trust the individual more. Default: 0.7.

agg_fn

Neighbor aggregation function. Options: "mean" (default), "median", "weighted_cM", "weighted_score", "weighted_cM_score", "trimmed_mean". Ignored by plain graph_prop.

n_iters

Number of phasing iterations. Ignored by oneshot. Default: 20.

min_neighbors

Minimum neighbors required on both haplotypes to phase a sample. Samples below this threshold receive mean-imputed values. Default: 1.

max_neighbors

Maximum neighbors loaded per haplotype. Default: 20.

ibs_output

(IBS method) Path to IBS neighbors file from computeIBSpbwt.

ibd_output

(IBD method) Path to iLASH .match output file.

min_length

(IBD method) Minimum IBD segment length in cM. Default: 0.5.

min_match

(IBD method) Minimum iLASH match score; lenient floor for long segments when match_at_min_length is set. Default: 0.70.

match_at_min_length

(IBD method, optional) Strict match score required at min_length cM. Enables a length-adaptive threshold that relaxes toward min_match for longer segments. Leave unset for a flat threshold.

weighted

(IBD method) If True, weight each neighbor by \(\mathrm{match}^3 \times \mathrm{cM}\) during loading. Default: False.

weight_scale

(IBD method, weighted) \(\lambda\) in bp for the Lorentzian distance weight function. Default: 1000000.

Note

Either ibs_output or ibd_output must be provided depending on the chosen method. See Computing IBS/IBD Neighbors for file format details and how to generate each neighbors file.


Full Example Config

grid/example_config.yaml
 1samples_file: "path/to/sample_ids.txt"
 2directory_loc: "path/to/crams/"
 3reference_genome: "path/to/hg38.fa"
 4output_dir: "path/to/output/"
 5threads: 4
 6file_type: "cram"
 7
 8# VNTR region of interest
 9chrom: "chr6"
10start_bp: 160605062
11end_bp: 160647661
12
13output_file_type: "tsv"
14
15index:
16  run: False
17  output_file_prefix: "index_file_results"
18
19count_reads:
20  run: True
21  output_file_prefix: "read_counts"
22  min_mapq: 1
23  flags:
24    - 83   # proper pair, reverse strand
25    - 99   # mate forward, reverse strand (unpaired-ish)
26    - 147  # proper pair, mate reverse
27    - 163  # mate forward, reverse strand (unpaired-ish)
28    - 81   # reverse, mate forward (unpaired-ish)
29    - 97   # forward, mate reverse (unpaired-ish)
30    - 145  # mate reverse
31    - 161  # mate forward
32
33mosdepth:
34  run: True
35  output_file_prefix: "mosdepth_results"
36  bin_size: 1000
37  work_dir: "path/to/mosdepth_work/"
38  remove_intermediate: True
39  example_bed: null
40
41  normalize:
42    run: True
43    output_file_prefix: "ID_scale_zdepths"
44    combined_bins_gz: null  # optional, defaults to output_dir/mosdepth_bins.txt.gz
45    repeat_mask_file: "path/to/repeat_mask_list.hg38.ucsc_bed"
46
47  neighbors:
48    run: True
49    output_file_prefix: "neighbor_coverage"
50    num_neighbors: 5
51    zmax: 2.0
52    sigma2_max: 1000
53
54  gc_normalize:
55    run: False
56
57compute_diploid_genotypes:
58  run: True
59  output_file_prefix: "diploid_genotypes"
60  n_nbr: 300
61
62compute_haploid_genotypes:
63  run: True
64  output_file_prefix: "haploid_genotypes"
65  method: "ibs"          # "ibs" or "ibd"
66  min_neighbors: 1
67  max_neighbors: 10
68  n_iters: 100
69
70  # IBS method: provide computeIBSpbwt output
71  ibs_output: "path/to/ibs_neighbors_chr6.tsv.gz"
72
73  # IBD method: provide iLASH output and filtering thresholds
74  # ibd_output: "path/to/ilash_output.txt"
75  # weighted: False         # True = Lorentzian distance+match weighting
76  # weight_scale: 1000000   # bp half-width for distance weight
77  # min_length: 0.5         # minimum IBD segment length (cM)
78  # min_match: 0.70         # minimum IBD match score