Skip to main content

Table 5 Notation used in ‘Methods’

From: Schmutzi: estimation of contamination and endogenous mitochondrial consensus calling for ancient DNA

Symbol

Definition

\(\mathbb {R}\)

Set of all fragments

\(\mathbb {E}\)

Set of all fragments from the endogenous genome

R j

a particular fragment in R, with l bases \(\{ r_{1}, \dots, r_{l} \}\) and respective error probabilities \(\{ \epsilon _{1}, \dots, \epsilon _{l} \}\), which are given by the per-base quality scores

E

The event that a sequencing error has occurred

D

The event that deamination has occurred

C

The event that R j was sampled from a contaminant mitochondrial genome

M

The event that R j was correctly mapped

\(m_{R_{j}}\)

Probability that R J is mismapped (P[¬M])

b e

The base from the endogenous genome

b c

The base from the contaminant genome

c

The base from the contaminant genome used by mtCont, obtained from a database

r i

The base at position i from fragment R j

ε i

The probability that base r i has a sequencing error as determined by the base caller

¬

Denotes the complement of an event (event has not occurred)

c d

Contamination rate, estimated by contDeam

c r

Contamination rate, estimated by mtCont

c c

Prior on contamination rate provided as input to endoCaller

endodist

log-normal distribution of the fragment length for the endogenous fragments

contdist

log-normal distribution of the fragment length for the contaminant fragments