Fig. 1

a Null distribution of the PERMANOVA test statistic. Asymptotic distribution of \(\tilde{\text {F}}_{AB}\) (green solid line) obtained by simulation as proposed in (3), scaled by \(\sum_{j=1}^q \lambda _j\), compared to the empirical distribution obtained using 10\(^6\) permutations (red dashed line). Simulation details: \(n =\) 300, \(q =\) 3, model (5), \(\varvec{y} \sim \mathcal {N}({\textbf {0}}, {\textbf {I}}_q)\) with factor B simulated under \(H_1\) and \(\Delta =\) 1 (see the “Methods” section). The upper tail of the distribution is zoomed-in. b Relative bias of asymptotic p values vs n/q ratio. Relative difference between asymptotic (p\(_A\)) and permutation-based (p\(_P\), 10\(^5\) permutations) p values for the interaction term (AB) as a function of the ratio between the total sample size and the number of dependent variables (n/q). We considered values of n ranging from 20 to 300, and values of q ranging from 2 to 20. For visualization purposes, we show values of \(n/q \in\) [0,50] and relative biases \(\in\) [− 1,0.5]. The horizontal solid red line marks the 0. The horizontal dashed red lines mark the 5% relative bias. A polynomial was fitted to the points using local fitting (LOESS), in order to describe the trend (fit in green, 95% confidence interval in gray). c Comparison of asymptotic and permutation-based p values when the asymptotic null holds (\(n =\) 300, \(q =\) 3). d Empirical running time as a function of sample size (n) for the asymptotic and permutation-based approaches. Each point corresponds to the mean running time across 5 runs with different input data (see the “Methods” section). Error bars represent the standard error of the mean (i.e., mean ± SEM). Axes are in \(\log _{10}\) scale. Dashed lines represent running time growth rates of n, \(n^2\) and \(n^3\)