can someone point me to a link showing that 2 n=20 studies showing the same thing (replication) is better than a single n=40 study?

— Gavin Buckingham (@DrGBuckingham) December 19, 2016

Each experiment tests for 20 variables and I computed the correlation for all possible pairs (190 independent correlations). There were 20 participants in each of the two experiments for the replication simulations and 40 participants in the single experiment case.

## Without corrections for multiple comparisons

**1 in 4 experiments (25%) allow you to replicate a significant correlations**between two variables in the two independent experiments. Note that this does take into account the fact that, to be replicated, the significant correlations must have the same sign.

When you choose a bigger group without replication, you get

**9.5 significant correlations per experiment**

## With corrections for multiple comparisons (Bonferroni)

**0.01% of the experiments allow you to replicate a significant correlations**between two variables in the two independent experiments. Note that this does take into account the fact that, to be replicated, the significant correlations must have the same sign.

When you choose a bigger group without replication,

**5% of the experiments allow you to get a significant experiment**, which is expected value given our significance threshold of 0.05.

## Response: go for replications if false positive errors are really important for you.

## Matlab code used for the simulations

internalreplication.m |

## Contents

Nsim = 10000;%%% number of simulations Nsub = 20;%%% number of participants in each group Nvar = 20;%%% number of variables that you measure

## two repliactions, only Nsub per group

for k=1:Nsim, % random variables - two studies A and B A= randn(Nsub,Nvar); B= randn(Nsub,Nvar); % correlation between each variable for each of the two studies % separately [Ra,pa]=corrcoef(A); [Rb,pb]=corrcoef(B); % extracting the relevant p-values and correlations (upper triangular matrix without diagonal) Ta = triu(pa,1); Tb = triu(pb,1); Rpa = triu(Ra,1); Rpb = triu(Rb,1); % matrix to vectors for non-zero entries Ca = Ta(Ta~=0); Cb = Tb(Tb~=0); Rca = Rpa(Ta~=0); Rcb = Rpb(Tb~=0); % detecting significant correlations PVa = Ca<0.05; PVb = Cb<0.05; % checking that the correlations have the same sign Sab = sign(Rca).*sign(Rcb)>0; % detecting when the two same variables correlate in the two % replications Sig2(k) = sum(PVa.*PVb.*Sab); end % average number of correlated pairs dectected across the two replications % Bear in mind, there should be none... disp(['average number of significant correlations present in both replications: ' num2str(mean(Sig2))])

average number of significant correlations present in both replications: 0.2388

## one experiment with 2*Nsub per group

for k=1:Nsim, % random variables C= randn(2*Nsub,Nvar); % correlation between each variable [Rc,pc]=corrcoef(C); % extracting the relevant p-values (upper triangular matrix without diagonal) Tc = triu(pc,1); % matrix to vectors for non-zero entries Cc = Tc(Tc~=0); % detecting significant correlations PVc = Cc<0.05; % detecting when the two same variables correlate in the two % replications Sig1(k) = sum(PVc); end % average number significant correlations detected % Bear in mind, there should be none... disp(['average number of significant correlations present in one bigger studies: ' num2str(mean(Sig1))])

average number of significant correlations present in one bigger studies: 9.4543