A binomial test of group differences with correlated outcome measures

Building on previous arguments for why educational researchers should not provide effect-size estimates in the face of statistically nonsignificant outcomes (Robinson & Levin, 1997), Onwuegbuzie and Levin (2005) proposed a 3-step statistical approach for assessing group differences when multiple outcome measures are individually analyzed within the same study. One suggested Step 3 strategy was to conduct a binomial (or "sign") test of the number of between-group outcome mean differences that are in the same direction. However, because multiple measures within a study typically are correlated, the binomial test's independence assumption will be violated. In the present investigation, the authors (a) performed a Monte Carlo simulation study to assess the Type I error behavior of the binomial test under varying degrees of independence-assumption violations, resulting in a table of adjusted critical values; and (b) illustrated use of this table by applying its adjusted critical values to a real research example.