Advice: Choosing a multiple comparisons test

Multiple comparisons after ordinary one-way ANOVA

If you are comparing three or more groups with one-way ANOVA, you may pick a post test to compare pairs of group means. Choosing a multiple comparisons test is not 100% straightforward, so you may receive different recommendations depending on who you ask. Here are our recommendations:

Select Dunnett's test if one column represents control data and you wish to compare all other columns to that control column but not to each other.

Select the test for linear trend if the columns are arranged in a natural order (e.g. dose or time) and you want to test whether there is a trend such that values increase (or decrease) as you move from left to right across columns.

Select the Bonferroni test for selected pairs of columns when you only wish to compare certain column pairs. You must select those pairs based on experimental design and ideally should specify the pairs of interest before collecting any data. If you base your decision on the results (e.g., compare the smallest with the largest mean), then you have effectively compared all columns, and it is not appropriate to use the test for selected pairs.

If you want to compare all pairs of columns, choose the Tukey test. This is actually the Tukey-Kramer test, which includes the extension by Kramer to allow for unequal sample sizes. We recommend you don't use the Newman-Keuls test or the Bonferroni test used to compare every pair of groups.

Multiple comparisons after repeated measures one-way ANOVA

The discussion above can also help you choose a multiple comparisons test after repeated measures one-way ANOVA.

All the multiple comparisons tests are based on the assumption that the values after each treatment were randomly drawn from populations with the same amount of scatter. But with some repeated measures designs, the scatter trends to increase with each sequential treatment. If the scatter systematically increases with each repeated treatment, then the multiple comparisons performed by Prism are not valid. There is an alternative approach to computing multiple comparisons, but this is not implemented in Prism. With this alternative test, each pair of groups is compared with a paired t test (without using any of the ANOVA results or degrees of freedom), with multiple comparisons being used to select the threshold for defining when a P value is low enough to be deemed "significant".

Multiple comparisons after nonparametric ANOVA

Prism only offers Dunn's post test, either to compare all pairs of groups, or just selected pairs.

For more information, see Applied Nonparametric Statistics by WW Daniel, published by PWS-Kent publishing company in 1990 or Nonparametric Statistics for Behavioral Sciences by S Siegel and NJ Castellan, 1988. The original reference is O.J. Dunn, Technometrics, 5:241-252, 1964.

Prism refers to the post test as the Dunn's post test. Some books and programs simply refer to this test as the post test following a Kruskal-Wallis test, and don't give it an exact name.