Overview

Prism 5.02 (Windows) and 5.0b (Mac) included a fix to a trivial bug in Fisher's exact test (when the two groups are identical, the P value should be 1.00 but earlier versions of Prism sometimes reported P values slightly greater than 1.0). Unfortunately, that fix introduced a new bug described here:

The bug occurs only when:

• You are using Prism 5.02 (Windows) or 5.0b (Mac). Earlier versions did not have this bug. Neither does InStat 3.0 or 3.1. 
• You have entered a symmetrical contingency table. A table is symmetrical when either the two row totals are identical, or the two column totals are identical. 
• You have chosen a two-tail (two-sided) P value. One-tail P values are computed correctly. 
• The sum of the four values entered into the contingency table is less than 2001. When the sum is greater than 2000, Prism performs chi-square test even if you asked for Fishers, because the Fisher computations would be so slow. 

The result of the bug is that the reported P value will be too low in some, but not all cases. In many cases, the discrepancy is tiny and won't affect your conclusions. In other cases, the discrepancy is larger and may affect your conclusion.

The problem occurs in about half the cases when the values are large (the sum of the four values exceeds 100). When the values are smaller, the problem occurs less often, but still can happen. It depends on the values entered. 

Of course, we will fix the bug in the next release of Prism: 5.03 and 5.0c.

Workaround

With symmetrical contingency tables, the two-tail P value is exactly twice the one-tail P value (that is not always true with contingency tables that are not symmetrical). Therefore, to bypass the bug, ask Prism to compute a one-tail (one-sided) P value. This is a choice in the Parameters dialog for analyzing contingency table. To compute a two-tail P value, simply double the one-tail P value. 

Example

	Outcome A	Outcome B	Total
Group 1	98	89	187
Group 2	20	29	49
Total	118	118	236

This table is symmetrical because the two column totals are identical (118 each). The table would also be symmetrical if the row totals were equal. Of course, you only enter four values into the contingency data table of Prism. You don't enter the column or row totals, which Prism computes. 

Prism 5.02 and 5.0b report that the two-tail P value is 0.1534. This is incorrect. To obtain the correct two-tail P value, ask Prism to compute the one-tail P value. The result is 0.0994. Manually double that value to obtain the correct two-tail P value, which is 0.1988.

Mathematical details

Computing a one-tail P value from a Fisher's test is straightforward, but there are three different algorithms for computing the two-tail P value. Prism (and also InStat, and the QuickCalc since 5-April-2004) compute the two-sided P value using the method of summing small P values. Most statisticians recommend this approach, but some programs use a different approach. 

The calculation works like this:

1. Create every possible contingency table with the same row and column totals as in the actual table.
2. For each table, compute the probability that these results would have occurred under the null hypothesis. These probabilities will sum to 1.00. 
3. Total all those probabilities that are less than or equal to the probability of the actual table. That sum is the two-tail P value. 

The third step is the problem. With a symmetrical contingency table, there is always another table that has exactly the same probability as the original, and its probability should be included in the total (step 3 above) that becomes the two-tail P value. However the calculations that generate that probability are done in a different order for the two equivalent tables, and computer round off issues can result in a probabilities that are not quite equal. If the other table ends up with a slightly higher probability than the original table, it is not included in the sum of probabilities of step 3, so Prism reports a two-tail P value that is smaller than it should be. The fundamental problem is that asking a computer whether A equals B is not reliable with floating point (fractional) values, because of these round off errors. 

Prism 5.03 and 5.0c will fix the bug by computing the two-tail P values for symmetrical tables differently than it computes asymmetrical tables. For symmetrical tables, Prism will compute the one-tail P value and double it (with special code to deal with a P value equal to 1.0).