Can parametric and non-parametric tests lead to different results?

[Bill Bryant]

This may be a basic econometrics question but I cannot find an answer. I know under what conditions a parametric test is preferable, but shouldn't the two techniques still give at least somewhat similar results?

In my study I have 500 subjects who took part in a decision-making experiments where they choose to "give" or "not give" in a set of scenarios. Thus, my dependent variable is binary and I am interested to see whether in scenario B more people gave compared to scenario A.

In scenario A, 84.8% of people gave; in scenario B, 85.4% of people gave. Yet, a tab chi2 tells me that the difference is significant [Pearson chi2(1) = 54.3799; n = 500]. However, if I run a xtreg fixed effects the coefficient for scenario B is obviously not significant.

I am asked to report both results, but don't understand how to reconcile them, or explain why reporting both wouldn't make sense.

Thanks!

[Joseph Coveney]

Did you mean to type xtlogit , fe?

One tests for independence between scenarios. They are not independent.

The other tests whether there's a substantial shift in choice between scenarios. There isn't.

[Bill Bryant]

Thanks Joseph,

You are right, and I think I found the solution by using McNemar's test with ratio of paired proportions, which basically allows me to test a before/after scenario within the sample sample. The Stata code is:

tab give_A give_B
mmc give_A give_B

Indeed, this tests does give identical results to the econometric regression (whether it being xtlogit, fe or xtreg, fe).

Thanks you!

[Carlo Lazzaro]

Bill:
in substantive terms the difference between the two scenarios sounds as immaterial, though.
That said, the different outcome you got may depend on the fact that the Pearson test considers your observations as independent, whereas in fact they aren't (due to their panel structure).

[Bill Bryant]

Thanks Carlo,

Indeed, the difference is almost non-existent and the McNemar test confirms that. It's true that chi2 was a wrong test since the samples were not independent, but the doubt remains as to how the chi2 test was still showing a significant difference...

[Carlo Lazzaro]

Bill:
this might be due to your sample size.

[Joseph Coveney]

. . .the doubt remains as to how the chi2 test was still showing a significant difference...

It wasn't.

Again, it's a test of independence. The chi-square test was showing a significant association (rejection of the null hypothesis of independence).