poisson vs. ols for difference in differences with extremely small sample

[Philip Gigliotti]

Hi,

I'm trying to calculate a difference in differences estimate with a very small sample (10 clusters, 4 years of data for 40 observations total. I've tried 3 models.

 
reg Count treatment treatxpost i.Year
    
poisson Count treatment treatxpost i.Year
 

nbreg Count treatment treatxpost i.Year 

The ols estimate is an effect size of 3 times the poisson with p-value of .000. The count models show an effect size 3 times smaller with p = .09.

This doesn't make sense to me, because I can look at the data in excel and calculate the "difference-in-differences" by hand and it is equal to the ols estimate. Unfortunately I can't post a data sample because its sensitive, but it is essentially all 0's and 1's and a single count of a much higher number in the treatment group and post treatment period. (1 treatment cluster and 1 post treatment period). Should be simple to simulate.

I don't know what kind of correction the poisson data could make to have this effect. It seems the poisson model is no longer estimating a diff in diff. Any idea what is happening here? And how to push back if poisson is requested rather than ols since this is count data?

Edit: Ok, I forgot there is different interpretation for OLS vs. Poisson. Converting the estimate yields an equivalent effect size. Does anyone know why the standard errors are so much larger in the poisson model vs. the ols?

[Clyde Schechter]

I'm trying to calculate a difference in differences estimate with a very small sample (10 clusters, 4 years of data for 40 observations total.,,, it is essentially all 0's and 1's and a single count of a much higher number in the treatment group and post treatment period. (1 treatment cluster and 1 post treatment period).

To be honest, I would not take seriously the results of any model under these circumstances.

But to indulge in the technical question posed, from what you describe, it seems that the homoscedasticity assumption for -reg- is violated here. While I don't normally make too big a deal out of that provided the more important assumption of there actually being a linear relationship is met, if you are worried about the standard error and want to live or die by a p-value, then the homoscedasticity assumption is important. I suspect that if you go to robust standard errors things may look different.

Also, if your "clusters" represent a group of entities (people, firms, whatever they are) who are repeatedly observed over 4 years, all of these models fail by virtue of not accounting for that within-cluster sampling design and need the corresponding -xt- command used. (If different entities within the clusters are observed in different years, i.e. serial cross-sections rather than longitudinal follow-up, then ignore this paragraph.)

[Philip Gigliotti]

To be honest, I would not take seriously the results of any model under these circumstances.

But to indulge in the technical question posed, from what you describe, it seems that the homoscedasticity assumption for -reg- is violated here. While I don't normally make too big a deal out of that provided the more important assumption of there actually being a linear relationship is met, if you are worried about the standard error and want to live or die by a p-value, then the homoscedasticity assumption is important. I suspect that if you go to robust standard errors things may look different.

Also, if your "clusters" represent a group of entities (people, firms, whatever they are) who are repeatedly observed over 4 years, all of these models fail by virtue of not accounting for that within-cluster sampling design and need the corresponding -xt- command used. (If different entities within the clusters are observed in different years, i.e. serial cross-sections rather than longitudinal follow-up, then ignore this paragraph.)[/FONT][/COLOR][/LEFT]

Thanks. Yes, the paper is descriptive in nature, documenting a broader trend, and then showing this anomalous case as an example. The goal of calculating a treatment effect and p-value is just to be able to say whether this spike in the count for the treatment cluster is statistically significant.

The standard errors show some weird patterns which is why I'm confused.

For reg, OLS errors are very small and the result is highly significant, robust errors are larger (this makes sense) though still highly significant and cluster robust are the SMALLEST and smaller than even ols by an order of magnitude. This pattern makes sense, as cluster robust errors are known to be significantly biased downward with less than 42 clusters, and especially in diff in diff with small treatment groups.

The poisson errors don't make sense.

The uncorrected poisson are huge, and the result is barely significant. Robust are smaller than uncorrected by approximately 25%, and once more cluster are orders of magnitude smaller.

I can explain the clustered errors with my knowledge of the properties in this case, but I cant explain why poisson are so much larger than ols, and why uncorrected poisson are larger than robust poisson.

[Clyde Schechter]

Well, I can give you a handwaving argument that may or may not actually be correct. The Poisson distribution has a long tail, so that for any given true mean value, there is a relatively wide range of estimated means that have reasonably large likelihoods for a data set drawn from a Poisson model: the large tail doesn't result in ruling out a lot of potential estimates for the mean. By contrast, the tails of the normal distribution taper off much more rapidly, so the range of estimated means that retain a large likelihood is narrower, all else equal.

As you might imagine from other things I have posted in this Forum I would not be reporting a "statistical significance" verdict on this in any case. But on the assumption that you are required to do so, I think an honest report of these results is to present the simple description of the data as being mostly zeroes and ones, but a single large count in one entity in the post-treatment period (perhaps giving a two-by-two table of means to accompany the description), along with the results of all of these models and observe that in this case all of the models are wrong and none of them is useful. The only conclusion that I think can be safely drawn here is that more and better data are needed to answer the research question.