What is the relationship between sample size and Type I error?

[Andreas Esbjornsen]

Dear clever statistics people,

I am writing a paper using a RD-design on three different data sets. The first data set has five times as many observations as the other two. The first data set grants me results that are significant and confirms my hypothesis. The two other data set grants me results, which for the most are insignificant, but some of them are significant results that confirms the opposite of my hypothesis.

How do I interpret and discuss these results?

I am fairly convinced that the insignificant results in the smaller data sets are due to Type II error, because smaller sample sizes in general leads to larger standard errors that make it difficult to detect significant results (Kellstedt & Whitten 2013: 141). Especially when it comes to RD-design that require a lot of statistical power (Cook & Wong 2008; Deke & Dragoset 2012).

The significant, but opposite results boggles my mind, however. I am absolutely certain that they are not correct, since I study school reforms that raise the compulsory schooling age, but these results show that it lowers the attained education. Therefore I conclude that they must be Type I errors. I suspect that the small sample size can explain the Type I error, but I am unable to find any papers/books that confirm this relationship.

Can you help me?

Are my Type I errors due to small sample sizes?

What kind of literature is there on this issue?

Or is it something else? Perhaps to do with the RD-design in itself?

Thank you very much,
Andreas Esbjørnsen
Department of Political Science
University of Copenhagen

[Maarten Buis]

You set the rate of the Type I error by choosing the significance level. So the rate of Type I error is independent of the sample size.

The only caveat might be that some tests are based on asymptotic arguments, which in small sample sizes may not work. That might cause a disagreement between the chosen significance level and the Type I error rate. The direction of such an error depends on many factors. In short, this is unlikely to be an feasible strategy to try to explain your findings.

I would look at the design itself. Maybe your control isn't a good control. Maybe other things happened simultaneously. Maybe you had to look at such a special case to get to a quasi experimental situation, that they are just weird and won't react the way you would expect people from the general population to react.

[Andreas Esbjornsen]

Dear Maarten,

Thank you for your answer. Can you point me in direction of literature of some kind that clearly states that the rate of type I error is independent of the sample size?

Best regards,
Andreas

[Maarten Buis]

Any introductory statistics textbook that discusses statistical testing. It will state that the chosen level of significance is the type I error rate. The consequence of that is that the type I error rate is thus independent of sample size.

[Joao Santos Silva]

Dear Andreas Esbjornsen,

In addition to valuable advice provided by Maarten Buis, please note that there are good reasons to choose the level of significance as a function of the sample size. For example, for a simple t-test we can use as a critical value the square root of the log of the sample size; this is essentially what is done when we use the BIC to choose between nested models. Maybe this makes a difference in your case?

Best wishes,

Joao

[Maarten Buis]

Just to make sure. Joao Santos Silva suggestion is not in conflict to mine: You choose your rate of Type I error. By far the most common approach is to just use a fixed number like 5% or 10% or 1%. In that case the Type I error is independent of sample size. However, since you can choose the Type I error rate, you can also choose to make it dependent on the sample size. This is what Joao suggested.

Now since Joao's approach is rather non-standard, I would suspect that someone who used this approach would be aware of it, and a) would not ask the question Andreas asked and b) would have told us. So I don't think it can explain the pattern Andreas found.

Also an influential initiative within the statistical sciences is moving to completely abolish the concept of significance, and with that Type I and Type II errors. (See this special issue of the American Statistician: https://www.tandfonline.com/toc/utas20/73/sup1 ) The arguments are sound and I hope that this time they are successful.

[Joao Santos Silva]

Dear Maarten Buis,

Indeed, our suggestions are not in conflict at all; as you say, we fix the size of the test, but we can do it as a function of the sample size.

The rationale for doing that is simple. Typically we consider Type I error more serious than Type II error, so we choose a small value for the probability of Type I error and accept the corresponding probability of Type II error. However, for a fixed probability of Type I error, the probability of Type II error goes down with the sample size (the power goes up), so we may end up with a situation where the probability of Type II error is close to zero but the probability of the more serious Type I error is still fixed at, say, 5%. So, we may want to choose the probability of Type I error as a function of the sample size. The leading reference for this is (see page 114)

Edward E. Leamer (1978), Specification Searches. Ad Hoc Inference with Nonexperimental Data, John Wiley & Sons

but see also http://bit.ly/2agM1jB

Best wishes,

Joao