Changing the instrument changes the Wald test of exogeniety results

[Parul Gupta]

I am running an -ivprobit- model with one endogenous regressor and one instrument. I am trying out two different instruments, let's call them A and B. If I instrument using A, I get the p-value for Wald's test of exogeniety as 0.04 (indicating that using -ivprobit- is appropriate due to the presence of endogeniety), but if I use instrument B, I get a p-value of 0.96 suggesting that there is no endogeniety.

This doesn't sound correct to me. I have the same endogenous variable--according to me, simply changing the instrument shouldn't change the Wald exogeniety test results.

Could someone explain how to interpret my results and if/where am I going wrong?

[Maarten Buis]

These tests are only as good as their instruments. In the extreme, with a bad instrument you cannot detect whether some variable is endogenous or not. So using different instruments can and should change the results of the test.The problem with these techniques is finding good instruments. In practice, I find it extremely rare to find a really convincing instrument. So almost always consider using these techniques, but in the end abandon it because there aren't any good instruments. I find you will do more harm with using bad instruments then by using other methods and clearly discuss their limitations.

[Parul Gupta]

Thank you for your response and suggestions. Would you suggest using -weakiv- to supplement the results?

[Joro Kolev]

Tests for weak instruments test what the name says, whether the instruments are weak.

In principle you cannot test whether the instruments are valid. But you can do an overidentification test using your two instruments, which will show you whether if you assume that one of them is valid, the other one is valid as well.

Thank you for your response and suggestions. Would you suggest using -weakiv- to supplement the results?

[Parul Gupta]

Tests for weak instruments test what the name says, whether the instruments are weak.

I meant to ask if the weak instruments test will give any additional information regarding the validity of the instruments. For example, is it sufficient to mention the Stock-Yogo cutoff of F-statistic for the first stage regression or should weakiv results be included?

[Maarten Buis]

Joro was clear about that:

In principle you cannot test whether the instruments are valid.

This is the big weakness of any technique that relies on instrumental variables: they are at least as much an act of faith as they are an act of science. That is of course a pretty damning statement, but consider what these methods try to do: They try to control for things you have not observed. That is, to put it mildly, pretty ambitious for an empirical scientist. So it should come as no surprise that you will have to "supplement" your empirical with some untestable assumptions.

[Parul Gupta]

they are at least as much an act of faith as they are an act of science.

Strong statement, indeed! So, there is no point reporting Wald/F/weakiv results? It is strange that examiners/reviewers still insist on these (I am from the economics field). Does this also mean one shouldn't use IV at all, as you hinted in #2?

[Maarten Buis]

A test tests a null hypothesis. If you are not interested in that null hypothesis, then that test is useless. If you are interested in that null hypothesis, then it is not useless. A weak IV test tests whether the instrumental variable is weak (sometimes, not often, but sometimes terminology in statistics makes actual sense). This is helpful information when interpreting your results. However, it does not tell you whether the instrumental variable is valid or not. That is just a different question.

You should use instrumental variables if you have a good instrumental variable. If you have a good instrumental variable, then this is a really cool and powerful technology. Problem is that in my discipline at least, good instrumental variables are rare as hen's teeth. Moreover, you cannot test within your data whether an instrumental variable is valid or not. This is why I called them an "act of faith", but there are cases where you have information outside the data that makes that instrument at least plausible. So I would not throw IV methods away completely, but I would not start a project expecting them to work. If it does work, then that is wonderful, but I would always have a plan B, and I would expect to land at that plan B in most cases. Where things go seriously wrong is when someone decides (s)he needs to use instrumental variables without a plan B and uses whatever happened to be available in her or his dataset.