Interpretation of Prob> F, VERY small R2 & yet significant?

[Eva Lan]

Hello everybody

I did measure the DiD estimation via: reg gl fa ## ki, r

The output is:
(1)
Number of obs = 152
F( 3, 34861) = 2.03
Prob > F = 0.1073
R-squared = 0.0002
Root MSE = 20.687

fa1#ki
1 1 P>|t| = 0,032; Coef=0,8; SD=0,4

(2)
Number of obs = 148
F( 3, 52579) = 3.70
Prob > F = 0.0112
R-squared = 0.0002
Root MSE = 20.005

fa2#ki
1 1 P>|t| = 0,031; Coef=0,9; SD=0,4

Now my questions would be:
1. Do I have to discard the model (1) because Prob> F is not significant? Shouldn't the Prob>F value only say whether R2 as such is valid or not?
2. BOTH models have a very very small R2. How do I handle this? (2) is significant, but only predicts gl in about 0.02%? Would I have to stop further calculations at this point? Or shall I go on due to its significance?
3. While R2 is very small, Root MSE is very large. How do I deal with this relationship?
4. Moreover, regarding the reporting of my data: Can I use p instead of P> | t | ? (e.g. p = 0.031 instead of P> | t | = 0.031)

To me my outcomes seem to be a bit contradictory. And I am not that sure about how to further proceed.
Up to now, I mainly use the Princeton pdf-slides to interpret my data: http://www.princeton.edu/~otorres/DID101.pdf & https://www.princeton.edu/~otorres/Panel101.pdf.
It would be great if someone knows how to handle the data and might help me.

Thanks in advance!

[Carlo Lazzaro]

Eva:
1 and 2 ) the low R-sq tells you that with such a large sample, the scant number of your predictors fails to give a fair and true view of the data generating process (that is, you probably have omitted variabale bias, that makes your results unreliable); hence, rely on the statistical significance totem, is not (as usual) the way to go. As an side, the F-test tells you whether, taken jointly, your coefficents are different from zero. If the F-test turns out to be non significant, your regression is not, in fact, different from a null model (that is, the mean of the dependent variable). This makes sense, when we consider that the regression framework focuses on investigating the contribution to variation of the conditional mean of the dependent variable caused by a given predictor when adjusted for the remaining ones;
3) the large RMSE is simply the reverse face of the low R-sq coin: if you look at the Residual/MS cell of the anova-like table produced by -regress- you will find out that the figure is (20.687)^2=427.95 (by the way: as per FAQ, please report what you typed and what Stata gave you back via CODE delimiters; if you folow thia approach, interested listers wiil not be forced to produce potentially unhelpful guess-works about what's the matter with you data. Thansk);
4) the issue does not concern reporting p or P, but the fact that P>|t| refers to two-tail test. Hence the question is: what are you interested in?

[Eva Lan]

Thank you very much for your response!
1+2) Okay, so there is no effect for (1). That's alright. But for (2) I should neither "rely on the statistical significance totem"? What do you mean with that, please?
4) I know that my output contains two-tailed values. If I would - nevertheless - use a directional hypothesis I had to divide stata's P>|t| in half. I was looking for the right wordings for my reporting. I am only used to report values of significances as "p".

[Carlo Lazzaro]

Eva:
1) it is possible (but without more details is difficult to give more than a guess) that your model is poorly specified;
2) the main meaning of my previous point is that you may well have a a significant F-test but this does not automatically imples that you have a well specified model (you may have omitted variable and heteroskedasticity biases)
4) Ok for p vs P.