How to obtain ML (Cox-Snell)R2 after using seemly unrelated regression (sureg)

[Jose Ferreras]

Dear forum,

I have a basic doubts of methodology for which many of you may know the answer.

I am using seemingly unrelated regression (sureg) to test the following model:
sureg breadthn2005 c.approp2005##c.approp2005 int_rd2005 grupo2005 size2005 inci12005 market_size2005

As an example of a paper that report results after applying sureg I am following the following article: Laursen, K., & Salter, A. J. (2014). The paradox of openness: Appropriability, external search and collaboration. Research policy, 43(5), 867-878.

In their table of results, they report ML (Cox-Snell) R2. However, when checking the information provided in the manual of Stata about the sureg sintax, I could not find information about which type of pseudo R2 is provided by the program after using sureg.

By searching in the stata forum I found that “fitstat” is a post-estimation command that computes a variety of measures of fit for many kinds of regression models. However, this command does not work after suing sureg so I cannot use it to obtain the ML (Cox-Snell) R2.

My question, is there any command I can use to obtain the ML (Cox-Snell) R2 after using sureg?

Also, I would like to ask, How can I interpret low values of ML (Cox-Snell) R2? Can it be interpreted as one would interpret an OLS R-squared?

Thank you very much in advance for the help .

[Andrew Musau]

sureg breadthn2005 c.approp2005##c.approp2005 int_rd2005 grupo2005 size2005 inci12005 market_size2005

This is a single regression, which you can estimate with regress. In this case, the OLS R-squared is a special case of the Cox-Snell R-squared.

$$\text{Cox- Snell }R^{2}= 1-\left(\frac{L_{0}}{L_{M}}\right)^{\frac{2}{n}} $$

where \(n\) is the sample size, \(L_{0}\) is the likelihood for the constant-only model, and \(L_{M}\) is the maximized likelihood for the model with regressors.

sysuse auto, clear
regress mpg
local l0=e(ll)
regress mpg weight trunk disp
local lm=e(ll)
scalar CSR2= 1- (`=exp(`l0')'/`=exp(`lm')')^(2/e(N))
di CSR2
assert round(CSR2, 5)==round(e(r2), 5)

Res.:

. regress mpg weight trunk disp

      Source |       SS           df       MS      Number of obs   =        74
-------------+----------------------------------   F(3, 70)        =     44.46
       Model |  1602.40017         3  534.133392   Prob > F        =    0.0000
    Residual |  841.059285        70  12.0151326   R-squared       =    0.6558
-------------+----------------------------------   Adj R-squared   =    0.6410
       Total |  2443.45946        73  33.4720474   Root MSE        =    3.4663

------------------------------------------------------------------------------
         mpg |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      weight |  -.0062226   .0012538    -4.96   0.000    -.0087233   -.0037219
       trunk |  -.0977386   .1281377    -0.76   0.448    -.3533011    .1578239
displacement |   .0054426    .009901     0.55   0.584    -.0143044    .0251896
       _cons |   40.35702   2.057341    19.62   0.000     36.25378    44.46026
------------------------------------------------------------------------------

.
. local lm=e(ll)

.
. scalar CSR2= 1- (`=exp(`l0')'/`=exp(`lm')')^(2/e(N))

.
. di CSR2
.65579159

.
. assert round(CSR2, 5)==round(e(r2), 5)

.

[Richard Williams]

The paper you refer to is at

https://www.sciencedirect.com/scienc...48733313001832

A couple of things:

• It says it uses suest, not sureg. Do you still want to model your analysis after it?
• It says "Model statistics are single equation statistics." Statistics, including R2, are then presented for each dependent variable. There isn't a single R2 that covers all the equations simultaneously.
• My guess, then, is that they ran a command (more specifically, a fractional response model), and followed it by fitstat (or else calculated it by hand). They then used suest to test hypotheses about equality of coefficients or whatever.

Since I don't have their data or their code, this is conjecture on my part. But, my guess is that the use of suest is separate from the calculation of the R2 statistics.

So, make sure this paper really is a good template for you. If it is, run fitstat after each model you estimate. But maybe write the authors to confirm that this is indeed what they did.

Incidentally, if an estimation command returns e(ll0), e(ll), and e(N), Cox-Snell is easy to calculate by hand. See

https://www3.nd.edu/~rwilliam/xsoc73994/L05.pdf

Specifically, see page 9A, panel 3A.

Again, I could be wrong. But this is my best guess based on my reading of the paper.

EDIT: Andrew shows you just how easy the calculation is.

[Richard Williams]

As Andrew points out, OLS R-squared is a special case of the Cox-Snell R-squared. If using OLS, you don't need to calculate anything, it is already done.

But the authors aren't using OLS; they are estimating fractional response models. That is why they are reporting a Pseudo R^2, and for some reason (not stated) they chose Cox Snell.

Given the date of the paper, my guess is they used the glm command (but they could have used fracreg if it was around back then).

If Jose isn't estimating fractional response models, that is another reason to reconsider whether this paper is a good template to model your own analysis after.

[Richard Williams]

As an aside -- as several sources show, including Appendix A of

https://www3.nd.edu/~rwilliam/xsoc73994/L05.pdf

several Pseudo R^2 measures have been created that are logical analogs of OLS R^2. People often ask which is the best, and the answer may be that none of them are all that great.

But, Cox-Snell may have the strongest claim to being the best pseudo R^2 -- because it isn't a logical analog to OLS R^2, it is the exact same formula as OLS R^2 -- although most people would have a hard time realizing that based on the way the formula is written.

[Jose Ferreras]

Dear Andrew and Richard:

Thank you very much for your quit reply and recommendation.

As I have a data set similar to the one that Lausen and Salter (2014) have in their paper (I have similar dependent variable and slightly similar independent variable), I am trying to replicate the analysis that they performed in table 4 (see table attached). That is the reason why I trying to use sureg for the analysis.

In my previous message I didn’t provide all the details of the analysis and the data set, sorry about that. Now I will provide you more information of the data set and my objectives. I will really appreciate if you can help me to see if the approach I am taking is the right one.

I have a sample with 2288 observations and the data set I have look like the following.

ident	breadthn2005	breadth_colln2005	approp2005	int_rd2005	grupo2005	size2005	inci12005	market_size2005	ind2005
6	0.3333333	0.3333333	0	0.0342286	0	3.091043	0	4	19
11	0.6666667	0.6666667	1	0.0558716	1	3.295837	0	3	43
20	0.6666667	0.6666667	0	0.0534132	1	4.174387	0	4	19
22	0.8333333	1	0	0.1314393	1	7.235619	0	4	28
25	0.3333333	0.5	1	0.0301993	1	5.062595	0	4	18
26	0.8333333	1	0	0.0387659	0	5.398163	0	4	19
27	0.8333333	1	1	0.0690034	1	5.955837	0	4	13
30	0.6666667	0.6666667	0	0.0248584	1	6.182085	0	4	15
31	1	1	0	0.0109029	0	4.189655	0	4	19
32	0.5	0.6666667	2	0.0172736	0	4.465908	0	4	29
35	0.6666667	1	0	0.0408332	0	5.521461	0	4	17
38	0.5	0.5	3	0.0247668	1	5.164786	1	4	36
39	0.3333333	0.3333333	2	0.1550166	0	3.871201	0	4	19
40	0.3333333	0.3333333	0	0.0296186	1	3.78419	0	4	11
43	0.1666667	0.8333333	0	0.0321804	1	4.382027	0	4	25
44	0.8333333	0.8333333	2	0.0396012	1	5.288267	0	3	19
45	0.3333333	0.6666667	0	0.0242549	0	3.610918	0	4	19
46	0.6666667	0.6666667	0	0.0108938	1	4.343805	0	4	21
47	0.6666667	0.6666667	1	0.0263197	0	3.610918	0	2	11
51	0.8333333	1	1	1.844079	0	3.526361	0	1	49
54	0.3333333	0.3333333	0	0.1345497	0	3.367296	0	2	24
57	0.5	0.5	3	0.0748345	1	3.135494	0	4	21
66	0.5	1	2	0.1113551	0	4.369448	0	4	21
68	0.5	0.6666667	1	0.0299287	1	4.828314	1	1	21

The table of output that I want to replicate from Laursen and Salter (2014) paper is the following:

As you see, in their table they report ML (Cox-Snell) R2. After using sureg, the output provides a pseudo R2 value but I not sure if that value corresponds to the ML (Cox-Snell) R2. I used the formula that Richard Williams suggested https://www3.nd.edu/~rwilliam/xsoc73994/L05.pdf to calculate it manually and the value I obtained looks the same to R2 squared provided in the ouput of sureg.

In my data set I used sureg to test the following models and I obtained the following table:
Model1: sureg breadthn2005 c.approp2005##c.approp2005 i.ind2005
Model2: sureg breadthn2005 c.approp2005##c.approp2005 int_rd2005 grupo2005 size2005 inci12005 market_size2005 i.ind2005
Model3: sureg breadth_colln2005 c.approp2005##c.approp2005 i.ind2005
Model4: sureg breadth_colln2005 c.approp2005##c.approp2005 int_rd2005 grupo2005 size2005 inci12005 market_size2005 i.ind2005

Table 4 (with my data set). Seemingly unrelated fractional response regressions: the relationship between appropriability strategy and openness.
	(1)	(2)	(3)	(4)
	breadthn2005	breadthn2005	breadth_colln2005	breadth_colln2005
main
approp2005	0.0671***	0.0611***	0.0834***	0.0742***
	(0.0167)	(0.0167)	(0.0166)	(0.0165)
approp2005 # approp2005	-0.0138*	-0.0128*	-0.0188**	-0.0172**
	(0.00661)	(0.00658)	(0.00655)	(0.00648)
int_rd2005		0.000934		0.000588
		(0.000982)		(0.000967)
2005 grupo		0.0105		0.0250*
		(0.0117)		(0.0115)
size2005		0.0110**		0.0189***
		(0.00426)		(0.00420)
2005 inci1		0.0585		0.0455
		(0.0382)		(0.0376)
market_size2005		0.0127*		0.0171**
		(0.00539)		(0.00530)
Constant	0.501***	0.409***	0.724***	0.578***
	(0.00804)	(0.0243)	(0.00798)	(0.0240)
Industry effects (18)	Yes	Yes	Yes	Yes
Log likelihood	-62.58	-52.39	-44.70	-17.51
ML (Cox–Snell) R2:	0.03	0.04	0.04	0.06
Observations	2288	2288	2288	2288
Standard errors in parentheses + p < 0.10, * p < 0.05, ** p < 0.01, *** p < 0.001

[Jose Ferreras]

My doubt here is about how to interpretate low value of R2 squared in my table. As R2 cannot be interpreted in the same way as with OLS, can I still consider that my model have goodness-of-fit?

[Jose Ferreras]

The paper you refer to is at

https://www.sciencedirect.com/scienc...48733313001832

A couple of things:

• It says it uses suest, not sureg. Do you still want to model your analysis after it?
• It says "Model statistics are single equation statistics." Statistics, including R2, are then presented for each dependent variable. There isn't a single R2 that covers all the equations simultaneously.
• My guess, then, is that they ran a command (more specifically, a fractional response model), and followed it by fitstat (or else calculated it by hand). They then used suest to test hypotheses about equality of coefficients or whatever.

Since I don't have their data or their code, this is conjecture on my part. But, my guess is that the use of suest is separate from the calculation of the R2 statistics.

So, make sure this paper really is a good template for you. If it is, run fitstat after each model you estimate. But maybe write the authors to confirm that this is indeed what they did.

Incidentally, if an estimation command returns e(ll0), e(ll), and e(N), Cox-Snell is easy to calculate by hand. See

https://www3.nd.edu/~rwilliam/xsoc73994/L05.pdf

Specifically, see page 9A, panel 3A.

Again, I could be wrong. But this is my best guess based on my reading of the paper.

EDIT: Andrew shows you just how easy the calculation is.

As I understood from the paper, I though they used sureg (seemingly unrelated regression) for the outputs displayed in table 4 and then, they used suest as a post-estimations test to confirm the findings displayed in table 4. The title they introduced in table 4 "Seemingly unrelated fractional response regressions: the relationship between appropriability strategy and openness" makes me think that they use sureg, but maybe I am wrong. Is it possible to test Seemingly unrelated fractional response regressions using the command fracreg? or do I need to use another one?

As I new using stata and performing this kind of analysis sometime I get confused, sorry.

[Andrew Musau]

Just because you use the sureg command does not mean that you are running a fractional response regression. In #2, I hinted that you should fit the same model with regress to convince yourself that the sureg estimates are the OLS estimates. Richard gave you some hints on estimators in #4. You can also read more on fractional response models here.

[Richard Williams]

As Andrew pointed out, your sureg commands are basically regress commands. When there is only one equation, the differences between sureg and reg are super-trivial. You can see that with this code:

webuse nhanes2f, clear
sureg health height weight i.female
reg health height weight i.female
fitstat

As the code further shows, fitstat does work after reg even though it does not work after sureg. Further, after regress, the Cox-Snell R^2 is identical to the OLS R^2.

I don't even understand why sureg is in the discussion here. It wasn't in the paper you are citing. Single-equation sureg's are kind of pointless; just use regress instead. And if you want to do fractional response regressions, you should be using fracreg or glm, not sureg.

Also, I think the R^2 reported by sureg is the same as the R^2 reported by regress. The manual might be clearer on this.

In short, I don't think you are cloning the article very well. Reread the article and reread these comments. Instead of running sureg, I think you want to run fracreg and suest. And if fracreg isn't right, my guess is that regress is, not sureg.

[Jose Ferreras]

Thank you very much Andrew and Richard for your comments and advice. Following your recommendations I used fracreg and suest to test my hypothesis. I have only one more doubt, when using fracreg, is it ok to have models with value of ML (Cox-Snell) R2 lower than 0.1? Does low ML (Cox-Snell) R2 values have a meaning?

As fare as I know, in OLS a thumb up rule is to have model with a R2 higher than 0.1 as that ensure that you model better explain the variability of the depend variable. I am not sure if that rule apply to fractional response regressions models.

[Richard Williams]

First off, there is no law that says you have to use Cox/Snell. I usually use the McFadden's pseudo R^2 reported by Stata. I don't know which, if either, tends to report higher R^2 values.

With any model, the goal should be to specify the model correctly, not get the biggest R^2 value possible. It may just be there is a huge random component with what happens in the world.

But of course, there is always the possibility that the model is not specified correctly. Have you left out some variables? Should you have an X^2 term, or maybe be using log of X instead of X? Could/should X be better measured than it is? These are always things you should be thinking about. A low R^2 doesn't mean the model is wrong, but it does make you more likely to suspect it is wrong -- and it may also make your readers and reviewers more suspicious that something is lacking in your model and/or data.

So, specify the best model you can. If you succeed in doing that, the R^2 will be as high as it could be.

[Kayode Olaide]

I'm using the CCE estimation technique for a research work. My dataset consists of eight cross-sectional units (N) and 24 time (T) dimensions in each cross-sectional unit, and I'm using Stata for my estimation. The cross-sectional dependence test shows that the panels are cross-sectionally dependent. Also, the variables have mixed order of integration (stationarity), i.e., I(0) and I(1). Both the CCEMG and CCEPMG don't seem to be quite appropriate for my dataset. Please, I need help finding a suitable estimation technique, and will really appreciate suggestions. Thank you.

[Richard Williams]

Kayode, I suggest you start a new thread, with a descriptive title. Placing an unrelated post at the bottom of another thread often means your question will be overlooked by people who might know the answer.