Seemingly unrelated regression (sureg) & Testing for significant differences of a coefficient across more than two equations

[Michael Steiner]

I will use sureg (seemingly unrelated regression) to analyze my data. I need to test my coefficients across the equations. I know how to do this when comparing the coefficients across two equations, but I am not sure if I can use this approach when having more than two equations.

I am using Stata 15 for Windows.

To be more precise:
When having two equations, one can simply run sureg and then test for differences of an coefficient between two equations by:
Test [dv1]iv = [dv2]iv
or
Test [dv1]iv - [dv2]iv = 0


The resulting F-test shows whether there are differences in the coefficients between both equations.

My problem:
I have six equations … thus, I intend to run all possible combinations of tests … which results in 15 tests.

My question:
Do I need to correct the p-values for alpha inflation? Or … is there a better way to perform such comparisons?

Kind regards
Michael

[Joro Kolev]

I am not sure what is the problem of having six equations, and why would you intend to run all possible combinations of tests. With 6 equations, you do it just the same way as you do it with 2 equations. Here it is done for 3 equations:

. sysuse auto, clear
(1978 Automobile Data)

. sureg (price foreign weight length) (mpg foreign weight) (displ foreign weight)

Seemingly unrelated regression
--------------------------------------------------------------------------
Equation             Obs   Parms        RMSE    "R-sq"       chi2        P
--------------------------------------------------------------------------
price                 74       3    1967.769    0.5488      89.74   0.0000
mpg                   74       2    3.337283    0.6627     145.39   0.0000
displacement          74       2    39.60002    0.8115     318.62   0.0000
--------------------------------------------------------------------------

------------------------------------------------------------------------------
             |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
price        |
     foreign |    3575.26   621.7961     5.75   0.000     2356.562    4793.958
      weight |   5.691462   .9205043     6.18   0.000     3.887307    7.495618
      length |  -88.27114    31.4167    -2.81   0.005    -149.8467   -26.69554
       _cons |   4506.212   3588.044     1.26   0.209    -2526.225    11538.65
-------------+----------------------------------------------------------------
mpg          |
     foreign |  -1.650029   1.053958    -1.57   0.117    -3.715748    .4156902
      weight |  -.0065879   .0006241   -10.56   0.000     -.007811   -.0053647
       _cons |    41.6797   2.121197    19.65   0.000     37.52223    45.83717
-------------+----------------------------------------------------------------
displacement |
     foreign |   -25.6127   12.50621    -2.05   0.041    -50.12441   -1.100984
      weight |   .0967549   .0074051    13.07   0.000     .0822411    .1112686
       _cons |  -87.23548   25.17001    -3.47   0.001    -136.5678   -37.90317
------------------------------------------------------------------------------

. 
. 


. test [price]foreign=[mpg]foreign=[displacement]foreign

 ( 1)  [price]foreign - [mpg]foreign = 0
 ( 2)  [price]foreign - [displacement]foreign = 0

           chi2(  2) =   42.00
         Prob > chi2 =    0.0000

[Carlo Lazzaro]

Michael:
elaborating a bit on Joro's comprehensive reply, if you want to adjust p-values for multiple testing, you can consider:

test [price]foreign=[mpg]foreign=[displacement]foreign, mtest(bonferroni)

[Michael Steiner]

Dear Joro,

I am not sure what is the problem of having six equations,

well, if you repeat tests, you migh have a problem with inflated alphas.

Thanks Carlo! I will use some kind of adjustment for the p-values.

Again, thank you to both of you!

Kind regards
Michael

[Joro Kolev]

Michael, it appears to me that you are confusing two completely different testing situations.

Consider the system
Y1 = b*X + E1
Y2 = c*X + E2
Y3 = d*X + E3.

A) The linear hypothesis: Ho:b=c=d, is a single linear hypothesis. This is not repeated testing situation. You do not need to adjust anything in this testing situation, which corresponds to the syntax that I showed you.

this case A) is to be contrasted with the completely different testing situation, where you are testing 2 separate linear hypotheses

B)

Ho1: a = b
Ho2: b = c.

In other words, null Ho can be expressed logically as Ho is equivalent to the intersection of (Ho1 & Ho2). In this case if you test Ho1 and Ho2 separately, you need to adjust the significance level for the joint hypothesis. This corresponds to the syntax that Carlo showed you.

[Michael Steiner]

Dear Joro,

I am very sorry for the confusion. Indeed, my aim is not to test Ho:b=c=d and instead I would like to test H1 a=b; H2 a=c; H3 b=c (and so on)

I have 6 products that were developed systematically, i.e. they differ in certain product features. The assumption is that the factors that drive the evaluation for these products differ between the 6 alternatives. For example, I am interested in testing the factors that matter for product 1 against those that matter for product 2, 3, 4, 5, and 6. ... and so on.

Again, thank you very, very much for both of you! Your help is highly appreciated!

Michael

[Michael Steiner]

Dear Carlo, dear Joro, dear Stata experts,

again, many thanks for your help and advice.

I still struggle with the problem of defining the paired comparisons. The code you suggested is:

test [price]foreign=[mpg]foreign=[displacement]foreign, mtest(bonferroni)

This makes the following comparisons:
[price]foreign=[mpg]foreign
[price]foreign=[displacement]foreign

Unfortunately, one comparison is missing:
[mpg]foreign=[displacement]foreign

Do you know how to test all possible paired comparisons. I have tried to use the "accum" (accumulate) option but this does not seem work in conjunction with the mtest option., i.e. I have used asimilar code as shown below which resulted in an error

test [price]foreign=[mpg]foreign=[displacement]foreign, mtest(bonferroni)
test [mpg]foreign=[displacement]foreign, accum mtest(bonferroni)

As noted, I have six question and would like to test all possible combinations across these six equations.

Kind regards,
Michael

[Joro Kolev]

One of these comparisons is redundant/implied by the other two. This is why Stata shows you the results of only two tests.

If (A=B) & (B=C), then A=C.

[Michael Steiner]

Dear Joro,

no, that is unfortunately wrong.

For example;
imagine the coefficient for A is 1, for B is 6 and for C is 7. Then, based on these tests provided by the code above I might learn that A ≠ B and that A ≠ C ... but still I have no information on the relationship between B and C ... I do not know whether B=C.

The core idea is similar to the ANOVA post-hoc tests. I.e. you need to perform all potential paired-comparisons and then adjust for alpha inflation (e.g. based on Sidak, Bonferroni, etc.).

Kind regards,
Michael

[Joseph Coveney]

Have you considered doing the pairwise tests using constraints and likelihood ratio tests, loop through the pairs, constraining their coefficients to be equal?

Maybe even do an omnibus test that all of the six coefficients are jointly equal.

version 15.1

clear *

quietly sysuse auto

sureg (price foreign weight length) (mpg foreign weight) (displ foreign weight), noheader
estimates store Full

constraint define 1 [price]foreign = [mpg]foreign
constraint define 2 [price]foreign = [displacement]foreign
sureg (price foreign weight length) (mpg foreign weight) (displ foreign weight), noheader constraints(1/2)
lrtest Full

exit

[Michael Steiner]

Dear Joseph,

good idea! In my regression, I do not only have 3 IVs but - unfortunately - many more ... (I have my experimental variation resulting in 3 IV but then I also have many, many control variables).

Essentially, I am only interested in testing the differences in the coefficients due to my experimental variation. Would you agree that I use your approach only for the (from my perspective) relevant coefficients (the 3 IVs) and not for all coefficients in the model?

Alternatively, how about the following approach:
I could also test all pairs of combinations based on the stata command and then simply adjust the results by hand? I.e. I could simply calculate the correction for alpha Inflation (such as Sidak or Bonferroni) by hand...

Kind regards
Michael

[Joseph Coveney]

I am only interested in testing the differences in the coefficients due to my experimental variation. Would you agree that I use your approach only for the (from my perspective) relevant coefficients (the 3 IVs) and not for all coefficients in the model?

Yes.

I could also test all pairs of combinations based on the stata command and then simply adjust the results by hand? I.e. I could simply calculate the correction for alpha Inflation (such as Sidak or Bonferroni) by hand...

Also, yes.

[Michael Steiner]

Dear Joseph,

I am sorry for my late reply; I was at a conference ...

Thanks A LOT for your help!!!!!!

Best regards
Michael

[Joro Kolev]

Dear Joro,

no, that is unfortunately wrong.

For example;
imagine the coefficient for A is 1, for B is 6 and for C is 7. Then, based on these tests provided by the code above I might learn that A ≠ B and that A ≠ C ... but still I have no information on the relationship between B and C ... I do not know whether B=C.

The core idea is similar to the ANOVA post-hoc tests. I.e. you need to perform all potential paired-comparisons and then adjust for alpha inflation (e.g. based on Sidak, Bonferroni, etc.).

Kind regards,
Michael

But Michael, if what I am saying "One of these comparisons is redundant/implied by the other two. This is why Stata shows you the results of only two tests. If (A=B) & (B=C), then A=C."
is "unfortunately wrong" then you should inform Stata Corp that they have a bug in their test command...

There are three constraints, and when asked to test three of them, Stata is testing only two .

[Matteo Brambilla]

But Michael, if what I am saying "One of these comparisons is redundant/implied by the other two. This is why Stata shows you the results of only two tests. If (A=B) & (B=C), then A=C."
is "unfortunately wrong" then you should inform Stata Corp that they have a bug in their test command...

There are three constraints, and when asked to test three of them, Stata is testing only two .

Hi everyone! I am facing the same problem, and this post has been extremely helpful. However, does anybody know if Stata has corrected this bug?
I would like to test (pairwise) the coefficients of 5 independent variables in a seemingly unrelated regression with 4 dependent variables.