Using Wald Test for Complex Interaction Models

[Siyu Li]

Dear all:

My name is Siyu. I am running a random-effects logit model involving many interaction terms. The model syntax is as follows:
xtlogit author c.ideodist##c.constraint_sd##i.statCase c.ideodist##c.bills_lag##i.statCase c.ideodist##c.divma_lag##i.StatCase i.cj c.saj c.newexpmx effnatct i.median##c.xmargin c.xmargin##c.ideodist c.xmargin##i.cj c.early_salience_est##i.cj c.early_salience_est##c.ideodist i.legalimp##c.ideodist i.legalimp##i.cj i.legalimp##c.newexpmx c.xdays##c.ideodist c.xdays##i.cj c.xdays##c.effnatct,re



The model output suggests that c.ideodist##c.constraint_sd##i.statCase and c.ideodist##c.divma_lag##i.StatCase are statistically significant, which supports my hypotheses.

Should I trust the p-value associated with these three-way interactions? Or should I perform separate Wald-test as some reviewers suggest. If I do need to test separately, what would be the proper syntax? Existing Q&A here do not seem to provide a satisfactory answer. What is more complicated is that "ideodist" also appears in multiple other interaction terms. But I am only interested in if the three-way interactions are statistically significant.

Appreciate your comments and answers!
Best,
Siyu

[Marcos Almeida]

You didn’t share the output, hence it is not easy to guess what’s going on.

Please read the FAQ. There you’ll find advice about sharing data/command/output in this forum.

That said, and generally speaking, adding two many interactions at once won’t help much. Neither in terms of really improving the model, nor with regards to the interpretation.

[Maarten Buis]

p-values belong to a test which has a null-hypothesis, in this case that that parameter equals 0. In case of a three way interaction it is sometimes difficult to see what that parameter means. It is not impossible, and there are hypotheses where your substantive hypothesis corresponds exactly with that parameter. In those cases the p-value reported in the output is exactly right. You will have to put a lot of effort in explaining that to your audience. If you are in this situation, then the fact that the referee did not get it means that you need to put more effort in explaining.

Moreover, it is not always the case that the substantive hypothesis corresponds with that parameter, and in those cases the p-value is meaningless. If you are in this situation, then the referee is right.

[Siyu Li]

Thanks for your responses and I apologize for not posting the output, which is now posted below.
My hypothesis is that: constraint_sd and divma_lag moderates the effects of ideodist, and that such moderating effect is only present in statCase (=1). So presume the parameter is consistent with the null-hypothesis of the p-value?

Thanks!


Random-effects logistic regression

author Coef.	Std. Err. z P>z	[95% Conf.	Interval]
ideodist -.0120252	.0034112 -3.53 0.000	-.018711	-.0053394
constraint_sd .0293666	.0430397 0.68 0.495	-.0549896	.1137229
c.ideodist#c.constraint_sd -.0014854	.0019957 -0.74 0.457	-.0053969	.0024261
1.statCase -.2900958	.0950655 -3.05 0.002	-.4764207	-.1037709
statCase#c.ideodist
1 .017256	.0043674 3.95 0.000	.0086961	.0258159
statCase#c.constraint_sd
1 -.0729574	.0571386 -1.28 0.202	-.1849471	.0390322
statCase#c.ideodist#
c.constraint_sd
1 .0052202	.0027407 1.90 0.057	-.0001514	.0105919
ideodist 0	(omitted)
bills_lag .0003184	.0038563 0.08 0.934	-.0072399	.0078767
c.ideodist#c.bills_lag .0000276	.00015 0.18 0.854	-.0002663	.0003216
statCase#c.bills_lag
1 .0012857	.0050798 0.25 0.800	-.0086707	.011242
statCase#c.ideodist#c.bills_lag
1 -.0001274	.0002156 -0.59 0.555	-.00055	.0002953
ideodist 0	(omitted)
divma_lag .1383402	.1113899 1.24 0.214	-.07998	.3566604
c.ideodist#c.divma_lag -.0046826	.0053705 -0.87 0.383	-.0152085	.0058433
statCase#c.divma_lag
1 -.2574167	.1394274 -1.85 0.065	-.5306894	.015856
statCase#c.ideodist#c.divma_lag
1 .015998	.0072345 2.21 0.027	.0018186	.0301773
1.cj -.1851289	.069847 -2.65 0.008	-.3220265	-.0482312
saj -.0627061	.0221365 -2.83 0.005	-.1060928	-.0193194
newexpmx .0394201	.0220211 1.79 0.073	-.0037406	.0825807
effnatct -.0049276	.0012638 -3.90 0.000	-.0074046	-.0024506
1.median -.0875963	.0621897 -1.41 0.159	-.2094859	.0342932
xmargin -.1596961	.0243508 -6.56 0.000	-.2074228	-.1119694
median#c.xmargin
1 -.1278079	.0436082 -2.93 0.003	-.2132784	-.0423373
xmargin 0	(omitted)
ideodist 0	(omitted)
c.xmargin#c.ideodist -.0023633	.0010799 -2.19 0.029	-.0044799	-.0002466
xmargin 0	(omitted)
cj#c.xmargin
1 .2810142	.0442236 6.35 0.000	.1943376	.3676908
early_salience_est -.0122522	.0579596 -0.21 0.833	-.125851	.1013465
cj#c.early_salience_est
1 .1783628	.1016508 1.75 0.079	-.020869	.3775947
early_salience_est 0	(omitted)
ideodist 0	(omitted)
c.early_salience_est#c.ideodist -.001042	.0026201 -0.40 0.691	-.0061772	.0040933
1.legalimp -.167591	.1210958 -1.38 0.166	-.4049343	.0697524
ideodist 0	(omitted)
legalimp#c.ideodist
1 .0041547	.00491 0.85 0.397	-.0054688	.0137782
legalimp#cj
1 1 .6729527	.1941219 3.47 0.001	.2924807	1.053425
newexpmx 0	(omitted)
legalimp#c.newexpmx
1 .0302655	.0725486 0.42 0.677	-.1119271	.1724582
xdays -.0006613	.0008143 -0.81 0.417	-.0022573	.0009347
ideodist 0	(omitted)
c.xdays#c.ideodist -.0000413	.0000218 -1.89 0.058	-.0000841	1.44e-06
xdays 0	(omitted)
cj#c.xdays
1 .0014794	.0009589 1.54 0.123	-.0003999	.0033588
xdays 0	(omitted)
effnatct 0	(omitted)
c.xdays#c.effnatct .0000275	.0000184 1.50 0.135	-8.53e-06	.0000636
_cons -1.281659	.0912278 -14.05 0.000	-1.460463	-1.102856
/lnsig2u -14.22023	6.345034	-26.65627	-1.784188
sigma_u .0008168	.0025913	1.63e-06	.4097967
rho 2.03e-07	1.29e-06	8.06e-13	.0485665
LR test of rho=0: chibar2(01) = 0.00	Prob >= chibar2 =	1.000

[Maarten Buis]

The reviewer is right: you have 4 hypotheses

1. constraint_sd moderates ideodist when statCase = 1
2. constraint_sd does not moderate ideodist when statCase = 0
3. divma_lag moderates ideodist when statCase = 1
4. divma_lag does not moderate ideodist when statCase = 0

So you need to perform 4 tests

[Siyu Li]

Thanks Maarten for your prompt response. I do have two follow-up questions.
1. for each test, I just do "test c.constraint_sd#c.ideodist#1.statCase =0" ?Is this the correct syntax?
2. Does it matter that ideodist, the variable being moderated, also appears in a number of other interaction terms?

Thanks!
Siyu