Direct effect switches sign when moderator is added - what does this mean?

[Jana Schue]

Hello!

I have examined a panel and performed various analyses in STATA, whose results you can see below. I am very confused when it comes to the interpretation. If you only look at the total direct effects model (Models 6), the direct effect of Y3 on X is (slightly) positively significant (ß = 0.782, p < 0.1). However, if you then look at the table displaying the addition of the moderator, to be concrete on model 11, the sign of the direct effect changes (ß=-1.381, p<0.05). What does this change in sign tell me? I have already done a lot of research, but never found anything that helps me interpret this.


Evaluation of direct effects:

Model	(1)	(2)	(3)	(4)	(5)	(6)
Dependent Variable: X	Controls only	Direct effect Y1	Direct effect Y2	Direct effect Y3	Direct effect Y4	Total direct effects
Independent Variables
Y1		-5.381***				-6.457***
		(1.596)				(1.704)
Y2			-2.271***			-2.367***
			(0.391)			(0.397)
Y3				0.344		0.782*
				(0.412)		(0.427)
Y4					2.990***	2.218***
					(0.669)	(0.687)

Addition of moderator

Models	(7)	(8)	(9)	(10)	(11)
Dependent Variable: X	Interaction effect Y1	Interaction effect Y2	Interaction effect Y3	Interaction effect Y4	Total interaction effects
Independent Variables
Y1	-9.447***	-8.834***	-9.673***	-9.720***	-8.684***
	(2.814)	(1.756)	(1.756)	(1.778)	(2.936)
Y2	-1.993***	-0.986*	-2.460***	-2.085***	-1.051*
	(0.421)	(0.548)	(0.411)	(0.408)	(0.575)
Y3	0.745* (0.435)	0.768* (0.431)	-1.727*** (0.627)	0.968** (0.438)	-1.381** (0.656)
Y4	1.700** (0.700)	1.997*** (0.707)	1.650** (0.703)	-0.429 (1.022)	-1.029 (1.047)
Moderator
Z	-19.777** (9.234)	-15.004*** (2.597)	-63.774*** (8.538)	-37.686*** (6.798)	-82.127*** (15.035)
Interaction Effects
Y1 x Z	5.796 (87.087)				-31.517 (88.761)
Y2 x Z		-31.345*** (11.536)			-48.128*** (12.954)
Y3 x Z			101.427*** (18.607)		101.955*** (19.109)
Y4 x Z				70.127*** (24.330)	106.046*** (26.560)

Thank you very much in advance!

All the best,
Jana

[Mike Lacy]

Your table doesn't come out particularly readable on my screen, so I won't comment on it, but here's a general principle that should be useful: If the direct and total effect of some variable X differ in sign, then X must have an indirect effect in a different direction than its direct effect. (Think: Total = Direct + Indirect) In some literatures, the moderator would be called a "distorter" or "suppressor" variable, so you might search on that term. Drawing a causal model diagram would also likely help.

[Clyde Schechter]

Comparing the coefficient of Y3 in the two models is misleading because they do not represent the same thing.

In the first model (without the moderation) that coefficient is an estimate of the "average" effect of Y3 on your outcome variable. It is, in a sense, averaged across all of the unobserved (in this model) values of the moderator. I use quotes around "average" because it is not actually calculated as an average value, but the point is that it is a "one size fits all" estimate of the effect of Y3. If the moderation model is a valid description of the world, then this one size fits all estimate is, at best, a second rate estimate of the effect of Y3, something one might fall back on in a situation where the moderator variable is not available for use.

By contrast, in the second model, you are, in effect, stipulating that there is no such thing as a single effect of Y3. Rather in this model, you are stipulating that the effect of Y3 varies as a function of the value of the moderator. In the output, what is called the coefficient of Y3 is actually the effect of Y3 when (and only when) the moderator = 0. As such the coefficient of Y3 may or may not be of any use or interest. If 0 is a value that the moderator actually takes on in real life with some appreciable frequency, then it would probably be interesting to think and talk about, as well as looking at the effect of Y3 when the moderator takes on other realistic values. And in this case, if the difference between the 0.782 of the non-moderated model and the -1.381 of the moderated model is large enough to be of practical importance in the real world, then you might conclude that 0, even if it is a possible and realistic value of the moderator, it is not very typical. Whether such a conclusion is of interest to you or anybody else would depend on what we're talking about in the real world and what your research goals are.

But if the range of possible values of the moderator excludes 0, then the value of that coefficient has no real-world meaning at all. It is, if you will, a statistical fantasy about what might be seen in some hypothetical but non-existent situation. The correct interpretation of the moderation model respects the fact that the effect of Y3 is a function of the moderator, specifically given by the equation effect of Y3 conditional on a value of Z = -1.381 + Z*101.955. You should actually evaluate this for a range of realistic values of the moderating variable Z, and perhaps make a graph of them to gain an appreciation for how Z and Y3 jointly affect your outcome variable.

Crossed with #2. The table showed up fairly readably on my screen, and I do not think O.P. is dealing with direct and indirect effects of a simple mediator variable, though she does use that language. It appears to me, though I could be misinterpreting it, that she is looking at a series of explanatory variables Y1 through Y4 and has estimated their separate effects in the absence of the moderator variable in models (2)-(5), and in model (6) she has run a model that includes all of them. Similar considerations apply to models (7)-(10), where there is also effect modification by a moderating variable Z.

[Mike Lacy]

Clyde is right here--I seized on the language of "direct effects" and gave insufficient attention to the comments about a "moderator."