How to inform readers the context of hierarchical logistic regression that removes significant main effects

[Michael Hojo]

Hello all,

I'm examining a hypothetical scenario to determine how living alone and the mechanism of feedback affects a person's willingness to turn themselves into police for a crime. Mechanism of feedback refers to the person being told the positive or negative consequences of turning themselves in (e.g. a positive mechanism would tell the person all of the good things that come with turning him or herself in while a negative mechanism would tell the person all the bad things that come with turning him or herself in - a neutral mechanism doesn't mention any good or bad things).

I'm running a hierarchical logistic regression model using two steps: the main effects on the first step and the interaction on the second step where v1 is the dichotomous variable "Lives alone" (yes/no) and v2 is the categorical variable "Feedback mechanism" with three categories (positive/negative/neutral)

nestreg, lr: logistic turnselfin (i.livesalone ib0.feedbackmech) (i.livesalone#ib0.feedbackmech)

Which spits out:

note: 0.livesalone omitted because of estimability.
note: 0.feedbackmech omitted because of estimability.
note: 0.livesalone#0.feedbackmech omitted because of estimability.
note: 0.livesalone#1.feedbackmech omitted because of estimability.
note: 0.livesalone#2.feedbackmech omitted because of estimability.
note: 1.livesalone#0.feedbackmech omitted because of estimability.

Block 1: 1.livesalone 1.feedbackmech 2.feedbackmech

Logistic regression                                     Number of obs =    308
                                                        LR chi2(3)    =  31.75
                                                        Prob > chi2   = 0.0000
Log likelihood = -186.56561                             Pseudo R2     = 0.0784

------------------------------------------------------------------------------
  turnselfin | Odds ratio   Std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
  livesalone |
Lives alone  |   1.935901    .499876     2.56   0.011     1.167055    3.211256
             |
feedbackmech |
   Positive  |   4.574841   1.510175     4.61   0.000      2.39547    8.736979
    Neutral  |   3.450492    1.14231     3.74   0.000     1.803369     6.60203
             |
       _cons |    .142871   .0454746    -6.11   0.000     .0765622    .2666085
------------------------------------------------------------------------------
Note: _cons estimates baseline odds.

Block 2: 1.livesalone#1.feedbackmech 1.livesalone#2.feedbackmech

Logistic regression                                     Number of obs =    308
                                                        LR chi2(5)    =  33.36
                                                        Prob > chi2   = 0.0000
Log likelihood = -185.76036                             Pseudo R2     = 0.0824

-----------------------------------------------------------------------------------------
             turnselfin | Odds ratio   Std. err.      z    P>|z|     [95% conf. interval]
------------------------+----------------------------------------------------------------
             livesalone |
           Lives alone  |    1.93617    1.10679     1.16   0.248     .6314858    5.936405
                        |
           feedbackmech |
              Positive  |   5.727273   3.287981     3.04   0.002     1.859002    17.64476
               Neutral  |   2.757576   1.597961     1.75   0.080     .8856718    8.585826
                        |
livesalone#feedbackmech |
  Lives alone#Positive  |   .6943834    .486771    -0.52   0.603     .1757507    2.743479
   Lives alone#Neutral  |   1.451546   1.028417     0.53   0.599     .3620396    5.819764
                        |
                  _cons |   .1428571   .0682988    -4.07   0.000     .0559693    .3646315
-----------------------------------------------------------------------------------------
Note: _cons estimates baseline odds.


  +----------------------------------------------------------------+
  | Block |        LL       LR     df  Pr > LR       AIC       BIC |
  |-------+--------------------------------------------------------|
  |     1 | -186.5656    31.75      3   0.0000  381.1312  396.0516 |
  |     2 | -185.7604     1.61      2   0.4470  383.5207  405.9013 |
  +----------------------------------------------------------------+

My question is how to interpret and explain this in lay terms. My two real struggles are:
1. Living alone loses its significance when controlling for the interaction between living alone and the mechanism of feedback.
2. Positive mechanism of feedback retains significance after the interaction, but the interactions themselves are not significant.

And I still struggle to understand exactly how to interpret interaction effects in this way. Is it correct to say that living alone does not significantly impact someone turning themselves in when they live alone and are presented with feedback (Block 2)? But, living alone without considering feedback does significantly increase turning themselves in (Block 1)?

Asking how to interpret and discuss these results for the lay person might be elementary, but I've looked everywhere and virtually all threads, videos, etc. interpret hierarchical regression with interactions in a statistical way. I suppose I'm asking how I would explain this result to my grandmother, as the saying goes.

Thanks for all the help.

[Clyde Schechter]

Before you try explaining them to a lay audience, you need to actually get a correct understanding yourself: at the moment you do not have that. The main thing is that the coefficient of living alone does not represent the same thing in the two models and they are not comparable to each other. (Same is true of the coefficients of the feedback mechanisms.)

When you run an interaction model you are stipulating that there is no such thing as the effect of any variable that is a constituent of an interaction. In this case, that means that in the interaction model, there is no such thing as the effect of living alone. Rather there are, in your case, 3 different effects of living alone, one corresponding to each of the three types of feedback mechanism. In the interaction model, the coefficient of living alone means the effect of living alone conditional on the feedback mechanism being in its reference category, which, in your model, is Negative feedback. By contrast, in your non-interaction model, the coefficient of living alone is an overall effect of living alone, independent of the feedback mechanism. Since the coefficients represent different things in the two models, they cannot be directly compared, and there is, in general, no reason to expect they will be the same, nor similar, nor of the same sign, nor have any important resemblance to each other.

I think that understanding the output of a logistic regression of any kind is tough for a lay audience: odds ratios are going to make them queasy. When you then try to explain interaction ratios of odds ratios they will be outright nauseated. So I wouldn't even attempt it, no matter how well I understood it myself. What I would do instead is re-run these models and follow each of them with a margins command showing the predicted probabilities of turning oneself in under the corresponding conditions:

logistic turnselfin i.livesalone ib0.feedbackmech
margins livesalone#feedbackmech

logistic turnselfin i.livesalone ib0.feedbackmech i.livesalone#ib0.feedbackmech
margins livesalone#feedbackmech

You will see that the predicted probabilities differ a bit between the models, although (see below) I expect they will not differ by very much. I think these -margins- results will be much more comprehensible to a lay audience.

That said, your interaction coefficients are small. And the comparison of the interaction model with the non-interaction model, whether by the LR test or by looking at AIC and BIC suggests that the interaction terms are not adding much information at all. This implies that the stipulation of three different effects of living alone is not really supported by your data: it seems that the three different effects are at least close to the same, if not exactly so. And I expect that the -margins- results I recommended above will reflect this. So, frankly, you might as well just skip the interaction model anyway in your presentation. If somebody sophisticated in the audience asks whether you looked at an interaction you can then say that you did and the results were not materially different.