OR from multi-level fixed logistic regression model

[liam yao]

I have a dataset comparing approach 1 vs approach 2 for an measurement, the total events rate in both groups are 32% vs 27% (kindly of similar), but given the data are nested under each approach, I sued multi-level logistic regression to calculate the OR, see the OR in the below figure

however, the OR from multi-level fixed model is much higher than the standard OR using 2X2 table (OR: 2.9 vs 1.2).

I am trying to explain the rationale behind it.

Appreciate it if anyone who can provide some clues


thanks

[Clyde Schechter]

You do not show the code you used, so we can only imagine what the model you estimated might be and whether it is appropriate to your data or not. Moreover, while you express surprise at the results because the OR is higher than you get from a 2x2 table, but you don't show the results from the 2x2 table, so we cannot tell if your expectations are simply unwarranted or if there is a real discrepancy here. Please post back with more information: show the exact code you ran and the exact output you got from Stata by copy/pasting from the Results window or your log file into this Forum editor. Be sure to surround that by code delimiters so at aligns readably. If you are not familiar with code delimiters, please read Forum FAQ #12 for information.

That said, I am concerned when I read "the data are nested under each approach," I worry that you have misused multi-level modeling. If the approach variable was a level in your model, that is not appropriate. Nesting refers to circumstances whereby a common attribute (membership in a household, or residency in a country or something like that) is itself a hidden variable that is associated with the outcome variable apart from the manifest variables included in the fixed-effects component of the multi-level model. Treatment-like variables (which, from the name "approach," I'm guessing we're talking about) almost never should be handled in that way. They just belong in the fixed effects. So when you post back, please do explain what these variables are and what you meant about the data being nested under each approach.

[liam yao]

Thanks for your information


the study compared the discordant recommendations (strong recommendation based on low quality evidence) between the consensus and evidence-based guidelines, see the 2 x 2 table below

	Discordant recommendations
Approach	Yes	No	total
1. Consensus	92	200	292
2. Evidence	30	82	112
total	122	282	404

I can generate a OR by using the 2 x 2 table = 92/200*82/30=1.26

While because the recommendations (lower-level units) were nested in guidelines (higher-level units), I used multilevel mixed-effect logistic regression, in which, the responses were clustered within guideline panels.

See the codes I used in Stata below, the outputs showed the OR’ from the regression is 2.9 which is significant higher than the OR from the 2 x 2 table (2.9 vs 1.26)

code: melogit discordant_recommendations type_of_guideline || guideline_ID:,or

One reviewer commented there is clearly an important effect of the clustering in each guideline with regards to the impact on discordant recommendations. Because the rates of discordant recommendations without taking clustering into account are very similar, I should explore the potential reasons why the OR from clustering is much higher than the OR’ from the 2 x 2 table.

see what you think what might be the reasons leading to this important difference (OR 2.9 vs 1.3)



Thanks for advising

[Clyde Schechter]

Thanks for the clear explanation; I never would have imagined from #1 the information provided in #3.

First, look at the confidence interval around the odds ratio you are getting from melogit: 1.09 to 7.85 (to 2 decimal places). 1.26 falls within that. Admittedly it is near the lower end of that interval, but that suggests that we are not dealing with a drastic inconsistency here. To the extent there is any need to explain the difference, it will require looking at what goes on within each guideline group. The degree of variation across guideline groups is large, as shown by the 5.1 variance component at that level. The distribution of outcomes and approaches within each guideline likely differ--you can check that in your data. So, guideline becomes a confounding variable in the study. When you go to a multi-level model you are, in effect, including guideline as an additional variable in the model. Its effects are represented in a different way from just directly including guideline as a fixed effect, but conceptually you are still adjusting the data for it. There is no surprise that adding a confounding variable to a model can change the results--by any amount and in either direction. (This is sometimes called Simpson's Paradox or Lord's Paradox. If you are not familiar with that, there is an excellent Wikipedia page about it.) Anyway, operationally, to explain this, you would want to group the guidelines into subsets with high and low frequencies of discordant recommendations, and then contrast the frequencies in which the two approaches were used in those groups.

[liam yao]

that's very helpful, thanks