multiple treatment effects using regression

[Mary John]

Hello everyone,

I'm trying to find the effects of certain treatments from non-randomized data. Because I want the model to be as easy as possible, I'm using normal regressions and controlling for confounders, a method I read about in "Causal inference using regression on the treatment variable" by Andrew Gelman and Jennifer Hill. In my case, there are multiple treatment options of which a person can take one, multiple or none. I would like to calculate the effects of each treatment on certain outcomes. I have found a lot of literature on how to compute multiple treatment effects using the propensity score method, but I would want to stick to the simplest model possible.

I'm not sure on where to go from this point. Do I run multiple regressions, regressing the outcome seperatly on each service and the confounders, and use that coefficient? But then that is probably confounded because I did not include the other treatments which probably influece both the outcome and the decision of getting the treatment I included. Do I include all of the treatments in my regression and then interpret each coefficient as the average treatment effect for each treatment? To me this sounds more convincing than the first option, but I'm still not sure whether I'm capturing all relevant aspects.

It would be amazing if someone could help me out,

Mary

[Clyde Schechter]

This is an extremely complicated question, and I think that there is, at the very least, a tension, if not an outright contradiction, between your goals of simplicity and validity.

Every model makes compromises between feasibility of implementation and verisimilitude to the real world data generating process. Adding simplicity of interpretation/explanation to others to the mix may entail still further compromises.

Your problem is made even more complicated by the presence of multiple treatments. You have already sensed that simply studying each treatment in isolation is unlikely to give an accurate picture of treatment effects. But you have even underestimated that problem. It is possible that the effect of treatment A differs according to whether or not treatment B is also used. Worse still, it might differ according to whether treatment B is used before or after! You are looking at a potential combinatorial explosion that will quickly outrun the available data, or even your ability to get new data.

If you do not have good and deep understanding of the content area in which these treatments operate and what is known or generally believed about how they work, then the first thing you need to do is get a collaborator who does. Otherwise you are likely to end up with a model that has little contact with reality, no credibility to people in the discipline, and fits your available data set well and never fits another data set in the future! A good knowledge of the underlying science will enable you to determine which complications of the type described in the last paragraph need to be taken into account, and which can be ignored as being either implausible or known to be negligible. (Negligible is always to be understood relative to the precision needed for the results to be useful.)

You also need to think about what the target audience for your work is, and what it is they want to learn from your work and how they will use it. To take a silly example, age-adjusted mortality from breast cancer in the final quarter of the 20th century can be well fit by a simple quadratic function of calendar year. Such a model obviously takes no account of relevant underlyling developments such as the advent and dissemination of mammographic screening and the advent and dissemination of adjuvant hormonal and chemotherapy. It probably will not surprise you to learn that the quadratic model breaks down once we hit the 21st century as yet more changes in screening and treatment supervene. But if the audience had no need of a model that projects the future well and needed only some rules of thumb to estimate what the mortality was during that historical era, the quadratic model would be fine for their purposes. At the other end, even models that do a reasonably good job of representing the evolution of screening and treatment and their changing impact on breast cancer mortality have their limitations. Breast cancer is a heterogeneous disease, with tumors having different biochemical and genetic properties that influence their underlying prognosis and their response to different treatments. A model that does not take these into account may well give good estimates of the -marginal- mortality rate, but would not be satisfactory for oncologists seeking to assess individualized treatment protocols.

So a model is designed for a purpose, and the level of complexity, and ability to work well outside of its development sample, need to be tailored to that purpose. Returning to your question, "relevant aspects" must be interpreted relative to your project's goals, which, in turn, need to be tuned to the purposes of your project's target audience. A model that faithfully reflected the real world data generating process would, in principle, produce results that everybody could use. But one is rarely in a position to actually construct such a model outside of toy cases and classroom exercises. Moreover, implementing a model with that level of complexity without error is typically a daunting task. And if you finally constructed it, the calculations could be so complex that the results might not be available in your lifetime.

I imagine you were looking for more concrete guidance, and guidance with a more optimistic tone. As for optimistic tone, notwithstanding everything I have said, it is possible to build models complex enough to be useful but simple enough to be workable and there are many examples in the literature of nearly every discipline. This kind of work has been a significant part of my own career. So these points are made not to discourage you but to provoke you to think explicitly about the tradeoffs involved. These questions do have answers, but not simple answers.

George Box is well known for having said much of the above more succinctly:

Now it would be very remarkable if any system existing in the real world could be exactly represented by any simple model. However, cunningly chosen parsimonious models often do provide remarkably useful approximations. For example, the law PV = RT relating pressure P, volume V and temperature T of an "ideal" gas via a constant R is not exactly true for any real gas, but it frequently provides a useful approximation and furthermore its structure is informative since it springs from a physical view of the behavior of gas molecules.

For such a model there is no need to ask the question "Is the model true?". If "truth" is to be the "whole truth" the answer must be "No". The only question of interest is "Is the model illuminating and useful?".

What I hope I have added to that is that the term "illuminating and useful" itself must be interpreted relative to specific goals and intended uses.

[Mary John]

First of all thank you so much for your in depth answer Clyde. It did in no way discourage me, as I am only getting started in this area of research, so there is a lot to learn. I came across this data set some time ago and after reading about treatment effects a bit I wanted to go back to it and think about how I would implement the things I read. So in this case, the target audience is mainly myself at the moment and I am "just" trying to model the effects specifically in that sample. Nonetheless, I am trying to get this right.

As I am using published data of a big study I have no way of getting new information on the order of the treatments recieved. Generating new variables indicating if a person recieved multiple treatments is possible though. So would you agree that, with no further information to go off, running one regression of the output on every treatment, every combination of treatments and all confounding variables is "better" (but of course no where near perfect) than the two methods I originally described? Or is even this statement impossible to make? (I am assuming this is the part where I would need collaborator who can tell me about how these treatments are known to interact with each other, and where I can hope that he/she tells me the interactions are negligible - sadly I don't know anyone in that area).

I realize I am trying to push for, as you called it, "concrete guidance" and I am sorry for that as it probably hurts people who have significant knowledge in this area to answer these questions with a simple yes or no. In any case I want to thank you again for your great answer, I will surely get back to it in the future when I am thinking about model specification and feel like I'm loosing track of what I wanted to do.

[Clyde Schechter]

So would you agree that, with no further information to go off, running one regression of the output on every treatment, every combination of treatments and all confounding variables is "better" (but of course no where near perfect) than the two methods I originally described? Or is even this statement impossible to make? (I am assuming this is the part where I would need collaborator who can tell me about how these treatments are known to interact with each other, and where I can hope that he/she tells me the interactions are negligible - sadly I don't know anyone in that area).

Well, given that this is really a learning exercise for you, my reactions are:

1. I would actually do separate regressions on each treatment first. This is likely to be highly unrealistic, but I think you may find it enlightening to compare those results with what you get from more complicated models.

2. Whether I would consider a model with every possible combination of treatments would depend on the number of treatments. With 3 or 4, this could be workable (depending on the size of your data set and the noisiness of the outcome measures you are using). But with 5 you are talking about 2⁵ = 32 combinations, which is probably going to prove unwieldy. And with any more than that I would say that you would be left with a huge pile of results that would be impossible to summarize or understand well. (Moreover, unless your data set is immense, with that many treatment combinations, or an even higher power of two, the number of observations for each combination is likely going to be small and your estimates will be very noisy.)

3. A model in which all of the treatments is entered, but no additional modeling of combinations would probably be both feasible and, if not realistic, at least not laughable.

4. Yes, you really need a content expert to help you decide about what interactions are needed and which can be safely dismissed.

5. Again, if I knew the content, there might be other suggestions. Might the response be a function of the total number of treatments? (Again, a content expert would be helpful. Though, in this case, if your model in #3 showed similar coefficients for each treatment variable, that would be suggestive that replacing all of those with just a count of the number of treatments might be a good model of the data.)