Meta Analysis of Studies without a Control by using an External Control Group

[Lawson Faulkner]

I have a large dataset of around 100 studies totalling around 25,000 participants. About half of these studies have controls, the other half do not. I have already carried out the stats for the half with controls using a random-effects model for binary outcomes and a log odds ratio for the effects - forest plot etc all came out looking great.

However when carrying out the stats for the other half, I need to use a different analysis because there are no controls - but I'd quite like to pool the controls from the other studies that do have controls as a comparrsion group to each of the studies without controls.

My first question is: Can I pool the controls from these other studies and use them as a common comparrison group in the meta analysis against all these other studies?

My second question is: If so, how should I go about doing it - would I just do the same random-effects model for binary outcomes with the log-odds ratio as I already did with the control studies, but where their empty control groups are put the pooled one? or is there another way to do this? (i assume there is as the former option just feels wrong)

[Tiago Pereira]

Hi, Lawson. Welcome to the Statalist. Answers:

1) No.
2) You should conduct a meta-analysis including all studies that have a control group (and that's it).

The data from the non-controlled studies could be used as prior information in a more complex model, involving Bayesian inference, but this would be more exploratory and of uncertain validity. We would need to check the studies in more details and the assumptions.

[Bruce Weaver]

I agree with Tiago Pereira. Nevertheless, if I would have been the first to respond, I would have been very tempted to say, "You can do anything you want." But of course, I would have been thinking about the following excerpt from the preface to Statistics as Principled Argument.

For many students, statistics is an island, separated from other aspects of the research enterprise. Statistics is viewed as an unpleasant obligation, to be dismissed as rapidly as possible so that they can get on with the rest of their lives. Furthermore, it is very hard to deal with uncertainty, whether in life or in the little world of statistical inference. Many students try to avoid ambiguity by seizing upon tangible calculations, with stacks of computer output to add weight to their numbers. Students become rule-bound, thinking of statistical practice as a medical or religious regimen. They ask questions such as, "Am I allowed to analyze my data with this method?" in the querulous manner of a patient or parishioner anxious to avoid sickness or sin, and they seem to want a prescriptive answer, such as, "Run an analysis of variance according to the directions on the computer package, get lots of sleep, and call me in the morning."

For years, I always responded to students who asked, "Can I do this?" by saying something like, "You can do anything you want, but if you use method M you’ll be open to criticism Z. You can argue your case effectively, however, if you use procedure P and are lucky enough to get result R. If you don’t get Result R, then I’m afraid you’ll have to settle for a weaker claim."

Eventually, I began to appreciate an underlying implication of the way I found myself responding: namely, that the presentation of the inferences drawn from statistical analysis importantly involves rhetoric. When you do research, critics may quarrel with the interpretation of your results, and you had better be prepared with convincing counterarguments. (These critics may never in reality materialize, but the anticipation of criticism is fundamental to good research and data analysis. In fact, imagined encounters with antagonistic sharpsters should inform the design of your research in the first place.) There are analogous features between the claims of a statistical analyst and a case presented by a lawyer--the case may be persuasive or flimsy (even fishy), the style of inference may be loose or tight, prior conventions and rules of evidence may be invoked or flouted, and so on. (p. xii)

Abelson, R. P. (2012). Statistics as principled argument. Psychology Press.

[Lawson Faulkner]

Thank you both, the answers have been very helpful!