Differences in levels between control and treated unit.

[Sumedha Gupta]

Dear All,
I am trying to do a difference in difference estimation. There is a big levels difference in the dependent variable between the treated and control unit, BUT they follow the same time trend prior to the policy change, the necessary identification assumption, the effect of which I am trying to estimate. But since I essentially have a bi-modal distribution can I interpret the regression results which Stata estimates? I believe Stata assumes a t-distribution? I am not sure how I should proceed:

1. The results can be interpreted in their current format?
2. Should I specify an alternate distribution that stata should use for regression
3. Should the control dependent variable be somehow scaled? I am specifying as aweights the number of individual's in treatment and control unit.
4. Even though in the output the test statistic is about 10 it is not statistically significant. So I think Stata is somehow using a different distribution already? Is that correct?

*OUTPUT*

* (1) total MME - bimodal
. eststo: reg MME did i.treated2 post i.week2 c.week2#c.treated2 /*jan feb mar apr may jun jul a
> ug sep oct nov*/ ///
> age026 age2640 age4055 age5565 age6575 malepat durc2 durc3 durc4 durc5 durc6 durc7 durc8 yr201
> 2 yr2013 ///
> hydrocodone oxycodone morphine hydromorphone methadone fentanyl oxymorphone buprenorphine [aweig
> ht=pats], cluster(treated2)
(sum of wgt is 1.5913e+07)
note: 100.week2 omitted because of collinearity
note: yr2012 omitted because of collinearity
note: yr2013 omitted because of collinearity
note: buprenorphine omitted because of collinearity

Linear regression Number of obs = 200
F(3, 1) = .
Prob > F = .
R-squared = 0.9854
Root MSE = 5.8e+06

(Std. Err. adjusted for 2 clusters in treated2)

Robust
MME Coef. Std. Err. t P>t [95% Conf. Interval]

did 2338347 223475.4 10.46 0.061 -501177.2 5177870
1.treated2 2.98e+08 2964874 100.63 0.006 2.61e+08 3.36e+08
post -8.44e+07 646630 -130.55 0.005 -9.26e+07 -7.62e+07

c.week2#c.treated2 -256570.4 18998.53 -13.50 0.047 -497969.6 -15171.27

age026 9.97e+08 6.08e+07 16.39 0.039 2.24e+08 1.77e+09
age2640 -2.56e+08 1.26e+07 -20.35 0.031 -4.16e+08 -9.63e+07
age4055 -6.31e+08 3.41e+07 -18.50 0.034 -1.06e+09 -1.98e+08
age5565 -6.37e+08 1.13e+07 -56.53 0.011 -7.81e+08 -4.94e+08
age6575 -1.17e+09 3.38e+07 -34.73 0.018 -1.60e+09 -7.45e+08
malepat -1.14e+09 4824606 -235.82 0.003 -1.20e+09 -1.08e+09
durc2 1.47e+08 3890101 37.86 0.017 9.78e+07 1.97e+08
durc3 1.65e+08 3724339 44.31 0.014 1.18e+08 2.12e+08
durc4 1.64e+08 1143211 143.17 0.004 1.49e+08 1.78e+08
durc5 1.80e+08 1421227 126.83 0.005 1.62e+08 1.98e+08
durc6 1.41e+08 945973 148.81 0.004 1.29e+08 1.53e+08
durc7 1.15e+08 325468.4 353.05 0.002 1.11e+08 1.19e+08
durc8 1.18e+08 100477.2 1169.53 0.001 1.16e+08 1.19e+08
yr2012 0 (omitted)
yr2013 0 (omitted)
hydrocodone 3.96e+07 4.62e+07 0.86 0.549 -5.48e+08 6.27e+08
oxycodone -1.16e+09 8539465 -135.45 0.005 -1.27e+09 -1.05e+09
morphine 1.12e+09 1.21e+08 9.29 0.068 -4.12e+08 2.65e+09
hydromorphone 1.61e+09 1.31e+08 12.25 0.052 -6.00e+07 3.28e+09
methadone 1.66e+09 8.13e+07 20.40 0.031 6.26e+08 2.69e+09
fentanyl -2.77e+09 1.09e+07 -252.84 0.003 -2.90e+09 -2.63e+09
oxymorphone 6.48e+08 7.48e+07 8.66 0.073 -3.02e+08 1.60e+09
buprenorphine 0 (omitted)
_cons 1.07e+09 1.52e+07 70.49 0.009 8.81e+08 1.27e+09


Apologies for not using an existing Stata data for this question. I couldn't find one that I could use to represent the problem in question. Many thanks in advance for any help.
Best,
Sumedha.

[Clyde Schechter]

1. The results can be interpreted in their current format?
2. Should I specify an alternate distribution that stata should use for regression
3. Should the control dependent variable be somehow scaled? I am specifying as aweights the number of individual's in treatment and control unit.
4. Even though in the output the test statistic is about 10 it is not statistically significant. So I think Stata is somehow using a different distribution already? Is that correct?

The existence of a level difference between your treatment and control groups is not a problem. So the answers to your four questions are:

1. Yes (but see answers to 3 and 4).
2. No..
3. No. As for weights, you don't explain why you are weighting the data at all. If the observations are at the individual level, it seems to me that no weights are needed, or even allowable unless you can give a good reason why they should be there. Be sure you understand what aweights are. If you assign, say, an aweight of 4 to a given observation, what you are saying is that the dependent variable in that observation was obtained by averaging 4 measurements on that same person. Is that the kind of data you have here? It doesn't sound like it.
4. No, Stata is using the t-statistic. But it is a t-statistic with only 1 degree of freedom because you used the cluster robust variance estimator and you only have two clusters. The use of the cluster robust variance estimator is only appropriate when there is a large number of clusters. While different experts will give you differing advice on how many clusters is large enough, I don't think anybody would tell you that two is sufficient. Drop the -cluster()- option. If you want to use -robust-, that's OK.

In he future, when posting Stata output, please put it between code delimiters (see FAQ #12 for instructions how) so that tables line up readably.

[Sumedha Gupta]

Thank you so much Prof. Schechter. You have helped me in the past as well and I really appreciate it.

Thank you for pointing out that I shouldn't use the cluster option. With regards your response '3' above, my raw data is individual level milligrams of narcotics dispensed. But the relevant question from a policy perspective is whether the total milligrams of narcotics dispensed decreased in response to a policy change in the treated group. So I aggregated the individual level data to construct total milligrams of narcotics dispensed each week. So 2 observations per week, one for the treated and the other for the control group. Then, I was trying to control for the difference in the population size of the treated and the control groups. Initially I didn't think this was necessary, but all the related literature I am reading presents results 'weighted by the relevant population size'. Hence, my query.

This then brings me to a deeper question,
1. Should I not collapse the data to 2 clusters only and run the regression at the individual level? But then the results are not directly interpretable for the total population...
2. If I do aggregate, should I somehow account for the difference in population size and hence the volume difference between the control and treated group. If the control group is twice as big as the treated group, a small response to the policy is effectively a big one. But, then I am thinking that the diff-in-diff would account for that already. Sorry I am confused, by why everyone is weighting their regressions by the population size of the cluster.

I will really appreciate your input. Thank you sooo much.

[Clyde Schechter]

Well, I think you need to be clear on what you're trying to analyze here. It appears that the treatment and control group can differ in two ways. The policy may affect the number of milligrams of narcotic that a patient will have dispensed. Or, the treatment and control groups can differ in the way they "attract" patients seeking the narcotic in the first place. Generally speaking, when decision makers implement policies that they hope will alter the dispensing of narcotics, the intent is that the dose will be decreased (or perhaps a non-narcotic substituted for the narcotic). If the policy, as a secondary effect, causes narcotic-seeking patients to avoid the facility and "take their business elsewhere" that will be reflected in a decrease in total milligrams of narcotic disepensed, but normally this effect is not something that the policymaker wants to "take credit" for. While it isn't normally the case, it can be the case that driving away narcotic-seeking patients is considered an intended and desired effect of the policy. So you need to be clear on this issue. The analysis would be different.

The ideal data set, in either case, is a sample of observations of all opportunities for dispensing narcotics (whether they end up being dispensed or not.) The outcome variable can be the milligrams dispensed (which may be zero), and the associated patient characteristics, times, etc. that serve as predictors of the outcome, as well as the treatment vs control group variable.

A. If the purpose of the policy is not to drive away narcotic seeking patients but only to modify the dispensing practices, then the appropriate analysis is not the total milligrams of narcotics dispensed in each group but, rather, the average milligrams dispensed per opportunity in each group. And you would carry out this analysis on individual opportunity-level transactions. No -collapse- would be involved and no weighting would be needed assuming that both the treatment and control groups were sampled in the same way.

B. If, however, it is intended to "take credit" for driving narcotic-seekers away, then the total milligrams dispensed is the appropriate outcome. You would aggregate (-collapse-) the data up to observations of each group at several time periods. In this situation, the sampling error of each observation's outcome would differ depending on the number of observations in the group during that time period. So unweighted linear regression would not be correct here. But -aweights- will not help you with a total: in fact they will make matters worse! In this situation, with the total as outcome, you need a different model such as Poisson or negative binomial.

Digging deeper, I would think that if the policy maker does want to "take credit" for driving away narcotic-seekers as well as for modifying dispensing practice, it would make sense to model those two effects separately. So the effect on dispensing would be modeled as described in A. The effect on number of narcotic-seeking patients treated would be dealt with by using the number of opportunities in each group in each time period as the outcome variable. Again, it is likely that a Poisson or other count-data model would be more appropriate than linear regression here.

[Sumedha Gupta]

Thank you for your detailed response Prof. Schechter. I agree that this is ideally a two stage model with an extensive margin (whether the patient is given narcotics or not) and the intensive margin (how many milligrams of narcotics a patient is prescribed). I have data problem though in accurately measuring the extensive margin. I only have data on dispensed opioids - so I do not observe the individuals who were not prescribed opioids in the data at all.

Then my question is:
1) Could I assume that the entire population of the treatment or the control group is the pool of individuals who could potentially use opioids? But even if that be the case, I cannot distinguish between individuals who were denied opioids due to policy change and those who recovered and did not need opioids anymore. I can though estimate a change in the (ratio of opioid patients/ total group population) weighted with [aweights=total population of group]. Would that make sense? Would a linear regression be appropriate for this or would it still have to be a count data like in your suggestion 'B' above?
2) For the intensive margin - I can estimate the impact of the policy on (average milligrams of narcotics per patient) constructed by collapsing (mean) at the treatment and control groups, this time [aweights=# narcotic patients in group]? Would this make sense? Or would your recommendation 'A' above be a better specification?

Sorry for going into such details... but as you pointed out there could be multiple effects going on and I am trying to estimate them all, with the limitation that my data is essentially truncated.
I am grateful for your advise and look forward to it. Thank you.
Sumedha.

[Clyde Schechter]

Well, this is complicated. Even on the intensive margin, if you have no patients who received 0 milligrams of narcotics, then you are biasing the intensive effect estimate as well. Ideally you would have data on patients who are "opportunities to prescribe a narcotic." Perhaps this would be defined as all patients meeting certain diagnostic and perhaps demographic criteria. Then you could estimate both the extensive and intensive effects.

But that is not what you have. It sounds to me somewhat like a Heckman selection problem. But I really don't know that well enough to advise you how to proceed. I hope somebody else will chime in here, as I feel I'm at a loss.