Bootstrapping in a difference-in-difference test using frequency weights

[Henrik Dalriksson]

Hi!

I am examining whether a policy change leads to higher return on assets (ROA) for a treatment group (TREAT=1) relative to a control group (TREAT=0) using a difference-in-difference test with frequency weights obtained from propensity score matching.

As a robustness test, we are interested in block bootstrapping standard errors. We have tried adding vce(bootstrap, varlist(FIRM)) to the regression but receive the error message below. Note that we are trying to bootstrap standard errors at the firm-level since it is there that we have the variation.

weights not allowed r(101);

This is the regression we are running

reg ROA TREAT POST TREAT*POST i.FIRM i.INDUSTRY*YEAR i.COUNTRY*YEAR [fweight=WEIGHT], vce(bootsrap, varlist(FIRM)

ROA: Endogenous variable
TREAT=1 if treatment firm that receive the treatment and =0 if control firm
POST equals 0 before policy adoption and 1 after
TREAT*POST is the difference-in-difference estimator
i.FIRM denotes firm fixed effects
i.INDUSTRY*YEAR denotes industry-year fixed effects
i.COUNTRY*YEAR denotes country-year fixed effects

Does anyone know how to solve this? I have seen other solutions when you have probability weights (pweights) but not for frequency weights.

Many thanks!

[Carlo Lazzaro]

Henrik:
as far as I can see it, your code shoud have been ill-pasted since, as its current face-value, it is illegal.
However, the main issue is that -bootstrap- standard errors are not allowed when -weights- are present.

[Bryan Sayer]

In addition to what Carlo says, frequency weights are when you actually observe the number of observations in the weight, for example in survival analysis you can have data in the form of events and trials, or have a contingency table for epidemiological analysis. I can't conceive of a situation where one would get frequency weights from a propensity score, as you don't actually observe the results - they are estimated.

[Henrik Dalriksson]

In addition to what Carlo says, frequency weights are when you actually observe the number of observations in the weight, for example in survival analysis you can have data in the form of events and trials, or have a contingency table for epidemiological analysis. I can't conceive of a situation where one would get frequency weights from a propensity score, as you don't actually observe the results - they are estimated.

I have used the frequency weights generated by psmatch2 and incorporated them into my regression. Is that ill–advised?

[Henrik Dalriksson]

Henrik:
as far as I can see it, your code shoud have been ill-pasted since, as its current face-value, it is illegal.
However, the main issue is that -bootstrap- standard errors are not allowed when -weights- are present.

Sorry for the typo!

Is there a way to incorporate weights? Or would it make sense to drop weights and use bootstrapping? The problem is that I use weights in my baseline regression and removing them and together with removing weights would imply changing two things at once. This might make it difficult to observe the actual effect from bootstrapping as opposed to the effect from dropping the weights.

[Carlo Lazzaro]

Henrik:
as per -help regress- you can choose among different standard errors that allow -weight-.
So, why bothering yourself with -bootstrap- (which is not feasible with -weight-)?

[Henrik Dalriksson]

Henrik:
as per -help regress- you can choose among different standard errors that allow -weight-.
So, why bothering yourself with -bootstrap- (which is not feasible with -weight-)?

It was meant to be used as a robustness check to other methods that I am already using (e.g. clustered standard errors). But maybe this does not make sense and I should refrain from doing this?

I used the code below but received this message: insufficient observations to compute bootstrap standard errors no results will be saved. I have ca 1400 observations, is it not good enough?

bootstrap, reps(100) seed(1): reg ROA TREAT POST*TREAT i.FIRM i.IY_FE i.CY_FE

i.IY_FE denotes industry-year fixed effects
i.IC_FE denotes country year FE

[Bryan Sayer]

I have used the frequency weights generated by psmatch2 and incorporated them into my regression. Is that ill–advised?

I don't know enough to say for sure, but I have a hard time imagining how frequency weights are appropriate.

[Carlo Lazzaro]

Henrik:
have you detected missing values in your dataset?

[Henrik Dalriksson]

Henrik:
have you detected missing values in your dataset?

Yes, there exist several missing values in my dataset which have not been droppped. Do I need to drop them in order for the boostrapped standard errors to work?

I noticed that if I remove my fixed effects the boostrapped standard errors worked, is there a way to overcome this problem?

[Carlo Lazzaro]

Henrik:
I suspect that the problem is related to both missing values and time-invariant predictors (i.IY_FE).
Again, if you want to test the robustness of your regression model (but the first question that I address to myself in those instances is: robusteness with respect to what? heteroskedsticity? so called outliers? else?), I think that the best way is to cluster your standard errors on -i.FIRM-.
Eventually, take a look at -fvvarlist- for creating interactions.

[Henrik Dalriksson]

I don't know enough to say for sure, but I have a hard time imagining how frequency weights are appropriate.

Would it be more appropriate to use pweights where i use the propensity scores as weights? Does this approximate the inverse-probability weighting method?