xtdcce2fast with only some pooled coefficients

[Justas Dainauskas]

Dear JanDitzen,

I’m working with a dynamic heterogeneous panel and a large unbalanced panel data set, where N=32,942 and T=74. After removing missing observations (or logs of 0), I am left with 800,000+ observations (NxT).

The good news is that xtdcce2fast runs and converges in ~25min. The bad news is that xtdcce2 crashes STATA without producing estimates, because the machine runs out of memory (Apple M1, 16GB RAM).

The problem is that the empirical model specification is motivated by theory, where only lagged coefficient values are heterogeneous, not contemporaneous. So ideally, I would want to estimate pooled contemporaneous coefficients and allow lagged coefficients to be heterogeneous. Unfortunately, xtdcce2fast seems to pool all or none of the coefficients and only xtdcce2 would in principle estimate a mixture of pooled and heterogeneous coefficients.

Is there any way to work around that?

Thanks,
Justas

[JanDitzen]

Hi Justas,

thanks for using xtdcce2.

On xtdcce2: correct, xtdcce2 is not able to handle such large datasets. In technical terms, it posts all results to matrices to be stored in e() which is the bottleneck given your data.
On xtdcce2fast: it is good to hear that xtdcce2fast is sufficiently fast. You might want to add the option nocd to avoid calculating the CD test statistic which is time consuming in large N panels. Unfortunately there is no way to estimate an estimation with homo- and heterogeneous coefficients at the same time. Doing so is relatively time consuming and I avoided this scenario on purpose when coding xtdcce2fast.

There is a workaround though: you can use factor variables to create individual coefficient and pool those. I think (!!!!) the SEs for the pooled coefficients should be correct. The SE for the heterogeneous coefficients can be estimated by using the sample variance of the individual coefficients. For example, say x1 are the pooled coefficient, x2 the heterogeneous coefficients.Then the following might work:

xtdcce2fast y x1 i.id#c.x2, cr(y x1 x2) p(_all) notable
*** Select coefficients for x2
mata b=  st_matrix("e(b)")[selectindex(regexm(st_matrixcolstripe("e(b)")[.,2],"x2"))]
*** MG estimate and SE
mata mean(b')
mata variance(b')/cols(b)

As I said, this should yield the same results, but I am not 100% sure.