xtdcce2 too many groups

[Steven Archambault]

Hey Folks,

I am trying out xtdcce2 for the first time, with an unbalanced panel. I get the error "too many groups". What gives? I do have a few groups that are singletons, maybe it doesn't like that?

Thanks for any insight.

-Steve

[JanDitzen]

Hi Steven,
the problem is the dataset contains too many cross sectional groups. xtdcce2 is using currently Stata's "separate" command, which has a maximum of 1500 groups. I am working on a work around of this problem for the next version of xtdcce2. A beta version called xtdcce213 is available from my personal server:

net install xtdcce213 , from(http://www.ditzen.net/Stata)

Please let me know if there are any further problems.
Thanks!
Jan

[Nick Cox]

Interesting. I was the original author of separate but clearly it's an official command now.

The syntax (added after my input, I think, but no matter) is visible in the code but I don't know where the 1500 limit comes from.

Perhaps it's a way of saying "We guess this really isn't a good idea. You're asking for more than 1500 extra variables. Did you realise that? Do you really want them?"

So, in the case of xtdcce2 is there a limit on what makes sense?

[Joao Pancada]

Dear Ditzen,

I have a very similar problem using xtdcce2: "More variables (.) than observations (162410)."

My sample has N = 745, T = 218 but the panel is unbalanced. I am trying to calculate a pooled dynamic model with 12 independent variables but even with only one Stata shows the same error.

Please take a look at the code: xtdcce2 asvi log_size, crosssectional(log_size) fullsample pooledconstant pooled(log_size) cr_lags(1)

The beta version you mentioned cannot be found on either Stata or the website you mentioned.

Thank you, João

[JanDitzen]

Hi Joao,
which version of xtdcce2 do you use?

The current version including beta versions are available from Stata here:

net from http://www.ditzen.net/Stata/xtdcce2_beta

Please try to install the latest version, version 1.32.

About Nick's questions (apologies, I overlooked this post): As long as no iv options are used, xtdcce2 does not use separate any longer. Internally the estimations is done in mata and therefore no additional variables are created. If the iv options are used, then xtdcce2 relies and ivreg2 and therefore needs to create new variables. In terms of econometric theory, it important is that both dimensions increase with the same rate.

Hope this helps and please let me know if there are any further questions.

[Joao Pancada]

Hi Jan,

What a quick reply! I've installed the latest beta version and re-run the regression but now there is a new error showing up, as follows:

Regardless of that, as you pointed out and I now confirmed in Pesaran and Chudik (2015), T cannot be small relative to N. Therefore, I will try to use another dynamic model. However, I run into the same problem as I wrote in a previous thread. Can you please take a look at this thread I created? [here]

Thank you very much!

[JanDitzen]

Hi Joao, thanks for replying so fast. Unfortunately I cannot see the picture you uploaded, but would be interested in the error. Would you mind checking it please?

[Joao Pancada]

Sorry, my bad. I am new to Statalist. My thesis deadline is coming up, I've to be fast . Here it goes:

[JanDitzen]

Thanks for this. This is indeed an interesting error. Does it happen if you do not pool the model or leave cross sectional averages out.

The source of the error might be the calculation of the unit specific covariance matrices and that your Stata version runs out of memory. Did you restart your computer or Stata recently?
Also, I just uploaded another new beta version which includes the option "fast", which prevents the calculation of unit specific covariances. However statistics such as F, R2 etc. are not calculated as well. The version count is still 1.32, so you need to force the update. Feel free to give this a try.

This does not really help you with your underlying question. However important to not is, that there are fundamental differences between the Arellano Bond type and CCE estimators. The differences are in terms of the dimensions (small vs. large T), the assumptions on the coefficients (homogenous vs. heterogenous) and the error term (no cross sectional dependence vs. cross sectional dependence / multi factor error structure) to name a few.

[Joao Pancada]

Hi Jan,

I have restarted my computer, updated the ado and combined the options in all sorts of ways (incl. fast) but I still have an error. However, now the error is different

Regarding your last point, I understand CCE is different from Arellano Bond. However, neither of those models should be used with my sample because Arellano assumes small T and CCE requires both dimensions to increase at the same rate as you said. Therefore, I should use another model, but I don't know which one, as I wrote here.

[Nada Shokry]

Joao Pancada I know it's 1.5 years later but I really need to know which dynamic model did you end up using? I have the same problem you have: N>20,000 and T=142. N is relatively much larger than T and I have difficulties running the xtdcce2 commands as well. I'd really appreciate it if you reply.
Thank you

[JanDitzen]

Nada Shokry , what is the error message?
The problem is not that N>>T, the problem might be the size of N. A general problem with "very large" datasets is the maximum size of a matrix in Stata. For MP and SE it is 11000 and for IC 800 rows or columns. If you have a dataset with more than 20,000 observations, then xtdcce2 cannot post any results into Stata matrices, because it tries to post the individual coefficients into a matrix. Of course this is not necessary, but it is a standard Stata routine.

[Fabio Motoki]

Hi JanDitzen! I couldn't manage to get it running with my dataset. I'm using Stata 15.1 SE, and even maxing out the matsize setting to 11000, it always gives the same error below (all variables are continuous, float or double):

The full dataset has 377,776 observations, with 6225 groups. The subset cd_state == 8 has 5,202 obs, 86 groups. Whether I use the full or restricted set, the error continues. I've tried running reduced models with less independent variables to no avail. I have also downloaded the beta xtdcce2135, but it does not work either. Any ideas?

Thanks!

[JanDitzen]

Fabio Motoki, is there any difference if you use the options noomitted or nodimcheck? Also does it work without the if condition?

[Fabio Motoki]

Hi JanDitzen, sorry for the tardy response, only now I had a chance of testing it. So, when I run the command for a subsample (cd_state == 8), the option "nodimcheck" suffices and I get the results. When I run the full sample, I always get the error below, regardless of adding "nodimcheck" or both "nodimcheck noomitted"

I've tried setting matafavor to speed and space, but the error persists. matsize is set to the max (11000), as well as maxvar (32767). The indices [370383, 49800] are always the same.