Error message: Cannot compute an improvement -- discontinuous region encountered

[Said Jafar]

Dear Stata Users,

I am trying to run xtpoisson command with two additional fixed effects, which are importer time and exporter time fixed effects (it and jt, respectively). I have with 85000 observations and for 11 years (originally 500K+ with 20 years). My panel is strongly balanced. I execute comand as below;

 
 xtpoisson  trade fta1 fta2 fta3 i.it i.jt, fe vce(r)

and, after some iterations, get an error message as:

Cannot compute an improvement -- discontinuous region encountered

Does anyone knows why this happens? And what would be the solution to this problem?

Note: I am using Stata 13 MP 8 core 64 bit, with 4 core, 64 bit, 6 ram pc. I have set max_memory to infinity, maxiter to 16000.

Kind Regards,
Dias Rafaj

[Carlo Lazzaro]

Dias:
Stata cannot improve ML. You may want to re-run your model, adding one predictor at a time to see when Stata computation starts chocking.
Obviously, you should consider that this procedure may take lot of your times.

[Phil Bromiley]

You might also try a different maximization algorithm.
Phil

[Said Jafar]

Dear Carlo and Phil,

Thank you for your response.

I did run it with only importer time fixed effects, but it did not converge. 10 iterations pasted, all with log pseudolikelihood = -5.445e-06.

As i stated above, my data is trade flows between countries, which are around 90, where roughly 50 percent of the trade flows are equal to zero (many countries do not trade with each other). I cannot drop those zero flows as it would lead to biases. Santos Silva and Tanyero (2006, 2010) reports that Possion Pseudo Maximum Likelihood (ppml) performs better in the presence of zero flow and estimations with ppml is benchmark now. I have tried the model with -ppml- and -glm .. .. ... family (poisson) link(log)- commands: by including the said fixed effects: ppml said initial values are not feasible (after 10 hours) while glm did not converge.

Do you know what would be the solution in this case?

Kind Regards,
Dias





Reference:
The log of gravity
JMC Santos Silva, S Tenreyro
The Review of Economics and Statistics 88 (4), 641-658

Further simulation evidence on the performance of the Poisson pseudo-maximum likelihood estimator
JMC Santos Silva, S Tenreyro
Economics Letters 112 (2), 220-222

[Carlo Lazzaro]

Dias:
let's hope that Joao is going to chime in!

[Joao Santos Silva]

Dear Dias,

There may be a couple of issues here (all of which were discussed in a paper we published the the Stata Journal (sorry, do not have the reference here). The first thing I would try to do would be to change the scale of the dependent variable so that the log-likelihood function has values, say, between 100 and 1000. Believe it or not, Stata is sensitive to that!

All the best,

Joao

[Said Jafar]

Dear Mr Joao,

Thank you for your contribution.

I had already rescaled my dependent variable and some other variables (divided by a zillion). Summarize command for variable trade yields;

. sum trade

    Variable |       Obs        Mean    Std. Dev.       Min        Max
-------------+--------------------------------------------------------
       trade |    547800    2.98e-11    3.13e-10          0   2.64e-08

I also tried _rmcol, but it did not drop any variable. I am wondering what might cause this error then..

[Nick Cox]

Dias Rafaj: Those values now look dangerously small. Truly, you can't win here easily, but numerically it doesn't follow that problems with very large numbers will be solved by making them all very small.

[Joao Santos Silva]

Dear Dias,

Nick is right; dividing by a zillion is too much. Get the original variable and try dividing by a million or so. As I said before, try to get the log-l between 100 and 1000, or something like that.

All the best,

Joao

[Said Jafar]

Dear Nick, Dear Joao,

Thank you for your advises. Apologizes, i did not have an access to the PCs because of Chinese New Year break.

I had rescaled variables at that level to avoid warnings of ppml command and to speed up the estimations. I multiplied the depended variable by a million and tried it with different scenarios. Firstly, i kept both importer time and exporter time fixed effects. Below is what i had after 9 hours of estimations in four PCs.

xtpoisson trade fta1 fta2 fta3  i.id_it i.id_jt, fe vce(r)
note: you are responsible for interpretation of non-count dep. variable
note: 1775 groups (35500 obs) dropped because of all zero outcomes
note: 3302.jt omitted because of collinearity
note: 3303.jt omitted because of collinearity
(17 jt's more) 
 
Iteration 0:   log pseudolikelihood = -44.996938  (not concave)
Iteration 1:   log pseudolikelihood = -43.187162  (not concave)
Iteration 2:   log pseudolikelihood = -43.183935  (not concave)
Iteration 3:   log pseudolikelihood = -43.100026  (not concave)
Iteration 4:   log pseudolikelihood =  -43.08152  (not concave)

Later run the first model with importer time fixed effect only, but, again, did not see any result;

 xtpoisson trade fta1 fta2 fta3  i.id_it, fe vce(r)
note: you are responsible for interpretation of non-count dep. variable
note: 1775 groups (35500 obs) dropped because of all zero outcomes
note: 3240.id_it omitted because of collinearity
 
Iteration 0:   log pseudolikelihood = -44.996938  (not concave)
Iteration 1:   log pseudolikelihood = -43.379474  (not concave)
Iteration 2:   log pseudolikelihood = -43.375329  (not concave)
Iteration 3:   log pseudolikelihood = -43.337605  (not concave)
Iteration 4:   log pseudolikelihood = -43.322499  (not concave)
Iteration 5:   log pseudolikelihood = -43.314915  (not concave)
Iteration 6:   log pseudolikelihood = -43.314256  (not concave)

.


Thankfully, interations continue to decrease but since labs closes and comps automatically shuts off, i cant keep model running.This would be rather general question but do you know what would be reason for not being concave in this case? I searched a lot for this but id not find anything relevant to my case.

Let me note that i also tried it with -xi: ppml- by including pair, importer-time, exporter-time fe but stata complaint that matsize is not enough (which is already 11000 ). I did the the same estimation with xtpoisson by including time fe only, and got results within 15 seconds.

Thank you for your inputs in advance.

Kind regards,
Dias

[Joao Santos Silva]

Dear Dias,

1) Multiplying by a million after dividing by a zillion may not be a good idea. Ideally, you should start from the original variable a divide by a suitable amount.

2) The "not concave" warning is typical in the first few iterations of non-linear models.

3) With a sample as large as yours, it is not surprising that the estimation takes many hours.

Good luck with it!

Joao