How to use lagged variable of interest in ppmlhdfe estimation?

[Koko DIBLONI]

Dear all,

Could you give me a hand, please?

I'm analysing the effects of Fiscal consolidation episodes on FDI inflows. But I face a technical problem when I use lagged variable. Following my code:

ppmlhdfe in_Flow_per_r L2.Fisc_r log_GDP_r log_POP_r res_rents_r remit_gdp_r access_elec_r gov_eff_r gdp_growth_r fin_dev_r elec_consump_r , vce(cl iso_r iso_p) absorb(iso_p#iso_r iso_p#year) nolog d keepsingletons separation(none)


output : ppmlhdfe in_Flow_per_r L2.Fisc_r log_GDP_r log_POP_r res_rents_r remit_gdp_r access_elec_r gov_eff_r gdp_growth_r fin_dev_r elec_consump_r , vce(cl iso_r iso_p) absorb(iso_p#iso_r iso_p#year) nolog d keepsingletons separation(none)
not sorted

But I already did this:
egen ID=group(iso_r iso_p)
xtset ID year

sort iso_p iso_r year


data :

* Example generated by -dataex-. For more info, type help dataex
clear
input long(iso_r iso_p) int year float in_Flow_per_r double Fisc_r float(res_rents_r remit_gdp_r access_elec_r gov_eff_r gdp_growth_r fin_dev_r elec_consump_r) str3 niveau_rev float(ID log_GDP_r log_POP_r InstDiff)
1 2 2006 0                 1.3 .5041744         0 33.5 -1.4736515  5.357403 23.343864 . "LIC" 1 7.565132 17.051208  .4894549
1 2 2007 0                 1.8 .4495006         0 38.4 -1.4401106  13.82632 23.833664 . "LIC" 1  7.67571 17.070133  .5451503
1 2 2008 0 -2.1999999999999997 .6016659  .8857237 42.4  -1.527795  3.924984  29.51228 . "LIC" 1 7.692344    17.092  .7065468
1 2 2009 0 -1.5999999999999999 .4246582 1.1331624 48.3 -1.5077522  21.39053  33.06096 . "LIC" 1 7.849722 17.128464   .762536
1 2 2010 0                  .7 .6253326  2.385158 42.7 -1.4783156  14.36244 35.350563 . "LIC" 1 7.954578  17.15781   .723882
1 2 2011 0  .09999999999999964 .8363897 1.0059847 43.2 -1.4741005  .4263548  36.62423 . "LIC" 1 7.921917 17.194725  .6552385
1 2 2012 0                   0 .6846755 1.1021874 69.1  -1.375535 12.752287 32.913692 . "LIC" 1 8.001462 17.235203 .54737365
1 2 2013 0                   0 .6035717  1.723211   68  -1.401463  5.600745  32.68202 . "LIC" 1 8.021775 17.269386   .498688
1 2 2014 0                -4.1 .5427565 1.2361138 89.5  -1.361646  2.724543   34.2096 . "LIC" 1 8.012331 17.305712  .4843357
1 2 2015 0                   0 .4699169  1.821996 71.5  -1.397395 1.4513147  33.92805 . "LIC" 1  7.99554  17.33691 .50561553
1 2 2016 0                  .5 .6174099  3.464843 97.7  -1.291468 2.2603142  37.22282 . "LIC" 1 7.992534 17.362268  .5838875
1 2 2017 0                 1.6 .7029817  4.387093 97.7 -1.3865968  2.647003  36.86126 . "LIC" 1 7.990577  17.39035  .6135066
1 2 2018 0  .09999999999999987 .8552642 4.4509864 93.4  -1.501758 1.1892282 36.607494 . "LIC" 1  7.97329  17.41946  .7226343
1 3 2006 0                 1.3 .5041744         0 33.5 -1.4736515  5.357403 23.343864 . "LIC" 2 7.565132 17.051208  1.193724
1 3 2007 0                 1.8 .4495006         0 38.4 -1.4401106  13.82632 23.833664 . "LIC" 2  7.67571 17.070133 1.3721994
1 3 2008 0 -2.1999999999999997 .6016659  .8857237 42.4  -1.527795  3.924984  29.51228 . "LIC" 2 7.692344    17.092 1.5467296
1 3 2009 0 -1.5999999999999999 .4246582 1.1331624 48.3 -1.5077522  21.39053  33.06096 . "LIC" 2 7.849722 17.128464 1.6216325
1 3 2010 0                  .7 .6253326  2.385158 42.7 -1.4783156  14.36244 35.350563 . "LIC" 2 7.954578  17.15781   1.57712
1 3 2011 0  .09999999999999964 .8363897 1.0059847 43.2 -1.4741005  .4263548  36.62423 . "LIC" 2 7.921917 17.194725 1.5147436
1 3 2012 0                   0 .6846755 1.1021874 69.1  -1.375535 12.752287 32.913692 . "LIC" 2 8.001462 17.235203  1.314634
1 3 2013 0                   0 .6035717  1.723211   68  -1.401463  5.600745  32.68202 . "LIC" 2 8.021775 17.269386  1.368342
1 3 2014 0                -4.1 .5427565 1.2361138 89.5  -1.361646  2.724543   34.2096 . "LIC" 2 8.012331 17.305712 1.4660298
1 3 2015 0                   0 .4699169  1.821996 71.5  -1.397395 1.4513147  33.92805 . "LIC" 2  7.99554  17.33691 1.4671912
1 3 2016 0                  .5 .6174099  3.464843 97.7  -1.291468 2.2603142  37.22282 . "LIC" 2 7.992534 17.362268  1.553858
1 3 2017 0                 1.6 .7029817  4.387093 97.7 -1.3865968  2.647003  36.86126 . "LIC" 2 7.990577  17.39035 1.6080048
1 3 2018 0  .09999999999999987 .8552642 4.4509864 93.4  -1.501758 1.1892282 36.607494 . "LIC" 2  7.97329  17.41946 1.5874147
1 4 2006 0                 1.3 .5041744         0 33.5 -1.4736515  5.357403 23.343864 . "LIC" 3 7.565132 17.051208 1.4444188
1 4 2007 0                 1.8 .4495006         0 38.4 -1.4401106  13.82632 23.833664 . "LIC" 3  7.67571 17.070133 1.5017364
1 4 2008 0 -2.1999999999999997 .6016659  .8857237 42.4  -1.527795  3.924984  29.51228 . "LIC" 3 7.692344    17.092 1.4801705
1 4 2009 0 -1.5999999999999999 .4246582 1.1331624 48.3 -1.5077522  21.39053  33.06096 . "LIC" 3 7.849722 17.128464 1.4113365
1 4 2010 0                  .7 .6253326  2.385158 42.7 -1.4783156  14.36244 35.350563 . "LIC" 3 7.954578  17.15781  1.493788
1 4 2011 0  .09999999999999964 .8363897 1.0059847 43.2 -1.4741005  .4263548  36.62423 . "LIC" 3 7.921917 17.194725   1.52897
1 4 2012 0                   0 .6846755 1.1021874 69.1  -1.375535 12.752287 32.913692 . "LIC" 3 8.001462 17.235203  1.260171
1 4 2013 0                   0 .6035717  1.723211   68  -1.401463  5.600745  32.68202 . "LIC" 3 8.021775 17.269386 1.2464997
1 4 2014 0                -4.1 .5427565 1.2361138 89.5  -1.361646  2.724543   34.2096 . "LIC" 3 8.012331 17.305712 1.0969124
1 4 2015 0                   0 .4699169  1.821996 71.5  -1.397395 1.4513147  33.92805 . "LIC" 3  7.99554  17.33691 1.1850852
1 4 2016 0                  .5 .6174099  3.464843 97.7  -1.291468 2.2603142  37.22282 . "LIC" 3 7.992534 17.362268  1.556127
1 4 2017 0                 1.6 .7029817  4.387093 97.7 -1.3865968  2.647003  36.86126 . "LIC" 3 7.990577  17.39035   1.62173
1 4 2018 0  .09999999999999987 .8552642 4.4509864 93.4  -1.501758 1.1892282 36.607494 . "LIC" 3  7.97329  17.41946 1.6087884
1 5 2006 0                 1.3 .5041744         0 33.5 -1.4736515  5.357403 23.343864 . "LIC" 4 7.565132 17.051208  1.271938
1 5 2007 0                 1.8 .4495006         0 38.4 -1.4401106  13.82632 23.833664 . "LIC" 4  7.67571 17.070133   1.35304
1 5 2008 0 -2.1999999999999997 .6016659  .8857237 42.4  -1.527795  3.924984  29.51228 . "LIC" 4 7.692344    17.092 1.4631615
1 5 2009 0 -1.5999999999999999 .4246582 1.1331624 48.3 -1.5077522  21.39053  33.06096 . "LIC" 4 7.849722 17.128464  1.532571
1 5 2010 0                  .7 .6253326  2.385158 42.7 -1.4783156  14.36244 35.350563 . "LIC" 4 7.954578  17.15781 1.4398786
1 5 2011 0  .09999999999999964 .8363897 1.0059847 43.2 -1.4741005  .4263548  36.62423 . "LIC" 4 7.921917 17.194725 1.4410716
1 5 2012 0                   0 .6846755 1.1021874 69.1  -1.375535 12.752287 32.913692 . "LIC" 4 8.001462 17.235203   1.36938
1 5 2013 0                   0 .6035717  1.723211   68  -1.401463  5.600745  32.68202 . "LIC" 4 8.021775 17.269386 1.3956867
1 5 2014 0                -4.1 .5427565 1.2361138 89.5  -1.361646  2.724543   34.2096 . "LIC" 4 8.012331 17.305712 1.1489593
1 5 2015 0                   0 .4699169  1.821996 71.5  -1.397395 1.4513147  33.92805 . "LIC" 4  7.99554  17.33691 1.1599008
1 5 2016 0                  .5 .6174099  3.464843 97.7  -1.291468 2.2603142  37.22282 . "LIC" 4 7.992534 17.362268 1.2031314
1 5 2017 0                 1.6 .7029817  4.387093 97.7 -1.3865968  2.647003  36.86126 . "LIC" 4 7.990577  17.39035 1.2650574
1 5 2018 0  .09999999999999987 .8552642 4.4509864 93.4  -1.501758 1.1892282 36.607494 . "LIC" 4  7.97329  17.41946 1.4307468
1 6 2006 0                 1.3 .5041744         0 33.5 -1.4736515  5.357403 23.343864 . "LIC" 5 7.565132 17.051208  .7115992
1 6 2007 0                 1.8 .4495006         0 38.4 -1.4401106  13.82632 23.833664 . "LIC" 5  7.67571 17.070133  .8247666
1 6 2008 0 -2.1999999999999997 .6016659  .8857237 42.4  -1.527795  3.924984  29.51228 . "LIC" 5 7.692344    17.092  .9614503
1 6 2009 0 -1.5999999999999999 .4246582 1.1331624 48.3 -1.5077522  21.39053  33.06096 . "LIC" 5 7.849722 17.128464 1.0159104
1 6 2010 0                  .7 .6253326  2.385158 42.7 -1.4783156  14.36244 35.350563 . "LIC" 5 7.954578  17.15781  .9347796
1 6 2011 0  .09999999999999964 .8363897 1.0059847 43.2 -1.4741005  .4263548  36.62423 . "LIC" 5 7.921917 17.194725  .8741065
1 6 2012 0                   0 .6846755 1.1021874 69.1  -1.375535 12.752287 32.913692 . "LIC" 5 8.001462 17.235203  .6699054
1 6 2013 0                   0 .6035717  1.723211   68  -1.401463  5.600745  32.68202 . "LIC" 5 8.021775 17.269386  .8315789
1 6 2014 0                -4.1 .5427565 1.2361138 89.5  -1.361646  2.724543   34.2096 . "LIC" 5 8.012331 17.305712  .7489748
1 6 2015 0                   0 .4699169  1.821996 71.5  -1.397395 1.4513147  33.92805 . "LIC" 5  7.99554  17.33691  .7303885
1 6 2016 0                  .5 .6174099  3.464843 97.7  -1.291468 2.2603142  37.22282 . "LIC" 5 7.992534 17.362268  .8461836
1 6 2017 0                 1.6 .7029817  4.387093 97.7 -1.3865968  2.647003  36.86126 . "LIC" 5 7.990577  17.39035  .8826642
1 6 2018 0  .09999999999999987 .8552642 4.4509864 93.4  -1.501758 1.1892282 36.607494 . "LIC" 5  7.97329  17.41946  .9068323
1 7 2006 0                 1.3 .5041744         0 33.5 -1.4736515  5.357403 23.343864 . "LIC" 6 7.565132 17.051208  .5469258
1 7 2007 0                 1.8 .4495006         0 38.4 -1.4401106  13.82632 23.833664 . "LIC" 6  7.67571 17.070133  .5398743
1 7 2008 0 -2.1999999999999997 .6016659  .8857237 42.4  -1.527795  3.924984  29.51228 . "LIC" 6 7.692344    17.092  .6244408
1 7 2009 0 -1.5999999999999999 .4246582 1.1331624 48.3 -1.5077522  21.39053  33.06096 . "LIC" 6 7.849722 17.128464  .6999505
1 7 2010 0                  .7 .6253326  2.385158 42.7 -1.4783156  14.36244 35.350563 . "LIC" 6 7.954578  17.15781  .5506468
1 7 2011 0  .09999999999999964 .8363897 1.0059847 43.2 -1.4741005  .4263548  36.62423 . "LIC" 6 7.921917 17.194725 .52818835
1 7 2012 0                   0 .6846755 1.1021874 69.1  -1.375535 12.752287 32.913692 . "LIC" 6 8.001462 17.235203  .3203721
1 7 2013 0                   0 .6035717  1.723211   68  -1.401463  5.600745  32.68202 . "LIC" 6 8.021775 17.269386  .4385171
1 7 2014 0                -4.1 .5427565 1.2361138 89.5  -1.361646  2.724543   34.2096 . "LIC" 6 8.012331 17.305712  .4806899
1 7 2015 0                   0 .4699169  1.821996 71.5  -1.397395 1.4513147  33.92805 . "LIC" 6  7.99554  17.33691  .2153752
1 7 2016 0                  .5 .6174099  3.464843 97.7  -1.291468 2.2603142  37.22282 . "LIC" 6 7.992534 17.362268 .12862122
1 7 2017 0                 1.6 .7029817  4.387093 97.7 -1.3865968  2.647003  36.86126 . "LIC" 6 7.990577  17.39035 .18129122
1 7 2018 0  .09999999999999987 .8552642 4.4509864 93.4  -1.501758 1.1892282 36.607494 . "LIC" 6  7.97329  17.41946  .1502489
1 8 2006 0                 1.3 .5041744         0 33.5 -1.4736515  5.357403 23.343864 . "LIC" 7 7.565132 17.051208  1.416827
1 8 2007 0                 1.8 .4495006         0 38.4 -1.4401106  13.82632 23.833664 . "LIC" 7  7.67571 17.070133  1.467792
1 8 2008 0 -2.1999999999999997 .6016659  .8857237 42.4  -1.527795  3.924984  29.51228 . "LIC" 7 7.692344    17.092  1.533236
1 8 2009 0 -1.5999999999999999 .4246582 1.1331624 48.3 -1.5077522  21.39053  33.06096 . "LIC" 7 7.849722 17.128464  1.526435
1 8 2010 0                  .7 .6253326  2.385158 42.7 -1.4783156  14.36244 35.350563 . "LIC" 7 7.954578  17.15781 1.4531566
1 8 2011 0  .09999999999999964 .8363897 1.0059847 43.2 -1.4741005  .4263548  36.62423 . "LIC" 7 7.921917 17.194725 1.4404615
1 8 2012 0                   0 .6846755 1.1021874 69.1  -1.375535 12.752287 32.913692 . "LIC" 7 8.001462 17.235203 1.2440605
1 8 2013 0                   0 .6035717  1.723211   68  -1.401463  5.600745  32.68202 . "LIC" 7 8.021775 17.269386 1.2713617
1 8 2014 0                -4.1 .5427565 1.2361138 89.5  -1.361646  2.724543   34.2096 . "LIC" 7 8.012331 17.305712 1.1441971
1 8 2015 0                   0 .4699169  1.821996 71.5  -1.397395 1.4513147  33.92805 . "LIC" 7  7.99554  17.33691 1.1363094
1 8 2016 0                  .5 .6174099  3.464843 97.7  -1.291468 2.2603142  37.22282 . "LIC" 7 7.992534 17.362268 1.2380135
1 8 2017 0                 1.6 .7029817  4.387093 97.7 -1.3865968  2.647003  36.86126 . "LIC" 7 7.990577  17.39035  1.268436
1 8 2018 0  .09999999999999987 .8552642 4.4509864 93.4  -1.501758 1.1892282 36.607494 . "LIC" 7  7.97329  17.41946 1.2641116
1 9 2006 0                 1.3 .5041744         0 33.5 -1.4736515  5.357403 23.343864 . "LIC" 8 7.565132 17.051208 1.2818816
1 9 2007 0                 1.8 .4495006         0 38.4 -1.4401106  13.82632 23.833664 . "LIC" 8  7.67571 17.070133 1.3794863
1 9 2008 0 -2.1999999999999997 .6016659  .8857237 42.4  -1.527795  3.924984  29.51228 . "LIC" 8 7.692344    17.092  1.498753
1 9 2009 0 -1.5999999999999999 .4246582 1.1331624 48.3 -1.5077522  21.39053  33.06096 . "LIC" 8 7.849722 17.128464  1.520483
1 9 2010 0                  .7 .6253326  2.385158 42.7 -1.4783156  14.36244 35.350563 . "LIC" 8 7.954578  17.15781 1.4623445
1 9 2011 0  .09999999999999964 .8363897 1.0059847 43.2 -1.4741005  .4263548  36.62423 . "LIC" 8 7.921917 17.194725 1.3389003
1 9 2012 0                   0 .6846755 1.1021874 69.1  -1.375535 12.752287 32.913692 . "LIC" 8 8.001462 17.235203 1.1297531
1 9 2013 0                   0 .6035717  1.723211   68  -1.401463  5.600745  32.68202 . "LIC" 8 8.021775 17.269386 1.0784836
1 9 2014 0                -4.1 .5427565 1.2361138 89.5  -1.361646  2.724543   34.2096 . "LIC" 8 8.012331 17.305712   .958196
end
label values iso_r iso_r_num
label def iso_r_num 1 "AFG", modify
label values iso_p iso_p_num
label def iso_p_num 2 "AGO", modify
label def iso_p_num 3 "ALB", modify
label def iso_p_num 4 "ARG", modify
label def iso_p_num 5 "ARM", modify
label def iso_p_num 6 "AZE", modify
label def iso_p_num 7 "BDI", modify
label def iso_p_num 8 "BEN", modify
label def iso_p_num 9 "BFA", modify

[Clyde Schechter]

OK, notwithstanding my earlier disclaimer that I don't know anything about -ppmlhdfe-, I see the problem. It has nothing to do with -ppmlhdfe- at all. The problem is with the sorting of the data. Let me explain.

When Stata sees the L2 lag operator (or any time series operator) it refers back to the -xtset- command that is currently in effect--in your case -xtset ID year-. It then expects the data to be sorted in exactly that way. But, after you -xtset- the data you then issued a different -sort- command, -sort iso_p iso_r year-. So Stata sees that the data are not sorted on ID and year but on iso_p, iso_r, and year. So it is not able to apply the lag operator.

You can verify this is the problem in two ways. First, try changing ppmlhdfe to any other regression command (eliminating any options that are not applicable to the other command) and you will see that you get exactly the same problem. Next, more conclusive, restore the -ppmlhdfe- command to its original state and delete the -sort iso_p iso_r year- command. Now rerun the code from the start. You will see that the -ppmlhdfe- command no longer complains about the data not being sorted. The -sort iso_p iso_r year- command was never necessary in the first place, because the -xtset- command itself, as part of its operation, applies the proper sort order. But in your case, it was worse than unnecessary: it undid the good work that -xtset- had done.

Now, I can almost hear you saying, "But, ID was defined as -egen ID = group(iso_p iso_r)- so isn't -sort iso_p iso_r year- the same thing as -sort ID year-?" Well, in a sense, they are the same: the actual ordering of the data set is the same with either of those. But when Stata wants to verify the current sort order of the data, it does not scan the entire data set to see that the observations are in the intended order. Instead, Stata maintains an internal variable (which, if you like, you can access with -display "`:sortedby'"-) which tells it how the data is currently sorted, and that internal variable is set by -xtset- or by -sort- (and a few other commands that include a sorting operation or otherwise change the sort order of the data). When Stata needs to know how the data are sorted, it just checks that variable. It does not further interrogate how those variables relate to other variables in the data set. As far as checking on the current sort order is concerned, Stata just looks at `:sortedby' and takes it at face value. And if what it sees there is not what it needs, it considers the data to be not properly sorted.

So, those are the details of what went wrong. All you need to do is remove that -sort- command.

As an aside, after correcting that problem, your example data still do not enable the -ppmlhdfe- command to run, but the error message is very different. And this, I am confident based on my knowledge of Poisson regression, is because in the example data, the outcome variable, in_Flow_per_r, is always 0, so that the Poisson coefficients are not estimable. I take it that in the full data set in_Flow_per_r takes on some non-zero values, and you won't encounter that problem.

[Koko DIBLONI]

Thank you for your help, that solution worked perfectly.