[Ercan Simsek]

I have dropped the duplicates by using:

. drop if flag==1
(582 observations deleted)

. tsset company_ID date1
       panel variable:  company_ID (unbalanced)
        time variable:  date1, 31jan2003 to 31mar2016, but with gaps
                delta:  1 day

Does this imply that I know can run my regression that qualifies the statistical procedure?

[Carole J. Wilson]

Dropping flag==1 is wrong [see below regarding duplicates]

You need to xtset your data. See help xtset. I assume you want to set it as:

xtset company_ID date1

If you try to xtset your data before getting rid of the duplicates, you will get the error that you reference in your original post.

Flag marks all cases that have the same date. This lets you examine the cases to make sure they are really the same. As you can see, flag is 1 for both instances of 31mar2015. You don't want to exclude both of them, just 1.

The duplicates command is better because it will drop all but 1 instance. You can also use the duplicates command to check if the observations are identical for all variables. Read help duplicates.

 duplicates drop date1

This will drop all but one instance of the duplicate cases (defined by having the same value on date1).

After correcting your duplicate date problem, you can now xtset your data.

Once your data is xtset you can run your regression, but you don't want to use reg as this defeats the purpose of declaring your dataset as a cross-sectional time-series. You want to use xtreg instead (see help xtreg). If you were just going to use reg, there is no reason to xtset your data as reg will ignore that information anyway.

[Ercan Simsek]

Thanks Carole J. Wilson for the useful comment. However, I get the following in Stata:

. duplicates drop date1
force option required with duplicates drop varlist
r(198);

. duplicates drop date1, force

Duplicates in terms of date1

(59,429 observations deleted)

.

[Ercan Simsek]

I guess we forgot to include company_ID variable in the above command:

. duplicates drop company_ID date1, force

Duplicates in terms of company_ID date1

(291 observations deleted)

.

Xtset data:

. xtset company_ID date1
       panel variable:  company_ID (unbalanced)
        time variable:  date1, 31jan2003 to 31mar2016, but with gaps
                delta:  1 day

.

Running regression using xtreg:

. xtreg CE lCE ATO accruals laccruals changesales negchangesales

Random-effects GLS regression                   Number of obs     =     36,122
Group variable: company_ID                      Number of groups  =      1,166

R-sq:                                           Obs per group:
     within  = 0.0341                                         min =          1
     between = 0.5489                                         avg =       31.0
     overall = 0.0492                                         max =         53

                                                Wald chi2(6)      =    1869.59
corr(u_i, X)   = 0 (assumed)                    Prob > chi2       =     0.0000

--------------------------------------------------------------------------------
            CE |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
---------------+----------------------------------------------------------------
           lCE |    .088218   .0051389    17.17   0.000     .0781459    .0982902
           ATO |  -5.80e-08   2.13e-06    -0.03   0.978    -4.24e-06    4.12e-06
      accruals |  -.0586675   .0015102   -38.85   0.000    -.0616275   -.0557076
     laccruals |   .0037709   .0015531     2.43   0.015     .0007268     .006815
   changesales |   .0005675   .0002451     2.32   0.021     .0000872    .0010478
negchangesales |  -4.926075   .7808714    -6.31   0.000    -6.456555   -3.395595
         _cons |  -.1345862   .5538775    -0.24   0.808    -1.220166    .9509939
---------------+----------------------------------------------------------------
       sigma_u |          0
       sigma_e |  71.838386
           rho |          0   (fraction of variance due to u_i)
--------------------------------------------------------------------------------

.

[Carlo Lazzaro]

Ercan:
are you sure that -xtreg, re- is the best specification for your regression model?
Did you investigate that topic via -hausman-?

[Rabih El Habta]

Dear All,

I am trying to set my panel and time variables using the xtset command for my unbalanced dataset, yet, stata replies with the same message as indicated above,
my dataset covers 8 countries and spans the period 1990-2017, reporting bilateral trade flows at the product level, thus I have near one million observation, for the seak of clarity
I kept only 2 countries and 2 products, here is the command I am using:

xtset id year

here is a sample of my data:

. dataex hs2012 productdesc year reporter partner imports exports id

----------------------- copy starting from the next line -----------------------

* Example generated by -dataex-. To install: ssc install dataex
clear
input str6 hs2012 str263 productdesc int year str20 reporter str6 partner double(imports exports) float id
"010121" " Purebred breeding animals" 1992 "Algeria" " World"     5.191       . 1
"010121" " Purebred breeding animals" 1993 "Algeria" " World"   165.213 116.962 1
"010129" " Other"                     1993 "Algeria" " World"         .    .257 1
"010129" " Other"                     1995 "Algeria" " World"    14.524       . 1
"010121" " Purebred breeding animals" 1996 "Algeria" " World"     7.847  34.651 1
"010121" " Purebred breeding animals" 1998 "Algeria" " World"    50.869       . 1
"010121" " Purebred breeding animals" 1999 "Algeria" " World"   277.454       . 1
"010121" " Purebred breeding animals" 2000 "Algeria" " World"    198.84       . 1
"010121" " Purebred breeding animals" 2001 "Algeria" " World"    15.794       . 1
"010121" " Purebred breeding animals" 2002 "Algeria" " World"    75.225  11.075 1
"010121" " Purebred breeding animals" 2003 "Algeria" " World"   767.156       . 1
"010121" " Purebred breeding animals" 2004 "Algeria" " World"   327.402    1.29 1
"010121" " Purebred breeding animals" 2005 "Algeria" " World"   147.659   3.864 1
"010121" " Purebred breeding animals" 2006 "Algeria" " World"   161.066       . 1
"010121" " Purebred breeding animals" 2007 "Algeria" " World"   106.795       . 1
"010121" " Purebred breeding animals" 2008 "Algeria" " World"    74.605       . 1
"010121" " Purebred breeding animals" 2009 "Algeria" " World"   216.574   2.858 1
"010129" " Other"                     2009 "Algeria" " World"     6.061       . 1
"010121" " Purebred breeding animals" 2010 "Algeria" " World"   190.657       . 1
"010129" " Other"                     2010 "Algeria" " World"     9.165       . 1
"010121" " Purebred breeding animals" 2011 "Algeria" " World"   667.423       . 1
"010129" " Other"                     2012 "Algeria" " World"    76.928       . 1
"010121" " Purebred breeding animals" 2012 "Algeria" " World"   866.706  71.776 1
"010121" " Purebred breeding animals" 2013 "Algeria" " World"  1320.711  14.163 1
"010129" " Other"                     2013 "Algeria" " World"    64.178       . 1
"010129" " Other"                     2014 "Algeria" " World"    64.178       . 1
"010121" " Purebred breeding animals" 2014 "Algeria" " World"   1320.71  20.466 1
"010121" " Purebred breeding animals" 2015 "Algeria" " World" 18088.109  86.724 1
"010129" " Other"                     2015 "Algeria" " World"    36.556       . 1
"010121" " Purebred breeding animals" 2016 "Algeria" " World"    978.23 126.137 1
"010129" " Other"                     2016 "Algeria" " World"    96.656       . 1
"010121" " Purebred breeding animals" 2017 "Algeria" " World"   477.123  44.056 1
"010129" " Other"                     2017 "Algeria" " World"    33.761       . 1
"010129" " Other"                     1994 "Jordan"  " World"   182.452   3.863 2
"010121" " Purebred breeding animals" 1995 "Jordan"  " World"    78.385       0 2
"010129" " Other"                     1995 "Jordan"  " World"    64.439  20.591 2
"010129" " Other"                     1997 "Jordan"  " World"    26.796   7.601 2
"010121" " Purebred breeding animals" 1997 "Jordan"  " World"   246.659  70.521 2
"010129" " Other"                     1998 "Jordan"  " World"     36.67   7.191 2
"010121" " Purebred breeding animals" 1998 "Jordan"  " World"    32.024  59.237 2
"010129" " Other"                     1999 "Jordan"  " World"    22.251  32.861 2
"010129" " Other"                     2000 "Jordan"  " World"    21.508  18.547 2
"010121" " Purebred breeding animals" 2000 "Jordan"  " World"    27.515   1.693 2
"010121" " Purebred breeding animals" 2001 "Jordan"  " World"    10.761  84.628 2
"010129" " Other"                     2001 "Jordan"  " World"    19.182  43.253 2
"010121" " Purebred breeding animals" 2002 "Jordan"  " World"    50.908       . 2
"010121" " Purebred breeding animals" 2003 "Jordan"  " World"         .       0 2
"010129" " Other"                     2003 "Jordan"  " World"         .  31.677 2
"010121" " Purebred breeding animals" 2004 "Jordan"  " World"     6.718       . 2
"010129" " Other"                     2004 "Jordan"  " World"    50.694  25.355 2
"010129" " Other"                     2005 "Jordan"  " World"    220.14       . 2
"010121" " Purebred breeding animals" 2005 "Jordan"  " World"    29.619  14.316 2
"010129" " Other"                     2006 "Jordan"  " World"    86.121       . 2
"010121" " Purebred breeding animals" 2006 "Jordan"  " World"   188.748       0 2
"010129" " Other"                     2007 "Jordan"  " World"   292.878       . 2
"010121" " Purebred breeding animals" 2007 "Jordan"  " World"   170.865       . 2
"010129" " Other"                     2008 "Jordan"  " World"         .  32.556 2
"010121" " Purebred breeding animals" 2008 "Jordan"  " World"   199.642  17.133 2
"010121" " Purebred breeding animals" 2009 "Jordan"  " World"   294.106  35.197 2
"010129" " Other"                     2010 "Jordan"  " World"    131.37  32.476 2
"010121" " Purebred breeding animals" 2010 "Jordan"  " World"   230.792   4.404 2
"010129" " Other"                     2011 "Jordan"  " World"   109.492       . 2
"010121" " Purebred breeding animals" 2011 "Jordan"  " World"   261.889   4.494 2
"010129" " Other"                     2012 "Jordan"  " World"   542.524 280.418 2
"010121" " Purebred breeding animals" 2012 "Jordan"  " World"   479.444  60.174 2
"010121" " Purebred breeding animals" 2013 "Jordan"  " World"    277.02  46.216 2
"010129" " Other"                     2013 "Jordan"  " World"   236.445 232.442 2
"010121" " Purebred breeding animals" 2014 "Jordan"  " World"   154.331  78.656 2
"010129" " Other"                     2014 "Jordan"  " World"     271.5  84.289 2
"010129" " Other"                     2015 "Jordan"  " World"     271.5 111.975 2
"010121" " Purebred breeding animals" 2015 "Jordan"  " World"   154.331  54.897 2
"010129" " Other"                     2016 "Jordan"  " World"   342.525  50.316 2
"010121" " Purebred breeding animals" 2016 "Jordan"  " World"    84.626       0 2
"010129" " Other"                     2017 "Jordan"  " World"   710.268  18.889 2
"010121" " Purebred breeding animals" 2017 "Jordan"  " World"    72.394   4.945 2
"010129" " Other"                     1990 "Jordan"  "World"          .       . 3
"010121" " Purebred breeding animals" 1990 "Jordan"  "World"          .       . 3
"010129" " Other"                     1991 "Jordan"  "World"          .       . 3
"010121" " Purebred breeding animals" 1991 "Jordan"  "World"          .       . 3
"010129" " Other"                     1992 "Jordan"  "World"          .       . 3
"010121" " Purebred breeding animals" 1992 "Jordan"  "World"          .       . 3
"010129" " Other"                     1993 "Jordan"  "World"          .       . 3
"010121" " Purebred breeding animals" 1993 "Jordan"  "World"          .       . 3
"010121" " Purebred breeding animals" 1994 "Jordan"  "World"          .       . 3
"010121" " Purebred breeding animals" 1996 "Jordan"  "World"          .       . 3
"010129" " Other"                     1996 "Jordan"  "World"          .       . 3
"010121" " Purebred breeding animals" 1999 "Jordan"  "World"          .       . 3
"010129" " Other"                     2002 "Jordan"  "World"          .       . 3
"010129" " Other"                     2009 "Jordan"  "World"          .       . 3
end
format %ty year

Any help or advice is highly appreciated, thank you so much.

Just one more question, does stata has _T and _t notations unlike N_ and _n denoting time? as I want also to get the difference of the variable imports, however, I am unable to do so using the suffix .d to generate the difference because I am receiving the message above, and unable to use the command gen diff= imports-imports[_n-1] because my dataset is unbalanced any idea?
thanks again.

Rabih.

[Nick Cox]

Observations 2 and 3 alone in your example show the problem, i.e. you have there two observations with id 1 and year 1993. So, there should be no surprise. Still, the question is what qualifies as a panel and, even more crucially, what commands do you intend to use that require xtset?.

does stata has _T and _t notations unlike N_ and _n denoting time?

No such notation is provided automatically. If it existed it would require xtset or tsset, so no go there.

Perhaps what you want is something like

bysort product country (year) : gen previous = imports[_n-1] if year - year[_n-1] == 1

[Rabih El Habta]

Thank you, dear Nick,

To answer your question I actually never wanted to use the xtset command I only wanted to generate the difference of imports and I totally missed out and overlooked that I can use the if option
as the first thing that I immediately thought about is the use of the d. suffix to sort that problem, and I needed first the set my time and panel variables, yet, I got stuck up with that error. After taking into account the duplicate, I re-generate the id variable, the xtset command worked:

egen id=group(reporter partner hs2012)
xtset id year
bys id: gen d_imports=d.imports

thanks, Neck.