Why has Stata dropped observations from my fixed effects model?

[Darcy Hill]

Hi,

I have a dataset of 318 local authorities and am running a fixed effects regression.
However when I run the regression Stata only performs this for 311 observations.
I ran the command
tab acode if !e(sample)
However it is still not obvious why 7 of these local authorities (acode) have been excluded, is there a way to find out why/are there common reasons why? I have looked at the dataset and it is not missing any observations?
Thank You

[William Lisowski]

Usually the output will give you some clues why observations are dropped, and perhaps you have overlooked or misinterpreted something it is telling you.

You say the dataset is not missing any observations. I assume you mean the seven omitted observations have no missing values for any of the variables that are used in the model, and perhaps you confirmed that with

browse if !e(sample)

Without seeing your command and its results, it's difficult to offer more definitive advice.

To assure maximum readability of results that you post, please copy them from the Results window or your log file into a code block in the Forum editor using code delimiters [CODE] and [/CODE], as explained in section 12 of the Statalist FAQ linked to at the top of the page. For example, the following:

[CODE]
. sysuse auto, clear
(1978 Automobile Data)

. describe make price

storage display value
variable name type format label variable label
-----------------------------------------------------------------
make str18 %-18s Make and Model
price int %8.0gc Price
[/CODE]

will be presented in the post as the following:

. sysuse auto, clear
(1978 Automobile Data)

. describe make price

              storage   display    value
variable name   type    format     label      variable label
-----------------------------------------------------------------
make            str18   %-18s                 Make and Model
price           int     %8.0gc                Price

[Darcy Hill]

Hi William, below is my output!

. xtreg recycling loginc logpopden md11 md12 md13 md14 md15 md16 md17 md18 md19 md20 md21 
> md22 md23 md24 md25 md26 md27 md28 md29 md291 wasteavg dryavg quarter2 quarter3 quarter4
>  year2 year3 year4 year5, fe vce(cluster acode)

Fixed-effects (within) regression               Number of obs     =      5,862
Group variable: acode                           Number of groups  =        311

R-sq:                                           Obs per group:
     within  = 0.3639                                         min =          4
     between = 0.0634                                         avg =       18.8
     overall = 0.0941                                         max =         20

                                                F(31,310)         =      53.22
corr(u_i, Xb)  = -0.6908                        Prob > F          =     0.0000

                                (Std. Err. adjusted for 311 clusters in acode)
------------------------------------------------------------------------------
             |               Robust
   recycling |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
      loginc |   6.532939   6.030714     1.08   0.280    -5.333371    18.39925
   logpopden |  -3.643093   2.239148    -1.63   0.105    -8.048943    .7627563
        md11 |   .3323071   .4808336     0.69   0.490    -.6138031    1.278417
        md12 |  -.6383587    .408626    -1.56   0.119     -1.44239    .1656725
        md13 |  -.9090128   .5404645    -1.68   0.094    -1.972456      .15443
        md14 |  -.1001455   .4725626    -0.21   0.832    -1.029981    .8296903
        md15 |   .5641031   .5839921     0.97   0.335    -.5849866    1.713193
        md16 |   .1721142   .7144448     0.24   0.810     -1.23366    1.577889
        md17 |  -.2672899   .3879316    -0.69   0.491    -1.030602    .4960221
        md18 |  -.0376456   .6234474    -0.06   0.952    -1.264369    1.189078
        md19 |   .0423547   .4558731     0.09   0.926    -.8546421    .9393515
        md20 |  -1.560784   .5148514    -3.03   0.003     -2.57383   -.5477388
        md21 |  -.8955305   .4952328    -1.81   0.072    -1.869973    .0789123
        md22 |  -1.264937   .5061731    -2.50   0.013    -2.260907    -.268968
        md23 |   .1594418   .5278072     0.30   0.763    -.8790958    1.197979
        md24 |   .3258512   .4136664     0.79   0.431    -.4880977      1.1398
        md25 |   .1241636   .9183427     0.14   0.893     -1.68281    1.931137
        md26 |   .5464737   .3863866     1.41   0.158    -.2137983    1.306746
        md27 |   1.420912   .3756702     3.78   0.000     .6817256    2.160097
        md28 |    .381145   .2994553     1.27   0.204     -.208077     .970367
        md29 |   .0880614   .6923709     0.13   0.899    -1.274279    1.450402
       md291 |   .4493586   .3743921     1.20   0.231    -.2873125     1.18603
    wasteavg |    1.43095   .5684451     2.52   0.012      .312451    2.549448
      dryavg |  -.6606502   .6478801    -1.02   0.309    -1.935449    .6141484
    quarter2 |  -4.473236    .121817   -36.72   0.000    -4.712929   -4.233544
    quarter3 |  -4.172665   .1183665   -35.25   0.000    -4.405568   -3.939761
    quarter4 |  -2.505768   .1010889   -24.79   0.000    -2.704675   -2.306861
       year2 |   .1246461   .2125739     0.59   0.558    -.2936241    .5429162
       year3 |  -.4222133   .3448504    -1.22   0.222    -1.100757    .2563301
       year4 |  -1.007958   .6575602    -1.53   0.126    -2.301804    .2858874
       year5 |  -1.431394    .692197    -2.07   0.039    -2.793393   -.0693957
       _cons |  -30.59105   60.24127    -0.51   0.612    -149.1246    87.94244
-------------+----------------------------------------------------------------
     sigma_u |  5.7402679
     sigma_e |  2.5635018
         rho |  .83372539   (fraction of variance due to u_i)
------------------------------------------------------------------------------

. 
end of do-file

. tab acode if !e(sample)

      Local |
  Authority |      Freq.     Percent        Cum.
------------+-----------------------------------
    6000005 |          4        0.82        0.82
    6000008 |          4        0.82        1.64
    6000014 |          4        0.82        2.46
    6000022 |          4        0.82        3.28
    6000025 |          8        1.64        4.92
    6000029 |          4        0.82        5.74
    6000030 |          4        0.82        6.56
    6000053 |         20        4.10       10.66
    6000055 |          4        0.82       11.48
    6000056 |         20        4.10       15.57
    7000037 |          4        0.82       16.39
    7000041 |          4        0.82       17.21
    7000071 |          8        1.64       18.85
    7000074 |          4        0.82       19.67
    7000081 |          4        0.82       20.49
    7000082 |          4        0.82       21.31
    7000087 |          4        0.82       22.13
    7000089 |          4        0.82       22.95
    7000093 |         16        3.28       26.23
    7000105 |          8        1.64       27.87
    7000109 |          4        0.82       28.69
    7000112 |          4        0.82       29.51
    7000113 |          4        0.82       30.33
    7000116 |         20        4.10       34.43
    7000117 |         16        3.28       37.70
    7000130 |         16        3.28       40.98
    7000140 |          4        0.82       41.80
    7000141 |          8        1.64       43.44
    7000142 |          4        0.82       44.26
    7000152 |         12        2.46       46.72
    7000154 |         12        2.46       49.18
    7000155 |          8        1.64       50.82
    7000171 |         16        3.28       54.10
    7000174 |          4        0.82       54.92
    7000178 |          4        0.82       55.74
    7000181 |          4        0.82       56.56
    7000192 |          8        1.64       58.20
    7000199 |          4        0.82       59.02
    7000206 |         12        2.46       61.48
    7000210 |          4        0.82       62.30
    7000219 |          4        0.82       63.11
    7000220 |          8        1.64       64.75
    7000222 |          4        0.82       65.57
    7000224 |          8        1.64       67.21
    7000234 |          4        0.82       68.03
    7000236 |          4        0.82       68.85
    7000237 |          4        0.82       69.67
    7000241 |          8        1.64       71.31
    7000242 |          4        0.82       72.13
    8000005 |          4        0.82       72.95
    8000025 |          8        1.64       74.59
    9000010 |         12        2.46       77.05
    9000012 |          8        1.64       78.69
    9000013 |         20        4.10       82.79
    9000020 |         20        4.10       86.89
    9000022 |         20        4.10       90.98
    9000030 |          4        0.82       91.80
    9000031 |          4        0.82       92.62
    9000032 |         20        4.10       96.72
    9000033 |         16        3.28      100.00
------------+-----------------------------------
      Total |        488      100.00

[William Lisowski]

Thank you for the output presented with CODE delimiters - it is very readable and reveals a lot about your problem,

First, we need to clarify terminology. To Stata, what Excel would call a "row" in your data is what Stata calls an "observation". So you have far more than 311 or 318 observations in your data.

Your dataset appears to consist of 6350 observations from 318 authorities. Each authority provides up to 5 years of quarterly data, so there are up to 20 observations from each authority.

Your tab output shows that Stata omitted 488 observations from 60 different authorities from your regression. Stata does this because for each omitted observation one or more of the variables in the model had a missing value. Thus your xtreg output reports that 5862 observations were included in the regression rather than 6350.

Of those 60 authorities with omitted observations, 7 of them had all 20 observations omitted. So nothing from those 7 authorities was included in the regression, and that is why xtreg reports 311 groups rather than 318. And each of these 60 authorities has 4, 8, 12, 16, or 20 omitted observations, suggesting that the missing values occurred in all four quarters for one or more years.

You can use the browse command to examine your data and see what variables have missing values in the excluded observations and decide whether that is what you expect, or if something went wrong in your data preparation. For example

browse if acode==6000005

will allow you to look at the 20 observations for authority 6000006 and see which values of which variables are missing.

Also the misstable command can help you get an overall understanding of the missing values in your dataset. You should read the output of help misstable for full details, but of particular use are

misstable summarize
misstable patterns

I hope this helps you understand your data better.