mixed model postestimation using predict

[cherry singhal]

Hello,

I am using Stata 14. I have a panel dataset - firms and years. I am running a mixed model which includes both fixed and random effects.
My total sample size is 52,801. The data set has missing values so entire dataset is not used when I run mixed model. The mixed model output shows the #observations used is 40595.

I ran the "predict ebX1,reffects" command to obtain BLUPS for random component of independent variable X1. Now since the mixed model analysis used 40,595 observations out of the total of 52,801 observations, I assumed that BLUPS would be created for exactly 40,595 observations.
However, when I summarize ebX1 (command used: summarize ebX1), I get #obs =51032 in the summary table. This #obs is what I do not understand. So now if I run the command as "summ ebX1 if e(sample)" then I get summary statistics for the 40,595 observations. So I am able to get the correct summary statistics but what I do not understand is why were BLUPS calculated for 51,032 (>40,595) observations in the first place?
Also, if I now want to say plot a histogram of the BLUPS using command "hist ebX1, freq", the graph is plotted for 51,032 observations which are incorrect for me. And I can not use "if e(sample)" with the histogram command. I can create an indicator variable to mark the 40,595 observations and go from there but I am curious why BLUPS were calculated for more observations than that.

thanks.

[cherry singhal]

Thank you Clyde for the elaborate explanation!
Thank you Joseph for the hands on example, it so helps!

[Joseph Coveney]

Take a look at the patterns of missing values in your dataset. (help missing and then scroll down to "Useful commands".) You can still get a BLUP under some patterns of missing data for a cluster. See the illustration of that below, which shows examples of circumstances under which you will and under which you won't get a prediction.

.ÿversionÿ14.2

.ÿ
.ÿclearÿ*

.ÿsetÿmoreÿoff

.ÿsetÿseedÿ1372490

.ÿ
.ÿquietlyÿsetÿobsÿ200

.ÿgenerateÿintÿpidÿ=ÿ_n

.ÿgenerateÿdoubleÿuÿ=ÿrnormal()

.ÿ
.ÿquietlyÿexpandÿ2

.ÿbysortÿpid:ÿgenerateÿbyteÿtimÿ=ÿ_n

.ÿ
.ÿgenerateÿdoubleÿrspÿ=ÿuÿ+ÿrnormal()

.ÿ
.ÿquietlyÿreplaceÿrspÿ=ÿ.ÿinÿ1

.ÿquietlyÿreplaceÿtimÿ=ÿ.ÿinÿ3

.ÿquietlyÿreplaceÿtimÿ=ÿ.ÿinÿ5/6

.ÿ
.ÿmixedÿrspÿi.timÿ||ÿpid:ÿ,ÿnolrtestÿnolog

Mixed-effectsÿMLÿregressionÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿobsÿÿÿÿÿ=ÿÿÿÿÿÿÿÿ396
Groupÿvariable:ÿpidÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿNumberÿofÿgroupsÿÿ=ÿÿÿÿÿÿÿÿ199

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿObsÿperÿgroup:
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿminÿ=ÿÿÿÿÿÿÿÿÿÿ1
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿavgÿ=ÿÿÿÿÿÿÿÿ2.0
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿmaxÿ=ÿÿÿÿÿÿÿÿÿÿ2

ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿWaldÿchi2(1)ÿÿÿÿÿÿ=ÿÿÿÿÿÿÿ5.56
Logÿlikelihoodÿ=ÿ-675.86184ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿProbÿ>ÿchi2ÿÿÿÿÿÿÿ=ÿÿÿÿÿ0.0184

------------------------------------------------------------------------------
ÿÿÿÿÿÿÿÿÿrspÿ|ÿÿÿÿÿÿCoef.ÿÿÿStd.ÿErr.ÿÿÿÿÿÿzÿÿÿÿP>|z|ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-------------+----------------------------------------------------------------
ÿÿÿÿÿÿÿ2.timÿ|ÿÿÿ.2412021ÿÿÿ.1023253ÿÿÿÿÿ2.36ÿÿÿ0.018ÿÿÿÿÿ.0406481ÿÿÿÿ.4417561
ÿÿÿÿÿÿÿ_consÿ|ÿÿÿ.0155886ÿÿÿ.1017302ÿÿÿÿÿ0.15ÿÿÿ0.878ÿÿÿÿ-.1837989ÿÿÿÿ.2149761
------------------------------------------------------------------------------

------------------------------------------------------------------------------
ÿÿRandom-effectsÿParametersÿÿ|ÿÿÿEstimateÿÿÿStd.ÿErr.ÿÿÿÿÿ[95%ÿConf.ÿInterval]
-----------------------------+------------------------------------------------
pid:ÿIdentityÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ|
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(_cons)ÿ|ÿÿÿ1.009803ÿÿÿ.1621702ÿÿÿÿÿÿ.7371185ÿÿÿÿ1.383363
-----------------------------+------------------------------------------------
ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿvar(Residual)ÿ|ÿÿÿÿ1.03397ÿÿÿ.1041779ÿÿÿÿÿÿ.8486823ÿÿÿÿ1.259711
------------------------------------------------------------------------------

.ÿ
.ÿpredictÿdoubleÿebe,ÿreffects
(2ÿmissingÿvaluesÿgenerated)

.ÿ
.ÿlistÿinÿ1/6,ÿnoobsÿsepby(pid)

ÿÿ+-------------------------------------------------+
ÿÿ|ÿpidÿÿÿÿÿÿÿÿÿÿÿÿuÿÿÿtimÿÿÿÿÿÿÿÿÿÿrspÿÿÿÿÿÿÿÿÿebeÿ|
ÿÿ|-------------------------------------------------|
ÿÿ|ÿÿÿ1ÿÿÿÿ.41822853ÿÿÿÿÿ1ÿÿÿÿÿÿÿÿÿÿÿÿ.ÿÿÿ1.0644975ÿ|
ÿÿ|ÿÿÿ1ÿÿÿÿ.41822853ÿÿÿÿÿ2ÿÿÿÿ2.4112619ÿÿÿ1.0644975ÿ|
ÿÿ|-------------------------------------------------|
ÿÿ|ÿÿÿ2ÿÿÿ-.32213358ÿÿÿÿÿ.ÿÿÿ-.80480298ÿÿÿÿ.0886456ÿ|
ÿÿ|ÿÿÿ2ÿÿÿ-.32213358ÿÿÿÿÿ2ÿÿÿÿ.43620344ÿÿÿÿ.0886456ÿ|
ÿÿ|-------------------------------------------------|
ÿÿ|ÿÿÿ3ÿÿÿÿÿ.6285637ÿÿÿÿÿ.ÿÿÿÿÿ1.388014ÿÿÿÿÿÿÿÿÿÿÿ.ÿ|
ÿÿ|ÿÿÿ3ÿÿÿÿÿ.6285637ÿÿÿÿÿ.ÿÿÿÿ.43723975ÿÿÿÿÿÿÿÿÿÿÿ.ÿ|
ÿÿ+-------------------------------------------------+

.ÿ
.ÿexit

endÿofÿdo-file


.

[Clyde Schechter]

Well, you don't say which variable defines your higher level in the model, but let's assume for the sake of discussion that it's firm.

Remember that the random effect at the firm level is a constant attribute of the firm; it does not vary among observations within firm. Now, if a firm has any observations in the estimation sample at all, a firm-level random effect can be calculated for that firm. And Stata does that and records that same value in every observation for that firm, even if that observation is not, itself, part of the estimation sample. So the only circumstance under which Stata will not calculate the random effect in an observation is if that observation's firm has no observations in the estimation sample. I'm guessing that this doesn't happen very often in your data (although apparently it does happen occasionally since you don't get 52,801 observations for ebX1.)

Actually, if you want to do a histogram of ebX1, even restricting to e(sample) is not appropriate because each firm is then counted as many times as it appears in the estimation sample. The actual distribution of ebX1 has to be viewed at the firm level. So something like this:

egen flag = tag(firm)
histogram ebX1 if flag

In fact pretty much anything you do with ebX1 should be conditioned on -if flag- so that each firm is counted (at most) once.