Data Cleaning

[aima khan]

I am trying to run panel regression. Variables in sample have different number of observations starting from 17000 to 24000. I want to use maximum no.of observations but have the same number of observations in all the tested models. I tried drop command and it brings observations to a few thousand only. How do I ensure the same number of observation accross models while ensuring maximum numbers?

[Maarten Buis]

What you are running into is called missing values. This is a normal and very annoying part of real data. If this is survey data, then this could represent people not wanting to answer a certain question, or during the interview something happened and the respondent breaks off the interview. It can also happen by design. You could ask if someone is in the labor market and than only ask those who are in the labor market what their wage is. All those who are outside the labor market will now have a missing value. Or the survey would be too long, so one group of respondents gets one set of questions, and another set of respondents gets another set of questions. Similar problems can occur with register data (do you really think that civil servants never take short cuts / make errors?) or process generated data (is there never down-time anywhere?). Real data is unfortunately messy.

So when dealing with this you need to spend a lot of time with the code book, the questionnaire, and other documentation that came with your data to figure out why those missing values occur. Based on that you start making decisions: do these people belong in your analysis sample? Can you recover their value? Do you really need a particularly troublesome variable? So there is not really a single technique.

You may come across the term multiple imputation as a way of dealing with missing values. This is a step one could take after the preparations I talked about above are done. However this is a pretty advanced technique and a lot can go wrong and it is hard to diagnose. So this is not a technique I would recommend for people for someone who asked the question you just asked. (There is nothing wrong with your question, everybody has to start somewhere, but it does quite clearly indicate that you are at the beginning of your data analysis journey)

[aima khan]

Hi, many thanks for your detailed response. The data is secondary and financial, so perhaps a lot of steps that you mentioned above do no apply here. The data is usually downloaded as is, and cleaned for using.

[Maarten Buis]

Data does not fall out the sky. You need to know how it was collected, and use that information to find out why those missing values exist. The data you downloaded should come with documentation. So studying that documentation, going back to the data, going back to the documentation, etc. until you understand what is happening with your data.This is the one step I suggested you should take in my first post, and that is still the step I recommend.

[Dirk Enzmann]

I am reading

I want to use maximum no.of observations but have the same number of observations in all the tested models

and

How do I ensure the same number of observation accross models while ensuring maximum numbers?

as "How to make sure that different regressions models use the same cases even if in some models some cases are missing?"

If you would accept casewise deletion of missing values (instead of missing imputations) per regression model, you can use a marker variable created by -mark- and modified by -markout-. Even better is to use -markobs- (SSC, see "New on SSC: markobs"). Having defined the marker variable to code the cases to use with -markobs-, you can call -markobs- (repeatedly) to modify the marker variable setting cases with missing values of the (list of) variables (cumulative) to 0. Subsequently use -if <marker variable>- in your regression models to exclude cases with value 0 of the marker variable.

Example:

which esttab                // for tables of regression models 
if _rc ssc install estout   // install if necessary 
which markobs               // to create and modify a marker variable 
if _rc ssc install markobs  // install if necessary 
 
* ------------------------------------------------------------------------------ 
sysuse auto, clear 
 
sum weight price rep78     // note: rep78 has only 69 obs, all other have 74 
 
markobs touse         // first call of markobs to define marker variable "touse" 
markobs touse price rep78  // collection of predictors for all regression models 
 
reg weight price if touse  // 1st model, note that price and weight have 74 valid obs 
est sto m1                 // store estimates of model 1 
reg weight price rep78 if touse  // 2nd model, rep78 has only 69 valid obs 
est sto m2 
 
esttab m1 m2               // both (!) models have the same 69 obs

yields

. which esttab                // for tables of regression models
/home/enzmann/ado/plus/e/esttab.ado
*! version 2.1.4  13apr2026  Ben Jann
*! wrapper for estout

. if _rc ssc install estout   // install if necessary
checking estout consistency and verifying not already installed...
all files already exist and are up to date.

. which markobs               // to create and modify a marker variable
/home/enzmann/ado/plus/m/markobs.ado
*! version 1.0.1  13aug2025

. if _rc ssc install markobs  // install if necessary

. * ------------------------------------------------------------------------------
. sysuse auto, clear
(1978 automobile data)

. sum weight price rep78     // note: rep78 has only 69 obs, all other have 74

    Variable |        Obs        Mean    Std. dev.       Min        Max
-------------+---------------------------------------------------------
      weight |         74    3019.459    777.1936       1760       4840
       price |         74    6165.257    2949.496       3291      15906
       rep78 |         69    3.405797    .9899323          1          5

. markobs touse         // first call of markobs to define marker variable "touse"
. markobs touse price rep78  // collection of predictors for all regression models

. reg weight price if touse  // 1st model, note that price and weight have 74 valid obs

      Source |       SS           df       MS      Number of obs   =        69
-------------+----------------------------------   F(1, 67)        =     28.73
       Model |  12829194.2         1  12829194.2   Prob > F        =    0.0000
    Residual |  29916521.7        67   446515.25   R-squared       =    0.3001
-------------+----------------------------------   Adj R-squared   =    0.2897
       Total |  42745715.9        68   628613.47   Root MSE        =    668.22

------------------------------------------------------------------------------
      weight | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
       price |    .149138   .0278232     5.36   0.000     .0936027    .2046733
       _cons |   2115.421    188.979    11.19   0.000     1738.217    2492.624
------------------------------------------------------------------------------

. est sto m1                 // store estimates of model 1
. reg weight price rep78 if touse  // 2nd model, rep78 has only 69 valid obs

      Source |       SS           df       MS      Number of obs   =        69
-------------+----------------------------------   F(2, 66)        =     28.49
       Model |  19804009.2         2   9902004.6   Prob > F        =    0.0000
    Residual |  22941706.8        66  347601.617   R-squared       =    0.4633
-------------+----------------------------------   Adj R-squared   =    0.4470
       Total |  42745715.9        68   628613.47   Root MSE        =    589.58

------------------------------------------------------------------------------
      weight | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
-------------+----------------------------------------------------------------
       price |   .1498586   .0245493     6.10   0.000     .1008444    .1988728
       rep78 |  -323.5309   72.22545    -4.48   0.000    -467.7336   -179.3281
       _cons |   3212.872   296.3529    10.84   0.000     2621.184     3804.56
------------------------------------------------------------------------------

. est sto m2
. esttab m1 m2               // both (!) models have the same 69 obs

--------------------------------------------
                      (1)             (2)   
                   weight          weight   
--------------------------------------------
price               0.149***        0.150***
                   (5.36)          (6.10)   

rep78                              -323.5***
                                  (-4.48)   

_cons              2115.4***       3212.9***
                  (11.19)         (10.84)   
--------------------------------------------
N                      69              69   
--------------------------------------------
t statistics in parentheses
* p<0.05, ** p<0.01, *** p<0.001

end of do-file