Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Updated -missings- available from SSC

    Thanks as always to @Kit Baum, an updated copy of the missings files is now available on SSC. Type

    Code:
    ssc install missings
    or

    Code:
    ssc install missings, update

    missings was first published in the Stata Journal in 2015 and updated there several times since.

    The version on SSC was, until just now, very out-of-date,

    As mentioned elsewhere, the command name missings is I think defensible -- as a concise and informative hint at what the command is all about -- but yet unfortunate insofar as a search for missings will yield much else -- which may be useful and even interesting, but not if your very specific query is about this command.

    dm0085 is an otherwise unpredictable search term tagging only the original paper and its updates (and their associated files).

    Code:
    . search dm0085, entry
    
    Search of official help files, FAQs, Examples, and Stata Journals
    
    SJ-26-3 dm0085_4  . . . . . . . . . . . . . . . . Software update for missings
            (help missings if installed)  . . . . . . . . . . . . . . .  N. J. Cox
            Q3/26   SJ 26(3):537
            new option added
    
    SJ-23-2 dm0085_3  . . . . . . . . . . . . . . . . Software update for missings
            (help missings if installed)  . . . . . . . . . . . . . . .  N. J. Cox
            Q2/23   SJ 23(2):595--596
            most important change is the addition of subcommand breakdown
    
    SJ-20-4 dm0085_2  . . . . . . . . . . . . . . . . Software update for missings
            (help missings if installed)  . . . . . . . . . . . . . . .  N. J. Cox
            Q4/20   SJ 20(4):1028--1030
            sorting has been extended for missings report
    
    SJ-17-3 dm0085_1  . . . . . . . . . . . . . . . . Software update for missings
            (help missings if installed)  . . . . . . . . . . . . . . .  N. J. Cox
            Q3/17   SJ 17(3):779
            identify() and sort options have been added
    
    SJ-15-4 dm0085  Speaking Stata: A set of utilities for managing missing values
            (help missings if installed)  . . . . . . . . . . . . . . .  N. J. Cox
            Q4/15   SJ 15(4):1174--1185
            provides command, missings, as a replacement for, and extension
            of, previous commands nmissing and dropmiss
    From that you will see that the latest update in the Stata Journal was in the most recent issue, released last month.

    Another update is needed, however. In the recent thread

    https://www.statalist.org/forums/for...missing-values

    Choonjoo Lee found two more bugs. See his posts #12 and #19 in that thread and my replies.

    I have submitted an update to the Stata Journal, but I doubt it will be published until the issue 27(1), which will appear about March. That is another reason for updating the version on SSC, which is now the latest version downloadable.

    Getting a systematic overview of the pattern of missing values in a dataset is a basic and vital need, and it's hardly surprising that there are many commands to do something in that line, both official (included in StataCorp's distributed software) and community-contributed (user-written). Several commands were mentioned in the thread linked just above, and there may well be others.

    Some people here are behind ferocious and fearsome firewalls, and unable to install extras, easily or at all, even after pleading with local IT personnel. So it's worth underlining that official Stata includes misstable, which has four subcommands. The most basic is exemplified by


    Code:
    . webuse nlswork
    (National Longitudinal Survey of Young Women, 14-24 years old in 1968)
    
    . misstable summarize
                                                                   Obs<.
                                                    +------------------------------
                   |                                | Unique
          Variable |     Obs=.     Obs>.     Obs<.  | values        Min         Max
      -------------+--------------------------------+------------------------------
               age |        24              28,510  |     33         14          46
               msp |        16              28,518  |      2          0           1
           nev_mar |        16              28,518  |      2          0           1
             grade |         2              28,532  |     19          0          18
          not_smsa |         8              28,526  |      2          0           1
            c_city |         8              28,526  |      2          0           1
             south |         8              28,526  |      2          0           1
          ind_code |       341              28,193  |     12          1          12
          occ_code |       121              28,413  |     13          1          13
             union |     9,296              19,238  |      2          0           1
            wks_ue |     5,704              22,830  |     61          0          76
            tenure |       433              28,101  |    270          0    25.91667
             hours |        67              28,467  |     85          1         168
          wks_work |       703              27,831  |    105          0         104
      -----------------------------------------------------------------------------

    Here is a comparison with missings breakdown.

    Code:
    . missings breakdown
    
    Checking missings in all variables:
    15082 observations / 28534 with missing values
    
      +-----------------------------+
      |            # missing      . |
      |-----------------------------|
      |      age          24     24 |
      |      msp          16     16 |
      |  nev_mar          16     16 |
      |    grade           2      2 |
      | not_smsa           8      8 |
      |-----------------------------|
      |   c_city           8      8 |
      |    south           8      8 |
      | ind_code         341    341 |
      | occ_code         121    121 |
      |    union        9296   9296 |
      |-----------------------------|
      |   wks_ue        5704   5704 |
      |   tenure         433    433 |
      |    hours          67     67 |
      | wks_work         703    703 |
      +-----------------------------+
    If we fool around with that dataset and add examples of extending missing values and empty strings (also regarded as missing by Stata)
    then a reissued missings breakdown grows the table:

    Code:
    . gen foo = .a
    (28,534 missing values generated)
    
    . gen bar = ""
    (28,534 missing values generated)
    
    . missings breakdown
    
    Checking missings in all variables:
    28534 observations / 28534 with missing values
    
      +---------------------------------------------+
      |            # missing   empty      .      .a |
      |---------------------------------------------|
      |      age          24       .     24       0 |
      |      msp          16       .     16       0 |
      |  nev_mar          16       .     16       0 |
      |    grade           2       .      2       0 |
      | not_smsa           8       .      8       0 |
      |---------------------------------------------|
      |   c_city           8       .      8       0 |
      |    south           8       .      8       0 |
      | ind_code         341       .    341       0 |
      | occ_code         121       .    121       0 |
      |    union        9296       .   9296       0 |
      |---------------------------------------------|
      |   wks_ue        5704       .   5704       0 |
      |   tenure         433       .    433       0 |
      |    hours          67       .     67       0 |
      | wks_work         703       .    703       0 |
      |      foo       28534       .      0   28534 |
      |---------------------------------------------|
      |      bar       28534   28534      .       . |
      +---------------------------------------------+
    How does misstable cope with that? You need to add the option all to get a mention of string variables:


    Code:
    . misstable summarize
                                                                   Obs<.
                                                    +------------------------------
                   |                                | Unique
          Variable |     Obs=.     Obs>.     Obs<.  | values        Min         Max
      -------------+--------------------------------+------------------------------
               age |        24              28,510  |     33         14          46
               msp |        16              28,518  |      2          0           1
           nev_mar |        16              28,518  |      2          0           1
             grade |         2              28,532  |     19          0          18
          not_smsa |         8              28,526  |      2          0           1
            c_city |         8              28,526  |      2          0           1
             south |         8              28,526  |      2          0           1
          ind_code |       341              28,193  |     12          1          12
          occ_code |       121              28,413  |     13          1          13
             union |     9,296              19,238  |      2          0           1
            wks_ue |     5,704              22,830  |     61          0          76
            tenure |       433              28,101  |    270          0    25.91667
             hours |        67              28,467  |     85          1         168
          wks_work |       703              27,831  |    105          0         104
               foo |              28,534         0  |      0          .           .
      -----------------------------------------------------------------------------
    
    . help misstable
    
    . misstable summarize, all
                                                                   Obs<.
                                                    +------------------------------
                   |                                | Unique
          Variable |     Obs=.     Obs>.     Obs<.  | values        Min         Max
      -------------+--------------------------------+------------------------------
            idcode |                        28,534  |   >500          1        5159
              year |                        28,534  |     15         68          88
          birth_yr |                        28,534  |     14         41          54
               age |        24              28,510  |     33         14          46
              race |                        28,534  |      3          1           3
               msp |        16              28,518  |      2          0           1
           nev_mar |        16              28,518  |      2          0           1
             grade |         2              28,532  |     19          0          18
          collgrad |                        28,534  |      2          0           1
          not_smsa |         8              28,526  |      2          0           1
            c_city |         8              28,526  |      2          0           1
             south |         8              28,526  |      2          0           1
          ind_code |       341              28,193  |     12          1          12
          occ_code |       121              28,413  |     13          1          13
             union |     9,296              19,238  |      2          0           1
            wks_ue |     5,704              22,830  |     61          0          76
           ttl_exp |                        28,534  |   >500          0    28.88461
            tenure |       433              28,101  |    270          0    25.91667
             hours |        67              28,467  |     85          1         168
          wks_work |       703              27,831  |    105          0         104
           ln_wage |                        28,534  |   >500          0    5.263916
               foo |              28,534         0  |      0          .           .
               bar | (string variable)              |
      -----------------------------------------------------------------------------
    Either command might do what you want, or you might prefer one to the other.

    I don't dismiss the problem of counting unique values, although I strongly prefer the word distinct to unique. And that is what the command distinct is all about, but that is a different story.

    Code:
    . search dm0042, entry
    
    Search of official help files, FAQs, Examples, and Stata Journals
    
    SJ-23-4 dm0042_5  . . . . . . . . . . . . . . . . Software update for distinct
            (help distinct, distinctgen if installed)  N. J. Cox and G. M. Longton
            Q4/23   SJ 23(4):1096
            comments out (and thus removes) a call to clear Mata at the
            close of work, which was frustrating some other projects
            also using Mata
    
    SJ-23-2 dm0042_4  . . . . . . . . . . . . . . . . Software update for distinct
            (help distinct, distinctgen if installed)  N. J. Cox and G. M. Longton
            Q2/23   SJ 23(2):595--596
            most important change is addition of distinctgen command
    
    SJ-20-4 dm0042_3  . . . . . . . . . . . . . . . . Software update for distinct
            (help distinct if installed)  . . . . . .  N. J. Cox and G. M. Longton
            Q4/20   SJ 20(4):1028--1030
            sort() option has been added
    
    SJ-15-3 dm0042_2  . . . . . . . . . . . . . . . . Software update for distinct
            (help distinct if installed)  . . . . . .  N. J. Cox and G. M. Longton
            Q3/15   SJ 15(3):899
            improved table format and display of large numbers of
            observations
    
    SJ-12-2 dm0042_1  . . . . . . . . . . . . . . . . Software update for distinct
            (help distinct if installed)  . . . . . .  N. J. Cox and G. M. Longton
            Q2/12   SJ 12(2):352
            options added to restrict output to variables with a minimum
            or maximum of distinct values
    
    SJ-8-4  dm0042  . . . . . . . . . . . .  Speaking Stata: Distinct observations
            (help distinct if installed)  . . . . . .  N. J. Cox and G. M. Longton
            Q4/08   SJ 8(4):557--568
            shows how to answer questions about distinct observations
            from first principles; provides a convenience command

    Last edited by Nick Cox; Today, 02:45.
Working...
X