-vl rebuild- : variable not found

[Dirk Enzmann]

When using -vl rebuild- to rebuild vl macros (before saving a Stata dataset or after opening a Stata dataset that has been saved previously with vl macros defined) I receive the error message that a variable is not found.

I can't use -dataex- here to create a reproducible example, hence I simply describe what I did an show respective Stata output.

1) I did use -vl- commands to set and create system and user vl macros. Running -vl rebuild- shows the following:

. vl rebuild
Rebuilding vl macros ...

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vldummy        |     204   0/1 variables
  $vlcategorical  |     138   categorical variables
  $vlcontinuous   |      91   continuous variables
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $preval         |      64   prevalence variables
  $mresp          |     102   multiple response variables
  $binary         |      33   binary variables (not preval or mresp)
  $scale          |      92   Likert scale variables (not multnom)
  $multnom        |      32   multinomial variables (not scale)
  $count          |      46   count variables
  $contin         |       9   (quasi) continuous variables
  $meta           |      24   metadata variables
  $paradat        |      31   paradata variables
  $IS_natmod      |       2   national module variables
-------------------------------------------------------------------------------

2) After that I create several new variables. At the end I use -vl set [varlist]- to add the new variables to the vl system macros followed by -vl modify- and -vl move- to add these variables to vl user macros and to move them into the vl system macro vlcontinuous:

vl set   truancy_ia truancy_ie vrobb_ia vrobb_ie vrobb_na vrobb_ne vassa_ia ///
         vassa_ie vassa_na vassa_ne vthef_ia vthef_ie vthef_na vthef_ne ///
         vhate_ia vhate_ie vhate_na vhate_ne vcyth_ia vcyth_ie vcyth_na ///
         vcyth_ne vshin_ia vshin_ie vshin_na vshin_ne vshin_ta vshin_te ///
         vcyhc_ia vcyhc_ie vcyhc_na vcyhc_ne vcyhc_ta vcyhc_te vphpu_ia ///
         vphpu_ie vpama_ia vpama_ie clfrnum_a clfrnum_e clfrethn_a ///
         clfrethn_e delpsln_a delpsln_e delpbun_a delpbun_e delpasn_a ///
         delpasn_e delpsin_a delpsin_e delphan_a delphan_e graf_ia graf_ie ///
         vand_ia vand_ie shop_ia shop_ie burg_ia burg_ie cart_ia cart_ie ///
         exto_ia exto_ie weap_ia weap_ie gfig_ia gfig_ie aslt_ia aslt_ie ///
         drud_ia drud_ie shin_ia shin_ie cyhc_ia cyhc_ie cyfr_ia cyfr_ie ///
         hack_ia hack_ie hacksuc_ia hacksuc_ie hackid_ia hackid_ie, list
         
vl modify count = count + ///
        (truancy_ia truancy_ie vrobb_ia vrobb_ie vrobb_na vrobb_ne vassa_ia ///
         vassa_ie vassa_na vassa_ne vthef_ia vthef_ie vthef_na vthef_ne ///
         vhate_ia vhate_ie vhate_na vhate_ne vcyth_ia vcyth_ie vcyth_na ///
         vcyth_ne vshin_ia vshin_ie vshin_na vshin_ne vshin_ta vshin_te ///
         vcyhc_ia vcyhc_ie vcyhc_na vcyhc_ne vcyhc_ta vcyhc_te vphpu_ia ///
         vphpu_ie vpama_ia vpama_ie clfrnum_a clfrnum_e clfrethn_a ///
         clfrethn_e delpsln_a delpsln_e delpbun_a delpbun_e delpasn_a ///
         delpasn_e delpsin_a delpsin_e delphan_a delphan_e graf_ia graf_ie ///
         vand_ia vand_ie shop_ia shop_ie burg_ia burg_ie cart_ia cart_ie ///
         exto_ia exto_ie weap_ia weap_ie gfig_ia gfig_ie aslt_ia aslt_ie ///
         drud_ia drud_ie shin_ia shin_ie cyhc_ia cyhc_ie cyfr_ia cyfr_ie ///
         hack_ia hack_ie hacksuc_ia hacksuc_ie hackid_ia hackid_ie)
       
vl move ($count) vlcontinuous

There is no error message and a subsequent -vl dir- shows the follwing result:

. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vldummy        |     204   0/1 variables
  $vlcategorical  |     138   categorical variables
  $vlcontinuous   |     175   continuous variables
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $preval         |      64   prevalence variables
  $mresp          |     102   multiple response variables
  $binary         |      33   binary variables (not preval or mresp)
  $scale          |      92   Likert scale variables (not multnom)
  $multnom        |      32   multinomial variables (not scale)
  $count          |     130   count variables
  $contin         |       9   (quasi) continuous variables
  $meta           |      24   metadata variables
  $paradat        |      31   paradata variables
  $IS_natmod      |       2   national module variables
-------------------------------------------------------------------------------

3) However, when running -vl build- now I receive the following error message:

. vl rebuild
Rebuilding vl macros ...
variable truancy_ia not found
    Run vl rebuild to rebuild vl macros.
r(111);

although the variable truancy_ia exists:

. sum truancy_ia

    Variable |        Obs        Mean    Std. dev.       Min        Max
-------------+---------------------------------------------------------
  truancy_ia |        395    6.103797    7.674988          0         43

Is there something I did not understand when using the -vl- commands? Is this a Stata bug?

[Dirk Enzmann]

I now understand what happens (the problem is caused by creating new variables via -clonevar-) and here is a solution (or work-around):

1) The following steps will reproduce the problem described in #1 using our famous auto dataset:

. sysuse auto, clear
(1978 automobile data)

.
. * ------------------------------------------------------------------------------
. * Create vl system and user macros containing the variable price:
. qui vl set price

. qui vl create contin = (price)

. qui vl move ($contin) vlcontinuous

. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       1   continuous variable
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $contin         |       1   variable
-------------------------------------------------------------------------------

. vl list

------------------------------------------------
Variable | Macro           Values         Levels
---------+--------------------------------------
   price | $vlcontinuous   integers >=0       74
   price | $contin         integers >=0       74
------------------------------------------------

.
. * Create the new variable price_new as a clone of variable price:
. clonevar price_new = price

.
. * This variable seems to be a variable in the vl system and user macros:
. vl list (price*)

-------------------------------------------------
 Variable | Macro           Values         Levels
----------+--------------------------------------
    price | $vlcontinuous   integers >=0       74
    price | $contin         integers >=0       74
price_new | $vlcontinuous   integers >=0       74
price_new | $contin         integers >=0       74
-------------------------------------------------

.
. * However, it is not yet defined in the vl system and user macros:
. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       1   continuous variable
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $contin         |       1   variable
-------------------------------------------------------------------------------

. vl list

------------------------------------------------
Variable | Macro           Values         Levels
---------+--------------------------------------
   price | $vlcontinuous   integers >=0       74
   price | $contin         integers >=0       74
------------------------------------------------

.
. * You can add price_new to the vl system and user macros:
. vl set price_new

Added variables
-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       0   continuous variables
  $vluncertain    |       1   perhaps continuous, perhaps categorical variable
  $vlother        |       0   all missing or constant variables
-------------------------------------------------------------------------------
Notes

      1. Review contents of vlcategorical and vlcontinuous to ensure they are correct.
         Type vl list vlcategorical and type vl list vlcontinuous.

      2. If there are any variables in vluncertain, you can reallocate them to
         vlcategorical, vlcontinuous, or vlother.  Type vl list vluncertain.

      3. Use vl move to move variables among classifications.  For example, type
         vl move (x50 x80) vlcontinuous to move variables x50 and x80 to the continuous
         classification.

      4. vlnames are global macros.  Type the vlname without the leading dollar sign ($)
         when using vl commands.  Example: vlcategorical not $vlcategorical.  Type the
         dollar sign with other Stata commands to get a varlist.

. vl modify contin = contin + (price_new)
note: 1 variable added to $contin.

. vl move ($contin) vlcontinuous
note: 2 variables specified and 1 variable moved;
      the other variable was already in $vlcontinuous.

------------------------------
Macro          # Added/Removed
------------------------------
$vlcategorical               0
$vlcontinuous                1
$vluncertain                -1
$vlother                     0
------------------------------

.
. * Now -vl dir- and -vl list- will show the new variable:
. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       2   continuous variables
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $contin         |       2   variables
-------------------------------------------------------------------------------

. vl list

-------------------------------------------------
 Variable | Macro           Values         Levels
----------+--------------------------------------
    price | $vlcontinuous   integers >=0       74
price_new | $vlcontinuous   integers >=0       74
    price | $contin         integers >=0       74
price_new | $contin         integers >=0       74
-------------------------------------------------

.
. * However, -vl rebuild- will produce an error message and makes the vl macros unusable:
. vl rebuild
Rebuilding vl macros ...
variable price_new not found
    Run vl rebuild to rebuild vl macros.
r(111);

2) Because after -clonevar- the new variables are "known" to vl but not yet defined in the vl system (and user) macros, you can use -vl rebuild- to "repair" this issue:

. * ==============================================================================
.// As an alternative (work-around?), you can use -vl rebuild- immediately after
. // having created the new variable price_new as a clone of the variable price:
.
. vl clear

. sysuse auto, clear
(1978 automobile data)

.
.* ------------------------------------------------------------------------------
. * Create vl system and user macros containing the variable price:
. qui vl set price

. qui vl create contin = (price)

. qui vl move ($contin) vlcontinuous

. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       1   continuous variable
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $contin         |       1   variable
-------------------------------------------------------------------------------

. vl list

------------------------------------------------
Variable | Macro           Values         Levels
---------+--------------------------------------
   price | $vlcontinuous   integers >=0       74
   price | $contin         integers >=0       74
------------------------------------------------

.
. * Create the new variable price_new as a clone of variable price:
. clonevar price_new = price

.
. * price_new is listed in the vl system and user macros but not in -vl dir-:
. vl list (price*)

-------------------------------------------------
 Variable | Macro           Values         Levels
----------+--------------------------------------
    price | $vlcontinuous   integers >=0       74
    price | $contin         integers >=0       74
price_new | $vlcontinuous   integers >=0       74
price_new | $contin         integers >=0       74
-------------------------------------------------

. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       1   continuous variable
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $contin         |       1   variable
-------------------------------------------------------------------------------

.
. * But -vl rebuild- at this point will include price_new into the vl system and user macros:
. vl rebuild
Rebuilding vl macros ...

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       2   continuous variables
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $contin         |       2   variables
-------------------------------------------------------------------------------

.
.* Check that price_new it is actually part of the vl sytem and user macros:
. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       2   continuous variables
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $contin         |       2   variables
-------------------------------------------------------------------------------

. vl list

-------------------------------------------------
 Variable | Macro           Values         Levels
----------+--------------------------------------
    price | $vlcontinuous   integers >=0       74
price_new | $vlcontinuous   integers >=0       74
    price | $contin         integers >=0       74
price_new | $contin         integers >=0       74
-------------------------------------------------

.
. * Now -vl rebuild- will do no harm anymore:
. vl rebuild
Rebuilding vl macros ...

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       2   continuous variables
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $contin         |       2   variables
-------------------------------------------------------------------------------

The issue of using new variables created via -clonevar- as described in (1) is not documented in the help to -vl- (this should be "repaired" if it is an intended feature of -vl-). However, I believe that it is unintended and you can use the work-around using -vl rebuild- immediately after having created new variables via -clonevar-.

[daniel klein]

The issue of using new variables created via -clonevar- as described in (1) is not documented in the help to -vl- (this should be "repaired" if it is an intended feature of -vl-). However, I believe that it is unintended and you can use the work-around using -vl rebuild- immediately after having created new variables via -clonevar-.

FWIW, I think this is an unintended (but expected) side-effect of clonevar copying all characteristics of a variable. With clonevar what is useful in some scenarios (e.g., copying variable and value labels in all languages) might prove problematic in others (e.g., vl, mi set, etc.).

[Dirk Enzmann]

Thank you daniel klein for alerting us to the general side effects of using -clonevar- , especially for -mi set- . If would be useful to know which other commands could be affected by this, for example -xtset-.

[daniel klein]

I[t] would be useful to know which other commands could be affected by this, for example -xtset-.

I do not believe that xtset defines characteristics at the variable level; I would not expect any side effects there. In general, I think that official Stata commands mostly define characteristics at the dataset level.* Some commands define characteristics at the variable level, e.g., destring, but I do not think that those are used by other commands; so, again, probably no side effects there.


* Characteristics at the dataset level have names _dta[]. A bit off-topic: notice that Stata allows _dta as a variable name, too. If you drop a variable with that name from the dataset, you also drop all characteristics defined at the dataset level (mi set, xtset, label languages, etc.). I think _dta should not be allowed as a variable name.

[Dirk Enzmann]

The most important lesson to me is that using -clonevar- may have unintended side-effects because -clonevar- also clones the characteristics of variables. Hence, one should carefully inspect the characteristics of variables created by -clonevar- (and modify or remove them if necessary).

In the lengthy demonstration below I show how the removal of the characteristics of variables created by using -clonevar- also prevents the problem I experienced in #1 (I also rename the value label of the new variable because I want that this variable has its own value label defined):

. * Demonstrate the use -vl- with new variables created by -clonevar-:
.
. cap which elabel

. if _rc ssc install elabel  // install -elabel- if necessary

.
. cap program drop renlab

. program renlab
  1.    // Rename value labels to name of variables (with labels defined):
.    syntax [varlist]
  2.    local nvars : word count `varlist'
  3.    qui {
  4.       if `nvars' > 1 {
  5.          elabel dir
  6.          local ll = r(names)
  7.       }
  8.       else {
  9.          cap elabel list (`varlist')
 10.          if !_rc local ll = r(name)
 11.       }
 12.       foreach lnam in `ll' {
 13.          elabel list `lnam', var
 14.          if "`r(varlist)'" != "" {
 15.             foreach v of varlist `r(varlist)' {
 16.                elabel list (`v')
 17.                elabel copy `v':`r(name)' `v', replace
 18.             label val `v' `v'
 19.             }
 20.          }
 21.       }
 22.    }
 23. end

.
. cap program drop remchar

. program define remchar
  1.    // remove all characteristics from a variable:
.    syntax varlist(max=1)
  2.    tempvar temp
  3.    qui gen `temp' = .
  4.    char rename `varlist' `temp'
  5.    drop `temp'
  6. end

.
. * ==============================================================================
. sysuse auto, clear
(1978 automobile data)

.
. * Add foreign and price to vl system and user macros:
. vl set (foreign price), dummy nonotes

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vldummy        |       1   0/1 variable
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       0   continuous variables
  $vluncertain    |       1   perhaps continuous, perhaps categorical variable
  $vlother        |       0   all missing or constant variables
-------------------------------------------------------------------------------

. vl create binary = (foreign)
note: $binary initialized with 1 variable.

. vl create contin = (price)
note: $contin initialized with 1 variable.

. vl move ($contin) vlcontinuous
note: 1 variable specified and 1 variable moved.

------------------------------
Macro          # Added/Removed
------------------------------
$vldummy                     0
$vlcategorical               0
$vlcontinuous                1
$vluncertain                -1
$vlother                     0
------------------------------

. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vldummy        |       1   0/1 variable
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       1   continuous variable
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $binary         |       1   variable
  $contin         |       1   variable
-------------------------------------------------------------------------------

.
. * Show characteristics of foreign and price:
. char list foreign[]
  foreign[_vlusernames]:      binary
  foreign[_vl_nlevels]:       2
  foreign[_vl_max]:           1
  foreign[_vl_min]:           0
  foreign[_vl_N]:             74
  foreign[_vlsysname]:        vldummy

. char list price[]
  price[_vlsysname]:          vlcontinuous
  price[_vlusernames]:        contin
  price[_vl_nlevels]:         74
  price[_vl_max]:             15906
  price[_vl_min]:             3291
  price[_vl_N]:               74

.
. * ------------------------------------------------------------------------------
. * Create foreign_new and price_new using -clonevar-:
. clonevar foreign_new = foreign

. clonevar price_new = price

.
. * Show characteristics of foreign_new and price_new:
. char list foreign_new[]
  foreign_new[_vlsysname]:    vldummy
  foreign_new[_vl_N]:         74
  foreign_new[_vl_min]:       0
  foreign_new[_vl_max]:       1
  foreign_new[_vl_nlevels]:   2
  foreign_new[_vlusernames]:  binary

. char list price_new[]
  price_new[_vl_N]:           74
  price_new[_vl_min]:         3291
  price_new[_vl_max]:         15906
  price_new[_vl_nlevels]:     74
  price_new[_vlusernames]:    contin
  price_new[_vlsysname]:      vlcontinuous

.
. * Rename value labels to name of variables (with labels defined):
. elabel list (foreign foreign_new)
origin:
           0 Domestic
           1 Foreign
origin:
           0 Domestic
           1 Foreign

. renlab foreign_new

. elabel list (foreign foreign_new)
foreign:
           0 Domestic
           1 Foreign
foreign_new:
           0 Domestic
           1 Foreign

.
. * Remove all characteristics from foreign_new and price_new:
. remchar foreign_new

. remchar price_new

.
. * Now foreign_new and price_new have no characteristics:
. char list foreign_new[]

. char list price_new[]

.
. * Add foreign_new and price_new to vl system and user vars:
. vl set (foreign_new price_new), nonotes  // vldummy exists, option dummy not needed

Added variables
-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vldummy        |       1   0/1 variable
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       0   continuous variables
  $vluncertain    |       1   perhaps continuous, perhaps categorical variable
  $vlother        |       0   all missing or constant variables
-------------------------------------------------------------------------------

. vl modify binary = binary + (foreign_new)
note: 1 variable added to $binary.

. vl modify contin = contin + (price_new)
note: 1 variable added to $contin.

. vl move ($contin) vlcontinuous
note: 2 variables specified and 1 variable moved;
      the other variable was already in $vlcontinuous.

------------------------------
Macro          # Added/Removed
------------------------------
$vldummy                     0
$vlcategorical               0
$vlcontinuous                1
$vluncertain                -1
$vlother                     0
------------------------------

. vl dir

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vldummy        |       2   0/1 variables
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       2   continuous variables
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $binary         |       2   variables
  $contin         |       2   variables
-------------------------------------------------------------------------------

.
. * Show characteristics of foreign_new and price_new:
. char list foreign_new[]
  foreign_new[_vlusernames]:  binary
  foreign_new[_vl_nlevels]:   2
  foreign_new[_vl_max]:       1
  foreign_new[_vl_min]:       0
  foreign_new[_vl_N]:         74
  foreign_new[_vlsysname]:    vldummy

. char list price_new[]
  price_new[_vlsysname]:      vlcontinuous
  price_new[_vlusernames]:    contin
  price_new[_vl_nlevels]:     74
  price_new[_vl_max]:         15906
  price_new[_vl_min]:         3291
  price_new[_vl_N]:           74

.
. * -vl rebuild- runs without error message:
. vl rebuild
Rebuilding vl macros ...

-------------------------------------------------------------------------------
                  |                      Macro's contents
                  |------------------------------------------------------------
Macro             |  # Vars   Description
------------------+------------------------------------------------------------
System            |
  $vldummy        |       2   0/1 variables
  $vlcategorical  |       0   categorical variables
  $vlcontinuous   |       2   continuous variables
  $vluncertain    |       0   perhaps continuous, perhaps categorical variables
  $vlother        |       0   all missing or constant variables
User              |
  $binary         |       2   variables
  $contin         |       2   variables
-------------------------------------------------------------------------------