Loop over all numeric variables except some

[Anna Mejldal]

Hi,
I have a large number of numeric variables (about 3-400) where I want to replace certain values with missing, and I use this loop:

quietly ds, has(type numeric)

foreach var of varlist `r(varlist)' {
display "Replacing missing in `var'"
list `var' if inlist(`var',99,999,888,-9)
replace `var' = . if inlist(`var',99,999,888,-9)
}

But for a few of these numeric variables, I don't want to replace these values, for example record_id, where 99 and 888 are valid id's.
Is it possible to remove som of the variables generated in ds? Or some other smart solution?

My only idea is to temporarily convert them to strings, but that's not so elegant :-)

Thanks, Anna

[Maarten Buis]

You can use extended macro functions to exclude variables from a list. I always forget the name of the helpfile, so what I do is type help macro and click on the link to extended macro functions. Within that help file there is a link to Macro extended functions for manipulating lists.

qui ds, has(type numeric)
local varlist `r(varlist)'
local toexclude "record_id"
local varlist : list varlist - toexclude

foreach var of local varlist {
    stuff
}

[Anna Mejldal]

Thanks, that worked perfectly!

[Shahida Pervin]

 local data: dir "`dir'" files "*.dta"

. di `data'
ind_cddb.dtarec21_1.dtarec22_2.dtarec31_3.dtarec32_4.dtarec33_5.dtarec34_6.dtarec41_7.dtarec42_8.dtaunitcode.dta

. local exclude `""ind_cddb" "unitcode""'

. di `exclude'
ind_cddbunitcode

. local data: list data  - exclude

. di `data'
ind_cddb.dtarec21_1.dtarec22_2.dtarec31_3.dtarec32_4.dtarec33_5.dtarec34_6.dtarec41_7.dtarec42_8.dtaunitcode.dta

Hello, I try to exclude some files from the folder, followed the -list- command here, it does not work. Is there any other command than -list- for files (not the variable)? Thank you.

[Nick Cox]

I guess that what "does not work" here is that you need double quotes in

di "`data'"

to insist to display that the argument is a literal string, so that spaces should be respected. As in other threads you've posted on

macro list

is a safer command if you are not on top of what display does and doesn't do, which is a little subtle.

Here list is not a command: it is a macro function.

[Shahida Pervin]

I want to exclude

      
 local exclude `""ind_cddb" "unitcode""'

in the

 macro list

exclusion of the files is not the result.

[Nick Cox]

As in earlier posts, we can't stress enough that your examples are hard to reproduce because we don't have your files and so dir will not give us the same filenames.

However, working harder to produce for us a reproducible example would make it easier to see the problem -- easier for us to see the problem and -- more importantly -- easier for you to get better at debugging your code.

I can reconstruct your problem as follows, by editing the mangled output in #4. (I already explained that it is mangled because you need a more subtle use of display. You didn't fix that by giving us a more readable version.)

. local data ind_cddb.dta rec21_1.dta rec22_2.dta rec31_3.dta rec32_4.dta rec33_5.dta rec34_6.dta rec41_7.dta  rec42_8.dta unitcode.dta


. local exclude ind_cddb unitcode


. local data: list data  - exclude


. di "`data'"
ind_cddb.dta rec21_1.dta rec22_2.dta rec31_3.dta rec32_4.dta rec33_5.dta rec34_6.dta rec41_7.dta rec42_8.dta unitcode.dta

Now the problem is evident:

1. As far as any local definitions are concerned, the elements are just arbitrary text. No command in the segment above knows or cares that the elements (in Stata terms, the "words") are your filenames or parts of your filenames.

2. No element (word) of exclude is an element of data. Hence none is removed. This is because you omitted the extension .dta in defining the exclude. Again, there is no intelligence here imputing the extension. The manipulations are totally literal and without any regard to what they mean for the user.

Irritating, but logical.

[Shahida Pervin]

I see, spot on, its just solved now. Glad stata forum is here to support. Thank you.

[Shahida Pervin]

cd "\MergeAppend"
local dir "\MergeAppend"
local data: dir "`dir'" files "*.dta"
foreach file of local data {
use `"`dir'/`file'"', clear
save, replace
}

foreach file of local data {
use `"`dir'/`file'"', clear
merge 1:1 ID_NO using "`file'"  , nogen
}

Hi, the codes generate the last file in the directory. Is this issue of specifying master file or anything else? Thank you.

[Shahida Pervin]

Please ignore, its sorted, I dont see post delete option. Thanks

[Nick Cox]

Good that you solved your problem, although the solution might be interesting or useful to others.

You can't delete posts. This is explained at https://www.statalist.org/forums/help#closure