Problems with SIEVE EGENMORE

[Isabelle Jacques]

I am using egenmore to clean my database, I have used the same code previously and all was well until today when my egenmore sieve fonction stopped working. It says that it has a syntax error. Can someone tell me what is going wrong, why doesn't my code that I have used previously not work today !. This is the Trace On that I get:
egen nn_fatot=sieve(mhab), char(0 1 2 3 4 5 6 7 8 9 . )
--------------------------------------------------------------------------------------------------------------------------------------------- begin egen ---
- version 6, missing
- local cvers = _caller()
- gettoken type 0 : 0, parse(" =(")
- gettoken name 0 : 0, parse(" =(")
- if `"`name'"'=="=" {
= if `"="'=="=" {
- local name `"`type'"'
= local name `"nn_fatot"'
- local type : set type
- }
- else {
gettoken eqsign 0 : 0, parse(" =(")
if `"`eqsign'"' != "=" {
error 198
}
}
- confirm new variable `name'
= confirm new variable nn_fatot
- gettoken fcn 0 : 0, parse(" =(")
- gettoken args 0 : 0, parse(" ,") match(par)
- if "`c(adoarchive)'"=="1" {
= if ""=="1" {
capture qui _stfilearchive find _g`fcn'.ado
if _rc {
di as error "unknown egen function `fcn'()"
exit 133
}
}
- else {
- capture qui findfile _g`fcn'.ado
= capture qui findfile _gsieve.ado
--------------------------------------------------------------------------------------------------------------------------------------- begin findfile ---
- version 8
- gettoken fn 0 : 0, parse(" ,")
- syntax [, ALL noDEScend PATH(string)]
- if `"`path'"'=="" {
= if `""'=="" {
- local path `"`c(adopath)'"'
= local path `"BASE;SITE;.;PERSONAL;PLUS;OLDPLACE"'
- }
- local n 0
- local subdir : adosubdir `"`fn'"'
= local subdir : adosubdir `"_gsieve.ado"'
- if `"`subdir'"' != "" {
= if `"_"' != "" {
- gettoken d path : path, parse(";")
- while `"`d'"'!="" {
= while `"BASE"'!="" {
- if `"`d'"' != ";" {
= if `"BASE"' != ";" {
- local d : sysdir `"`d'"'
= local d : sysdir `"BASE"'
- local ffn `"`d'`fn'"'
= local ffn `"C:\Program Files (x86)\Stata13\ado\base/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"C:\Program Files (x86)\Stata13\ado\base/_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
- if "`descend'"=="" {
= if ""=="" {
- local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
= local ffn `"C:\Program Files (x86)\Stata13\ado\base/_/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"C:\Program Files (x86)\Stata13\ado\base/_/_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `"`ffn'"'
local n 1
}
- }
- }
- gettoken d path : path, parse(" ;")
- }
- while `"`d'"'!="" {
= while `";"'!="" {
- if `"`d'"' != ";" {
= if `";"' != ";" {
local d : sysdir `"`d'"'
local ffn `"`d'`fn'"'
capture confirm file `"`ffn'"'
if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
if "`descend'"=="" {
local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
capture confirm file `"`ffn'"'
if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `"`ffn'"'
local n 1
}
}
}
- gettoken d path : path, parse(" ;")
- }
- while `"`d'"'!="" {
= while `"SITE"'!="" {
- if `"`d'"' != ";" {
= if `"SITE"' != ";" {
- local d : sysdir `"`d'"'
= local d : sysdir `"SITE"'
- local ffn `"`d'`fn'"'
= local ffn `"C:\Program Files (x86)\Stata13\ado\site/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"C:\Program Files (x86)\Stata13\ado\site/_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
- if "`descend'"=="" {
= if ""=="" {
- local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
= local ffn `"C:\Program Files (x86)\Stata13\ado\site/_/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"C:\Program Files (x86)\Stata13\ado\site/_/_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `"`ffn'"'
local n 1
}
- }
- }
- gettoken d path : path, parse(" ;")
- }
- while `"`d'"'!="" {
= while `";"'!="" {
- if `"`d'"' != ";" {
= if `";"' != ";" {
local d : sysdir `"`d'"'
local ffn `"`d'`fn'"'
capture confirm file `"`ffn'"'
if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
if "`descend'"=="" {
local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
capture confirm file `"`ffn'"'
if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `"`ffn'"'
local n 1
}
}
}
- gettoken d path : path, parse(" ;")
- }
- while `"`d'"'!="" {
= while `"."'!="" {
- if `"`d'"' != ";" {
= if `"."' != ";" {
- local d : sysdir `"`d'"'
= local d : sysdir `"."'
- local ffn `"`d'`fn'"'
= local ffn `"./_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"./_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
- if "`descend'"=="" {
= if ""=="" {
- local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
= local ffn `"./_/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"./_/_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `"`ffn'"'
local n 1
}
- }
- }
- gettoken d path : path, parse(" ;")
- }
- while `"`d'"'!="" {
= while `";"'!="" {
- if `"`d'"' != ";" {
= if `";"' != ";" {
local d : sysdir `"`d'"'
local ffn `"`d'`fn'"'
capture confirm file `"`ffn'"'
if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
if "`descend'"=="" {
local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
capture confirm file `"`ffn'"'
if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `"`ffn'"'
local n 1
}
}
}
- gettoken d path : path, parse(" ;")
- }
- while `"`d'"'!="" {
= while `"PERSONAL"'!="" {
- if `"`d'"' != ";" {
= if `"PERSONAL"' != ";" {
- local d : sysdir `"`d'"'
= local d : sysdir `"PERSONAL"'
- local ffn `"`d'`fn'"'
= local ffn `"c:\ado\personal/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"c:\ado\personal/_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
- if "`descend'"=="" {
= if ""=="" {
- local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
= local ffn `"c:\ado\personal/_/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"c:\ado\personal/_/_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `"`ffn'"'
local n 1
}
- }
- }
- gettoken d path : path, parse(" ;")
- }
- while `"`d'"'!="" {
= while `";"'!="" {
- if `"`d'"' != ";" {
= if `";"' != ";" {
local d : sysdir `"`d'"'
local ffn `"`d'`fn'"'
capture confirm file `"`ffn'"'
if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
if "`descend'"=="" {
local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
capture confirm file `"`ffn'"'
if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `"`ffn'"'
local n 1
}
}
}
- gettoken d path : path, parse(" ;")
- }
- while `"`d'"'!="" {
= while `"PLUS"'!="" {
- if `"`d'"' != ";" {
= if `"PLUS"' != ";" {
- local d : sysdir `"`d'"'
= local d : sysdir `"PLUS"'
- local ffn `"`d'`fn'"'
= local ffn `"c:\ado\plus/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"c:\ado\plus/_gsieve.ado"'
- if _rc==0 {
di as txt `"`ffn'"'
if "`all'"=="" {
ret local fn `"`ffn'"'
exit
}
if `n' {
ret local fn `"`return(fn)' "`ffn'""'
}
else ret local fn `""`ffn'""'
local n 1
}
- if "`descend'"=="" {
= if ""=="" {
- local ffn `"`d'`subdir'`c(dirsep)'`fn'"'
= local ffn `"c:\ado\plus/_/_gsieve.ado"'
- capture confirm file `"`ffn'"'
= capture confirm file `"c:\ado\plus/_/_gsieve.ado"'
- if _rc==0 {
- di as txt `"`ffn'"'
= di as txt `"c:\ado\plus/_/_gsieve.ado"'
- if "`all'"=="" {
= if ""=="" {
- ret local fn `"`ffn'"'
= ret local fn `"c:\ado\plus/_/_gsieve.ado"'
- exit
----------------------------------------------------------------------------------------------------------------------------------------- end findfile ---
- if (`"`r(fn)'"' == "") {
= if (`"c:\ado\plus/_/_gsieve.ado"' == "") {
di as error "unknown egen function `fcn'()"
exit 133
}
- }
- if `"`par'"' != "(" {
= if `"("' != "(" {
exit 198
}
- if `"`args'"' == "_all" | `"`args'"' == "*" {
= if `"mhab"' == "_all" | `"mhab"' == "*" {
version 7.0, missing
unab args : _all
local args : subinstr local args "`_sortindex'" "", all word
version 6.0, missing
}
- syntax [if] [in] [, *]
- if _by() {
local byopt "by(`_byvars')"
local cma ","
}
- else if `"`options'"' != "" {
= else if `"char(0 1 2 3 4 5 6 7 8 9 . )"' != "" {
- local cma ","
- }
- tempvar dummy
- global EGEN_Varname `name'
= global EGEN_Varname nn_fatot
- version 7.0, missing
- global EGEN_SVarname `_sortindex'
= global EGEN_SVarname __000000
- version 6.0, missing
- if ("`fcn'" == "mode" | "`fcn'" == "concat") {
= if ("sieve" == "mode" | "sieve" == "concat") {
local vv : display "version " string(`cvers') ", missing:"
}
- capture noisily `vv' _g`fcn' `type' `dummy' = (`args') `if' `in' `cma' `byopt' `options'
= capture noisily _gsieve float __000001 = (mhab) , char(0 1 2 3 4 5 6 7 8 9 . )
---------------------------------------------------------------------------------------------------------------------------------------- begin _gsieve ---
- version 7.0
- gettoken type 0 : 0
- gettoken g 0 : 0
- gettoken eqs 0 : 0
- syntax varname(string) [if] [in] [, KEEP(str) CHAR(str asis) OMIT(str asis) ]
- local nopts = ("`keep'" != "") + (`"`char'"' != "") + (`"`omit'"' != "")
= local nopts = ("" != "") + (`"0 1 2 3 4 5 6 7 8 9 . "' != "") + (`""' != "")
- if `nopts' != 1 {
= if 1 != 1 {
di as err "specify keep() or char() or omit()"
exit 198
}
- if `"`omit'"' != "" {
= if `""' != "" {
local char `"`omit'"'
local not "!"
}
- marksample touse, strok
- local type "str1"
- qui gen `type' `g' = ""
= qui gen str1 __000001 = ""
- local length : type `varlist'
= local length : type mhab
- local length = substr("`length'",4,.)
= local length = substr("strL",4,.)
- if "`keep'" != "" {
= if "" != "" {
local a 0
local n 0
local o 0
local s 0
foreach w of local keep {
local l = length("`w'")
if substr("alphabetic",1,max(1,`l')) == "`w'" {
local a 1
}
else if substr("numeric",1,max(1,`l')) == "`w'" {
local n 1
}
else if substr("other",1,max(1,`l')) == "`w'" {
local o 1
}
else if substr("spaces",1,max(1,`l')) == "`w'" {
local s 1
}
else {
di as err "keep() invalid"
exit 198
}
}
tempvar c
quietly {
gen str1 `c' = ""
forval i = 1 / `length' {
replace `c' = substr(`varlist',`i',1)
if `a' {
replace `g' = `g' + `c' if ((`c' >= "A" & `c' <= "Z") | (`c' >= "a" & `c' <= "z"))
}
if `n' {
replace `g' = `g' + `c' if (`c' >= "0" & `c' <= "9")
}
if `s' {
replace `g' = `g' + `c' if `c' == " "
}
if `o' {
replace `g' = `g' + `c' if !( (`c' >= "A" & `c' <= "Z") | (`c' >= "a" & `c' <= "z") | (`c' >= "0" & `c' <= "9") | (`c' == " ") )
}
}
}
}
- else {
- forval i = 1 / `length' {
= forval i = 1 / L {
invalid syntax
qui replace `g' = `g' + substr(`varlist',`i',1) if `not'index(`"`char'"', substr(`varlist',`i',1))
}
------------------------------------------------------------------------------------------------------------------------------------------ end _gsieve ---
- global EGEN_SVarname
- global EGEN_Varname
- if _rc { exit _rc }
----------------------------------------------------------------------------------------------------------------------------------------------- end egen ---
r(198);

[Nick Cox]

Your problem is nothing to do with a program being obedient yesterday and perverse today. Programs don't stop working like a washing machine in which a part has succumbed to old age and overuse.

My guess is that the difference today is that you are trying to process a strL variable.

The egen function sieve() is from egenmore (from SSC, as you are asked to explain).

Typing the code by e.g.

. ssc type _gsieve.ado

reveals at the outset

*! 1.0.0 NJC 23 Sept 2002 
program define _gsieve 
        version 7.0

i.e. this function was written in 2002 for Stata 7. It was thus unaware of, and in fact does not support, strL variables, which were introduced in a later version of Stata.

I gather that the author has no plans to update this function. But, no matter, it seems that you are trying to extract numeric parts from a string variable, for which a much better tool is moss, also from SSC.

[Isabelle Jacques]

Ok I did a recast str on my variables and now it works (even if initially this one was str108....)

[Robert Picard]

You can do this with a unicode regex function:

* Example generated by -dataex-. To install: ssc install dataex
clear
input str9 v
"a105/"    
"1abc2def3"
end

gen spart = ustrregexra(v,"[0-9]+","")
gen npart = ustrregexra(v,"[^0-9]+","")
list

and the results

. list

     +----------------------------+
     |         v    spart   npart |
     |----------------------------|
  1. |     a105/       a/     105 |
  2. | 1abc2def3   abcdef     123 |
     +----------------------------+