Add a variable to a global variable list only if it is not already included

[Francois Durant]

Hi,

I would like to add a variable to a global variable list only if they are not already included. I have a set of independent variables that I need to add one by one to a global "current" varlist. The code below adds the variable regardless if already included in the "current" variable list. I want to add the variable only if it is not already included. If already included, skip and add next one. Anyone knows how to modify the code below? Thanks.

* This is the CURRENT list of variables
global M3 "LogSize MOM12 DY MOM36   retvol Debt_Price Sales_Price  mom1 rd_mve turn    bm agr chatoia egr grCAPX grltnoa invest  ROE  CEI hire   ChLTI ill  TAN indmom idiovol nincr LogIssues36 ES divo Accruals_old chmom_old  sgr pchsale_pchrect"

* Defining new models to test. Starting from model M5, M6...
local t 5
* Add one by one variables. I would like to incorporate a condition that skip the variable in this list if already included in the CURRENT list (M3) above:
local plus "  ROA  LogAG MOM6   Accruals_old Accruals  LogIssues12 Beta StdDev Turnover  cash chmom chnanalyst         chcsho chinv  grltnoa invest pchsale_pchinvt  GP_A OP_BE  NOA  chmom_old grltnoa_old divi divo   pchgm_pchsale pchsale_pchrect pctacc sgr ms OSc  ChCOL ChNNCWC ChNNCOA ChNCOA  ChNCOL ChLTI NWCCh AL    stdev_turn_w turn_w"
foreach add in `plus' {
    global M`t' $M3 `add'
    di `t'
    di "${M`t'}"
    di "`add'"
    local t = (`t' + 1)
}

[Joseph Coveney]

The first line inside your loop could be, for example

if strpos("$M3", " `add'") > 0 continue

You might want to try other ways to isolate the candidate as a word if you're worried about substrings within other variable names.

Why are you using global macros?

[Francois Durant]

I need to use global because I'm calling the macro across several subroutines. I am not seeking to change that but I would like a reliable way to solve the problem. I am indeed worried about substring within other variables, indeed. What is the best way to solve this problem? Thanks a lot for your help in advance!

[Joseph Coveney]

You can still use a local macro even when you're referring to it across several programs. You just feed it to the programs as an argument. It's a lot cleaner, less liable to messups.

Anyway, for the problem at hand, I'd probably employ Mata, which has a couple of built-in functions that you can take advantage of. You can do the same thing entirely within Stata by parsing through the macros, but I think that it's just a little easier with the functions available in Mata.

.ÿ
.ÿversionÿ16.1

.ÿ
.ÿclearÿ*

.ÿ
.ÿlocalÿline_sizeÿ`c(linesize)'

.ÿsetÿlinesizeÿ79

.ÿ
.ÿmata:
-------------------------------------------------ÿmataÿ(typeÿendÿtoÿexit)ÿ-----
:ÿmataÿsetÿmatastrictÿon

:ÿ
:ÿvoidÿfunctionÿaddEm(stringÿscalarÿcurrent_list,ÿstringÿscalarÿplus)ÿ{
>ÿ
>ÿÿÿÿÿstringÿrowvectorÿM3,ÿAdd
>ÿÿÿÿÿM3ÿ=ÿtokens(current_list);ÿAddÿ=ÿtokens(plus)
>ÿ
>ÿÿÿÿÿrealÿscalarÿindex
>ÿÿÿÿÿforÿ(index=1;index<=cols(Add);index++)ÿ{
>ÿÿÿÿÿÿÿÿÿifÿ(!rowsum(strmatch(M3,ÿAdd[index])))ÿM3ÿ=ÿM3,ÿAdd[index]
>ÿÿÿÿÿ}
>ÿÿÿÿÿst_global("M3",ÿinvtokens(M3))
>ÿ}

:ÿ
:ÿ
:ÿend
-------------------------------------------------------------------------------

.ÿ
.ÿ#delimitÿ;
delimiterÿnowÿ;
.ÿglobalÿM3ÿLogSizeÿMOM12ÿDYÿMOM36ÿ
>ÿÿÿÿÿÿÿÿÿretvolÿDebt_PriceÿSales_Priceÿmom1ÿrd_mveÿturnÿ
>ÿÿÿÿÿÿÿÿÿbmÿagrÿchatoiaÿegrÿgrCAPXÿgrltnoaÿinvestÿROEÿCEIÿhireÿ
>ÿÿÿÿÿÿÿÿÿChLTIÿillÿTANÿindmomÿidiovolÿnincrÿLogIssues36ÿESÿdivoÿ
>ÿÿÿÿÿÿÿÿÿAccruals_oldÿchmom_oldÿsgrÿpchsale_pchrect;

.ÿlocalÿplusÿROAÿLogAGÿMOM6ÿAccruals_oldÿAccrualsÿ
>ÿÿÿÿÿÿÿÿÿLogIssues12ÿBetaÿStdDevÿTurnoverÿcashÿchmomÿchnanalystÿÿ
>ÿÿÿÿÿÿÿÿÿchcshoÿchinvÿgrltnoaÿinvestÿpchsale_pchinvtÿGP_AÿOP_BEÿ
>ÿÿÿÿÿÿÿÿÿNOAÿchmom_oldÿgrltnoa_oldÿdiviÿdivoÿpchgm_pchsaleÿ
>ÿÿÿÿÿÿÿÿÿpchsale_pchrectÿpctaccÿsgrÿmsÿOScÿChCOLÿChNNCWCÿChNNCOAÿ
>ÿÿÿÿÿÿÿÿÿChNCOAÿChNCOLÿChLTIÿNWCChÿALÿstdev_turn_wÿturn_w;

.ÿ#delimitÿcr
delimiterÿnowÿcr
.ÿ
.ÿmata:ÿaddEm("$M3",ÿ"`plus'")

.ÿ
.ÿdisplayÿinÿsmclÿasÿtextÿ"$M3"
LogSizeÿMOM12ÿDYÿMOM36ÿretvolÿDebt_PriceÿSales_Priceÿmom1ÿrd_mveÿturnÿbmÿagrÿch
>ÿatoiaÿegrÿgrCAPXÿgrltnoaÿinvestÿROEÿCEIÿhireÿChLTIÿillÿTANÿindmomÿidiovolÿnin
>ÿcrÿLogIssues36ÿESÿdivoÿAccruals_oldÿchmom_oldÿsgrÿpchsale_pchrectÿROAÿLogAGÿM
>ÿOM6ÿAccrualsÿLogIssues12ÿBetaÿStdDevÿTurnoverÿcashÿchmomÿchnanalystÿchcshoÿch
>ÿinvÿpchsale_pchinvtÿGP_AÿOP_BEÿNOAÿgrltnoa_oldÿdiviÿpchgm_pchsaleÿpctaccÿmsÿO
>ÿScÿChCOLÿChNNCWCÿChNNCOAÿChNCOAÿChNCOLÿNWCChÿALÿstdev_turn_wÿturn_w

.ÿ
.ÿsetÿlinesizeÿ`line_size'

.ÿ
.ÿexit

endÿofÿdo-file


.

[Joseph Coveney]

Oh, and it's

help macrolists

for the Stata approach, in particular the

{local | global} macname : list macname | macname

and

{local | global} macname : list uniq macname

for a two- or three-line solution.

[Francois Durant]

Thanks Joseph. Do I need to include both or either of them? What's the exact syntax for me to integrate it in my loop? Thanks, again!

[Francois Durant]

I have tried the following syntax but it is invalid:

global M3 "LogSize MOM12 DY MOM36   retvol Debt_Price Sales_Price  mom1 rd_mve turn    bm agr chatoia egr grCAPX grltnoa invest  ROE  CEI hire   ChLTI ill  TAN indmom idiovol nincr LogIssues36 ES divo Accruals_old chmom_old  sgr pchsale_pchrect Beta StdDev"

*** Add anomalies:
local t 5
local plus "LogSize LogBM MOM12 MOM12_old LogIssues36 ROA DY LogAG MOM6 MOM36 MOM36_old  Accruals_old Accruals  LogIssues12 Beta StdDev Turnover Debt_Price Sales_Price cash chmom chnanalyst mom1 nincr rd_mve turn    bm agr chatoia chcsho chinv egr grCAPX grltnoa invest pchsale_pchinvt ROE GP_A OP_BE CEI NOA CEI_old chmom_old grltnoa_old divi divo hire indmom pchgm_pchsale pchsale_pchrect pctacc sgr ms OSc ChCOA ChCOL ChNNCWC ChNNCOA ChNCOA  ChNCOL ChLTI NWCCh AL ES TAN idiovol ill retvol stdev_turn_w turn_w"
foreach add in `plus' {
    global M`t': list uniq $M3 `add'
    di `t'
    di "${M`t'}"
    di "`add'"
    local t = (`t' + 1)
}

as well as this:

global M3 "LogSize MOM12 DY MOM36   retvol Debt_Price Sales_Price  mom1 rd_mve turn    bm agr chatoia egr grCAPX grltnoa invest  ROE  CEI hire   ChLTI ill  TAN indmom idiovol nincr LogIssues36 ES divo Accruals_old chmom_old  sgr pchsale_pchrect Beta StdDev"

*** Add anomalies:
local t 5
local plus "LogSize LogBM MOM12 MOM12_old LogIssues36 ROA DY LogAG MOM6 MOM36 MOM36_old  Accruals_old Accruals  LogIssues12 Beta StdDev Turnover Debt_Price Sales_Price cash chmom chnanalyst mom1 nincr rd_mve turn    bm agr chatoia chcsho chinv egr grCAPX grltnoa invest pchsale_pchinvt ROE GP_A OP_BE CEI NOA CEI_old chmom_old grltnoa_old divi divo hire indmom pchgm_pchsale pchsale_pchrect pctacc sgr ms OSc ChCOA ChCOL ChNNCWC ChNNCOA ChNCOA  ChNCOL ChLTI NWCCh AL ES TAN idiovol ill retvol stdev_turn_w turn_w"
foreach add in `plus' {
    global M`t': list $M3 | `add'
    di `t'
    di "${M`t'}"
    di "`add'"
    local t = (`t' + 1)
}

Also, even if the syntax was valid, this solution would increment the counter M5, M6 etc.. with models that are not different than M3 when `add' is a duplicate. Would it be possible to fix this problem too? I would like to have each M* to have one different additional variable relative to M3. Thank you in advance!

[Bjarte Aagnes]

Maybe you can use elements form the following example:

sysuse auto
global xvars_3 make price mpg

foreach vn of varlist * {
    
    if ( "`vlist'" == "" ) {
        
        loc vlist $xvars_3   
    }
    
    else { 
        
        loc vlist : list vlist | vn
        loc nvars = wordcount("`vlist'")
        glo xvars_`nvars' `vlist'
    }
}

[William Lisowski]

Start by removing the duplicates from the list of additions, then just loop over the list of unduplicated items.

global M3 "LogSize MOM12 DY MOM36   retvol Debt_Price Sales_Price  mom1 rd_mve turn    bm agr chatoia egr grCAPX grltnoa invest  ROE  CEI hire   ChLTI ill  TAN indmom idiovol nincr LogIssues36 ES divo Accruals_old chmom_old  sgr pchsale_pchrect Beta StdDev"

local plus "LogSize LogBM MOM12 MOM12_old LogIssues36 ROA DY LogAG MOM6 MOM36 MOM36_old  Accruals_old Accruals  LogIssues12 Beta StdDev Turnover Debt_Price Sales_Price cash chmom chnanalyst mom1 nincr rd_mve turn    bm agr chatoia chcsho chinv egr grCAPX grltnoa invest pchsale_pchinvt ROE GP_A OP_BE CEI NOA CEI_old chmom_old grltnoa_old divi divo hire indmom pchgm_pchsale pchsale_pchrect pctacc sgr ms OSc ChCOA ChCOL ChNNCWC ChNNCOA ChNCOA  ChNCOL ChLTI NWCCh AL ES TAN idiovol ill retvol stdev_turn_w turn_w"

local newplus : list plus - global(M3)

macro list M3 _plus _newplus

. macro list M3 _plus _newplus
M3:             LogSize MOM12 DY MOM36 retvol Debt_Price Sales_Price mom1 rd_mve turn bm agr
                chatoia egr grCAPX grltnoa invest ROE CEI hire ChLTI ill TAN indmom idiovol
                nincr LogIssues36 ES divo Accruals_old chmom_old sgr pchsale_pchrect Beta StdDev
_plus:          LogSize LogBM MOM12 MOM12_old LogIssues36 ROA DY LogAG MOM6 MOM36 MOM36_old
                Accruals_old Accruals LogIssues12 Beta StdDev Turnover Debt_Price Sales_Price
                cash chmom chnanalyst mom1 nincr rd_mve turn bm agr chatoia chcsho chinv egr
                grCAPX grltnoa invest pchsale_pchinvt ROE GP_A OP_BE CEI NOA CEI_old chmom_old
                grltnoa_old divi divo hire indmom pchgm_pchsale pchsale_pchrect pctacc sgr ms
                OSc ChCOA ChCOL ChNNCWC ChNNCOA ChNCOA ChNCOL ChLTI NWCCh AL ES TAN idiovol ill
                retvol stdev_turn_w turn_w
_newplus:       LogBM MOM12_old ROA LogAG MOM6 MOM36_old Accruals LogIssues12 Turnover cash
                chmom chnanalyst chcsho chinv pchsale_pchinvt GP_A OP_BE NOA CEI_old grltnoa_old
                divi pchgm_pchsale pctacc ms OSc ChCOA ChCOL ChNNCWC ChNNCOA ChNCOA ChNCOL NWCCh
                AL stdev_turn_w turn_w

[Francois Durant]

Elegant and efficient William. Thanks a lot!

[Francois Durant]

Hi William,

I probably have a syntax error, but I do not understand why "CAR" appears in the Stata output `plus' considering it is included in M1. Any idea?

global All_ANO " CAR MOM1 MOM4 MOM6 MOM12 VAL ln_VAL CAR CARVOL beta idiovol retvol retvol_w "
di "$All_ANO"
global M1 " CAR MOM12 VAL MOM1 retvol "
di "$M1"

local benchmark $M1
di "`benchmark'"
local plus : list global(All_ANO) - benchmark
di "`plus'"

. global All_ANO " CAR MOM1 MOM4 MOM6 MOM12 VAL ln_VAL CAR CARVOL beta idiovol retvol retvol_w "

. di "$All_ANO"
 CAR MOM1 MOM4 MOM6 MOM12 VAL ln_VAL CAR CARVOL beta idiovol retvol retvol_w 

. global M1 " CAR MOM12 VAL MOM1 retvol "

. di "$M1"
 CAR MOM12 VAL MOM1 retvol 

. 
. local benchmark $M1

. di "`benchmark'"
CAR MOM12 VAL MOM1 retvol

. local plus : list global(All_ANO) - benchmark

. di "`plus'"
CAR MOM4 MOM6 ln_VAL CARVOL beta idiovol retvol_w

[William Lisowski]

CAR appears twice in $ALL_ANO and once in $M1. Apparently macro list removes at most one matching element for each element in the second macro.

. // example #1
. local a car bus car

. local b car

. local c : list a - b

. display "`c'"
car bus

. // example #2
. local a car bus car

. local b car car

. local c : list a - b

. display "`c'"
bus

. // example #3
. local a car bus car

. local aa : list uniq a

. local b car

. local c : list aa - b

. display "`c'"
bus

.

[Francois Durant]

Thanks for the clarification William, it helps!

[Francois Durant]

Is there an easy way to check that there are no duplicates in a macro list and remove the excess if any?

[Joseph Coveney]

Yes.

{local | global} macname : list uniq macname

Mentioned in this thread here and again here.