Converting varlist into a numbest

[Dominique Bourget]

Hello,
I have the code below:

clear

python:

from sklearn.naive_bayes import MultinomialNB
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import train_test_split
from sklearn import metrics # import scikit-learn metrics module for accuracy calculation
from sfi import Data
from sfi import Macro
import numpy as np
import pandas as pd

parameters = [{'alpha':np.linspace(0.1,4.6,10)}]

# right now the parameter is a list of dictionary that stores a numpy array
# the code below extracts the numpy array stored under the name 'alpha' and
# converts the numpy array into a python list before exporting it on stata
alpha_grid = parameters[0]['alpha'].tolist()

# export alpha_grid as a Stata variable
Data.setObsTotal(len(alpha_grid))
Data.addVarFloat('alphaGrid')
Data.store(var = 'alphaGrid', obs = None, val = alpha_grid)

end // end python mode

and I am trying to use the foreach loop on Stata with the line below:
foreach a1 of numlist alphaGrid {
...
}

however, doing so causes an error --- on my Stata window, the variable description appears to be that the variable 'alphaGrid' is a float type, and I am not sure why this line is causing an error.
The error is 'invalid numlist'

How can I resolve this issue?

Thank you,

[William Lisowski]

alphaGrid is a Stata variable, as your comment recognizes. A Stata variable is not a numlist, as the output of help numlist will show you.

Here are two examples of looping over a list of numbers.

. foreach a1 of numlist 3 1 4 {
  2.     display "`a1'"
  3. }
3
1
4

.
. local e 2 7 1 8

. foreach a1 of local e {
  2.     display "`a1'"
  3. }
2
7
1
8

.

Perhaps you should return your list of numbers to a Stata local macro rather than a Stata variable.

Alternatively, the following shows how to loop over the values of a Stata variable in the order they appear in the Stata dataset.

. input x

             x
  1. 3
  2. 1
  3. 4
  4. end

.
. local n = c(N)

. forvalues i=1/`n' {
  2.     display x[`i']
  3. }
3
1
4

.

[William Lisowski]

Let me demonstrate converting a list of numbers to a string and returning that string to a local macro in Stata, and then looping over the elements of the list.

. python
----------------------------------------------- python (type end to exit) ----------------------
>>> from sfi import Macro
>>> pi_list = [3, 1, 4]
>>> pi_str = ' '.join(str(e) for e in pi_list)
>>> Macro.setLocal("mypi",pi_str)
>>> end
------------------------------------------------------------------------------------------------

. 
. foreach a1 of local mypi {
  2.     display "`a1'"
  3. }
3
1
4

.

[Dominique Bourget]

Thank you

[William Lisowski]

On the topic at https://www.statalist.org/forums/for...-a-local-macro you posted a followup to what I wrote above.

I saw your reply and tried to import the python variable as a Stata local macro, but when I iterate the local macro in a loop, nothing is showing. and when I do -di "`alphaGrid'"-, the code runs without error but nothing is showing up neither. I don't think the -Macro.setLocal("alphaGrid",alpha_str)- is working for me.

A part of my code is below:

python:

# implementing grid search for Laplace smoothing in MultinomialNB on Python
# test the model for alpha = 0.1, 0.6, ...., 4.6
# set the range for alpha
parameters = {'alpha':np.linspace(0.1,4.6,10)}

# right now the parameter is a list of dictionary that stores a numpy array
# the code below extracts the numpy array stored under the name 'alpha' and
# converts the numpy array into a python list before exporting it on stata
alpha_grid = parameters['alpha'].tolist()
alpha_str = ' '.join(str(e) for e in alpha_grid)
Macro.setLocal("alphaGrid",alpha_str)

end

foreach a1 of local alphaGrid {

display " `a1' "

} // the output is empty

I expect that the problem is the definition of alpha_grid within Python. Here is code that shows the alpha_grid in Python and the alphaGrid local macro in Stata. You should similarly print(alpha_grid) in Python to start tracing where you are going wrong. This is basic problem-solving,

. python:
----------------------------------------------- python (type end to exit) ----------------------
>>> 
>>> from sfi import Macro
>>> 
>>> # right now the parameter is a list of dictionary that stores a numpy array
... # the code below extracts the numpy array stored under the name 'alpha' and
... # converts the numpy array into a python list before exporting it on stata
... alpha_grid = [0.1, 0.6, 1.1, 1.6, 2.1, 2.6, 3.1, 3.6, 4.1, 4.6]
>>> print(alpha_grid)
[0.1, 0.6, 1.1, 1.6, 2.1, 2.6, 3.1, 3.6, 4.1, 4.6]
>>> alpha_str = ' '.join(str(e) for e in alpha_grid)
>>> Macro.setLocal("alphaGrid",alpha_str)
>>> 
>>> end
------------------------------------------------------------------------------------------------

. 
. macro list _alphaGrid
_alphaGrid:     0.1 0.6 1.1 1.6 2.1 2.6 3.1 3.6 4.1 4.6

. 
end of do-file

.

[Dominique Bourget]

Hello,

I had tried your problem solving approach already before I posted the other question, but nothing seem to be wrong with the python portion of my code. The list is correctly displayed under python, and your code for alpha_str also works well under python. I think the problem is with the Macro.setLocal statement, which I don't understand why (since it works on your computer but not mine). The other lines of code that requires using the sfi python module (e.g. Data.get(), Data.addVarFloat(), etc.) successfully transfers variables onto Stata, but for some reason none of my Macro.setLocal commands create local macros on my Stata.

What should I do?

Thank you,

[William Lisowski]

Do you have a Do-file editor window open with the Python code followed by the Stata code that runs afterwards? And are you perhaps not running the entire Do-file at once? Have you first selected the Python code and run it, and then selected the Stata foreach loop and run it? If so, that is the source of the failure of your code to work.

Unlike Stata variables, local macros vanish when the do-file within which they were created ends. They are "local" to the do-file within which they are run (which is not necessarily the do-file into which the code was typed).

Below I reproduce your problem using my example from post #5. First I run the Python block, and then I run the Stata code following it.

If you look carefully at the results below, you'll see that when I selected the Python code to run, it was copied into a temporary do-file and run, so even though all the code is in the same window in the do-file editor, the separate selections are run as separate do-files, and local macro defined by Python vanishes at the end of the first do-file, and is undefined when the Stata code is run in the second do-file.

. do "/var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//SD77506.000000"

. python:
----------------------------------------------- python (type end to exit) ----------------------
>>>
>>> from sfi import Macro
>>>
>>> # right now the parameter is a list of dictionary that stores a numpy array
... # the code below extracts the numpy array stored under the name 'alpha' and
... # converts the numpy array into a python list before exporting it on stata
... alpha_grid = [0.1, 0.6, 1.1, 1.6, 2.1, 2.6, 3.1, 3.6, 4.1, 4.6]
>>> print(alpha_grid)
[0.1, 0.6, 1.1, 1.6, 2.1, 2.6, 3.1, 3.6, 4.1, 4.6]
>>> alpha_str = ' '.join(str(e) for e in alpha_grid)
>>> Macro.setLocal("alphaGrid",alpha_str)
>>>
>>> end
------------------------------------------------------------------------------------------------

.
end of do-file

. do "/var/folders/xr/lm5ccr996k7dspxs35yqzyt80000gp/T//SD77506.000000"

. macro list _alphaGrid
local macro `alphaGrid' not found
r(111);

end of do-file

r(111);