Cross-validation using "xv" - error code r(101) "factor-variable and time-series operators not allowed"

[Emma Williams-Baron]

Hello, Statalist!

I am attempting to perform a cross-validation (CV) exercise using the SSC package crossvalidate2, command xv, using Stata 15 on Windows 11.

I am trying to generate the CV with:

ssc install crossvalidate2
xv 0.8, metric(mse): reg log_wages screener_hispanic

Despite omitting factor-variable and time-series operators in my code, I am getting this error:

factor-variable and time-series operators not allowed
r(101);

I have tried running the CV using Stata 19 through a virtual desktop and received the same error, so I suspect this issue isn't explained by Stata version.

I have also tried running the xv command with other dependent and independent variables, and still get the same error, so I don't think the variables themselves explain the problem either.

I have considered using the H2O CV tools, but it seems they only work with gradient boosting machine (GBM) or random forest (RF) regression. I would like my CV to hew as closely as possible to my main set of analyses, which use Stata's regress and logistic commands. It isn't clear to me whether GBM or RF cross-validation results would be comparable to my primary analyses. And, H2O requires Stata 19; I can access Stata 19 through a virtual desktop, but it is slow, so I am crossing my fingers that I can find a way to get xv to work on my machine.

Any help in figuring out why this error persists would be much appreciated!

Here is example data and the commands I am running:

* Example generated by -dataex-. To install: ssc install dataex
clear
input float(log_wages screener_hispanic)
        . 0
2.2055233 0
        . 0
        . 0
        . 0
 3.445108 0
 2.626117 0
 2.795043 0
 3.054473 0
        . 0
3.0041964 0
        . 0
 2.977738 0
 3.630897 0
        . 0
 3.379974 0
  3.30707 0
 4.005422 0
        . 0
  3.63275 0
 3.715521 0
2.9493394 0
        . 0
        . 0
 2.995566 0
        . 0
        . 0
        . 0
3.4324806 0
        . 0
 2.902703 1
 3.583102 1
2.3025851 1
2.0869136 0
        . 0
 3.259634 0
  2.92352 0
3.4062955 0
 3.130846 0
 3.304442 0
  3.57169 0
 2.994064 0
1.8718022 0
2.3025851 0
        . 0
 3.264486 0
 3.264486 0
        . 0
3.1535904 0
  2.90526 0
        . 0
        . 0
1.6389967 1
 2.890372 0
        . 0
        . 0
 2.650892 0
 3.280911 0
3.0445225 0
        . 0
 3.243373 0
 2.924057 0
 3.678913 0
        . 0
        . 0
 2.833213 0
        . 0
        . 0
3.6156814 0
 3.141995 0
        . 0
 2.890372 0
4.0715876 0
        . 0
3.5695326 0
 2.860294 0
 3.438921 0
 3.203762 0
 4.047078 0
1.8771724 0
 2.650421 0
        . 0
        . 0
 2.014903 0
2.1690538 0
 2.184177 0
2.3191144 0
 2.486156 0
 3.165897 0
 3.928421 0
3.0170045 0
 2.817004 0
 3.371883 0
 2.878449 0
2.6957526 0
3.0150445 0
 3.111736 0
        . 0
2.3639936 0
3.3943965 0
end
label values screener_hispanic screener_hispanic
label def screener_hispanic 0 "Non-Hispanic", modify
label def screener_hispanic 1 "Hispanic", modify

*** Attempt at cross-validation:
ssc install crossvalidate2
xv 0.8, metric(mse): reg log_wages screener_hispanic // error code r(101): factor-variable and time-series operators not allowed

[Hemanshu Kumar]

I don't have a solution for you, but this might help trouble-shoot.

First off, your code actually cannot run as-is, at least not on a computer that does not have the additional community-contributed command splitit installed (from SSC). This is because xv.ado itself calls splitit.

And indeed, running your xv command with set trace on reveals that the error is thrown precisely by xv's call to splitit:

- splitit `props' `ifin', `uid' `tpoint' `kfold' `split'
= splitit 0.8 ,    split(__000003)
  ------------------------------------------------------------------------------------------------------------- begin splitit ---
  - version 11.2
  - syntax varlist (min=5 max=5) [,SRT(varlist) PORtions(integer 0) CVars]
factor-variable and time-series operators not allowed
  --------------------------------------------------------------------------------------------------------------- end splitit ---

The problem seems to be that splitit expects a variable list (notice its syntax is asking for a list of five variables), but instead it is being passed a number, 0.8.

Indeed, you get the same problem if you run this directly:

. splitit 0.8
factor-variable and time-series operators not allowed
r(101);

because when it encounters 0.8 when it is expecting a variable name, it thinks you are trying to use factor variable notation. An integer would instead produce:

. splitit 1
1 invalid name
r(198);

You will need to see whether this is a bug with xv, or a problem with the way you are calling xv.

[Emma Williams-Baron]

Thanks very much, Hemanshu! I apologize, I forgot that I'd installed splitit through SSC.

In good news, I came across the command crossfold, also available through SSC, which seems to accomplish my CV goals.

I am still curious about why splitit is giving the factor variable notation error, since I'm following the Stata Community Corner syntax in the xv command: https://www.stata.com/stata-news/new...crossvalidate/ I suspect there might be a bug in how xv is sending the CV details to splitit, perhaps connected to the updated splitit2 package (?). But since I can use crossfold instead, I am content to live with the mystery.

Thank you again!