Stata H2O - any way to compute residuals after rfregress using CV ?

[Matteo Pinna Pintor]

Dear list members,

I'm exploring the new h2o interface. Perhaps unsurprisingly, focus is overwhelming on point prediction. I was wondering if anyone knows if it's possible to access individual in-sample predictions to compute residuals when using cross-validation. I can't see anything in this sense in the postestimation suite.


Many thanks in advance

[Matteo Pinna Pintor]

no one knows this? Maybe some Stata Corp staff?

[Aram Dallakyan]

Dear Matteo,
Currently, the individual fold models and the predictions they produced are not available from the postestimation suite. What is available is

h2omlestat cvsummary

which reports the performance metrics for each fold along with their mean and standard deviation. If you supply your own fold variable through cv(colname) you at least know which rows went where, but the fold models themselves are not available.

If you are interested in the out-of-fold predictions, you can build them by hand with _h2oframe split and _h2oframe rbind plus one fit per fold. I am happy to post the details if that would help.

[Matteo Pinna Pintor]

Dear Aram,

as I feared. Yes I noted the CV summaries, but I wanted the observation-level data. And I also though that by perhaps, by treating each of my folds - currently defined in a variable, which I use for cv(colname) - as a validation frame, for which I saw that predictions could be obtained (correct me if I'm wrong) I might retrieve them. But let me explain the motivation - as maybe you could suggest better means to my end. I'd like some measure of in-sample uncertainty, and I was thinking that variation in fold-specific predictions could do. I know this would not incorporate out-of-sample (or "out-of-bag", I guess) uncertainty, but that's fine. For context: this is a cross-country panel, with multi-year averaging and country-grouped CV to address autocorrelation (or leakage, as per the ML jargon).