Stata H2O - any way to compute residuals after rfregress using CV ?

[Matteo Pinna Pintor]

Dear list members,

I'm exploring the new h2o interface. Perhaps unsurprisingly, focus is overwhelming on point prediction. I was wondering if anyone knows if it's possible to access individual in-sample predictions to compute residuals when using cross-validation. I can't see anything in this sense in the postestimation suite.


Many thanks in advance

[Matteo Pinna Pintor]

no one knows this? Maybe some Stata Corp staff?

[Aram Dallakyan (StataCorp)]

Dear Matteo,
Currently, the individual fold models and the predictions they produced are not available from the postestimation suite. What is available is

h2omlestat cvsummary

which reports the performance metrics for each fold along with their mean and standard deviation. If you supply your own fold variable through cv(colname) you at least know which rows went where, but the fold models themselves are not available.

If you are interested in the out-of-fold predictions, you can build them by hand with _h2oframe split and _h2oframe rbind plus one fit per fold. I am happy to post the details if that would help.

[Matteo Pinna Pintor]

Dear Aram,

as I feared. Yes I noted the CV summaries, but I wanted the observation-level data. And I also though that by perhaps, by treating each of my folds - currently defined in a variable, which I use for cv(colname) - as a validation frame, for which I saw that predictions could be obtained (correct me if I'm wrong) I might retrieve them. But let me explain the motivation - as maybe you could suggest better means to my end. I'd like some measure of in-sample uncertainty, and I was thinking that variation in fold-specific predictions could do. I know this would not incorporate out-of-sample (or "out-of-bag", I guess) uncertainty, but that's fine. For context: this is a cross-country panel, with multi-year averaging and country-grouped CV to address autocorrelation (or leakage, as per the ML jargon).

[Aram Dallakyan (StataCorp)]

Based on my understanding of your question, I put together a do-file that generates a toy example that mimics a country panel with country-grouped folds, shows how to compute an out-of-fold prediction, residual for every observation, and the RMSE. Let me know if you have questions.

[Matteo Pinna Pintor]

Dear Aram,

thanks a lot. Indeed, that's sort of what I had in mind, but wasn't entirely sure about. A theory question, if I may: do you know of reliable approaches to prediction intervals after random forest? Is there anything that could be based on out-of-fold predictions?

[Matteo Pinna Pintor]

Dear Aram,

thanks a lot. Indeed, that's sort of what I had in mind, but wasn't entirely sure about. A theory question, if I may: do you know of reliable approaches to prediction intervals after random forest? Is there anything that could be based on out-of-fold predictions?

I realized that this is the ground of conformal inference, and I see a lot of material is under development. Unfortunately, I also see that there's little directly available in Stata - although I can use the python integration. A more serious concern is that these methods require further held-out data - which in my case (a cross-country panel restricted and aggregated in various ways, with country-grouped CV to address leakage) is very likely to erode power.

[Aram Dallakyan (StataCorp)]

Dear Matteo,
I think conformal intervals are a good direction for your problem. We have a Stata blog post on obtaining prediction intervals with GBM for i.i.d data: https://blog.stata.com/2025/05/20/pr...sting-machine/. Unfortunately, I'm not very familiar with the literature for the type of data you're working with.