Validating individual coefficients in a (causal) probit model

[Antonio Cabrita]

Hello everybody!

I am working on a causal probit model, so my focus is not prediction, I rather want to examine the sign and magnitude of individual coefficients.
As the sample size is pretty small, I want to do some kind of validation in order to check for / avoid overfitting.

If I understand correctly commands like crossfold focus on the predictive power of a model: Statistics like root mean squared error (RMSE) or psuedo-R² inform about the model as a whole, not the individual regressors.

Given my interest in the individual variables, I want to validate my causal models like this:

*My main model is:
probit Success x1 x2 x3

*Randomly dividing data into two samples each:
gen v1 = rbinomial(1,.75)
gen v2 = rbinomial(1,.75)
gen v3 = rbinomial(1,.75)
gen v4 = rbinomial(1,.75)
gen v5 = rbinomial(1,.75)

*Validation:
probit Success x1 x2 x3 if v1==1
probit Success x1 x2 x3 if v2==1
probit Success x1 x2 x3 if v3==1
probit Success x1 x2 x3 if v4==1
probit Success x1 x2 x3 if v5==1

Finally, I would now compare every single regressor in the five validation regressions with the main regression that draws on the whole data.

As I am a rookie, doing such an analysis for the first time, it would be great if someone could tell me if it's an appropriate way that I want to go.

Or have I overlooked a well-established method for validating models when the focus is causal analysis and hence on every single regressor?

Thank you very much for your help!
Kind regards
Antonio

[Maarten Buis]

Here are some ideas:

If you draw a random sample *with replacement*, you can sample N out of N observations. Since you have a small sample, making it smaller can add complications that aren't necessarily in your full data. Moreover, this is simple to implement in Stata, as this is just the bootstrap.

// open some example data
sysuse auto, clear

// make a temporary file to store the bootstrap replications
tempfile bs

// estimate your model in your regular sample and in a 1000 samples from your
// data stored in `bs'
probit foreign weight price, vce(bootstrap, saving(`bs', replace) reps(1000))

// open the file
use `bs'

// now you can inspect the results in whatever way you like

An approach that is somewhat similar is to look at how the estimates change when one observation is excluded. This allows you to directly identify problematic observations, and make informed decisions on what to do with them. The downside is that this obviously won't detect clusters of influential observations. So I see this as a complementary check. Here is a link on how to implement this for general estimation commands including probit: https://blog.stata.com/2014/05/08/us...ential-points/

Recently I wrote a program doing a similar test, but instead of removing each observation, it "flips" the observation, i.e. turns a 0 to a 1 and *vice versa*. Below is the binflipcheck.ado file

*! version 1.0.0 MLB 06Sept2018
program define binflipcheck, sortpreserve rclass
        version 15.1
        syntax [, stub(passthru) ll]

        capture assert inlist(e(cmd), "logit", "probit", "cloglog")
        if _rc {
                di as err "{p}binflipcheck can only be used after logit, probit, or cloglog{p_end}"
                exit 198
        }
        local cmdline "`e(cmdline)'"
        local y "`e(depvar)'"
        local n = e(N)

        parsex, `stub' `ll'
        local nvarl "`r(nvarl)'"
        local x     "`r(xvarl)'"
        local k      `r(k)'
        local llvar "`r(llvar)'"

        tempvar nottouse
        gen `nottouse' = !e(sample)
        sort `nottouse'
        
        nois _dots 0, title(estimating models) reps(`n')

        forvalues i = 1 / `n' {
                quietly {
                        replace `y' = 1 - `y' in `i'
                        `cmdline'
                        forvalues j = 1 / `k' {
                                local var  : word `j' of `nvarl'
                                local xvar : word `j' of `x'
                                if e(converged) == 1 {
                                        replace `var' = _b[`xvar'] in `i'
                                }
                                else {
                                        replace `var' = .b in `i'
                                }
                        }
                        if "`ll'" != "" {
                                if e(converged) == 1 {
                                        replace `llvar' = e(ll) in `i'
                                }
                                else {
                                        replace `llvar' = .b in `i'
                                }
                        }
                        replace `y' = 1 - `y' in `i'
                        nois _dots `i' `=!e(converged)'
                }
        }
        
        label define _binflipcheck .a "not in estimation sample" ///
                                  .b "not converged", replace
        label value `nvarl' `llvar' _binflipcheck
        
        qui `cmdline'
        return local newvars `nvarl' `llvar'
end


program define parsex, rclass
        syntax [, stub(string) ll]
        if `"`stub'"' == "" {
                local stub "b_"
        }
        local x : colvarlist e(b)
        local cons "_cons"
        local hascons : list cons in x
        local x : list x - cons
        _ms_extract_varlist `x', noomitted
        local x `r(varlist)'
        local i = 0
        foreach var of local x {
                local varname`++i' : subinstr local var "." "_", all
                local varname`i'   : subinstr local varname`i' "#" "_X_", all
                local varname`i' "`stub'`varname`i''"
                qui gen double `varname`i'' = .a if !e(sample)
                label var `varname`i'' "coefficient for `var'"
                local nvarl "`nvarl' `varname`i''"
                local k = `i'
        }
        if `hascons' {
                qui gen double `stub'cons = .a if !e(sample)
                label var `stub'cons "coefficient for the constant"
                local nvarl "`nvarl' `stub'cons"
                local x "`x' _cons"
                local k = `k' + 1
        }
        if "`ll'" != "" {
                qui gen double `stub'll = .a if !e(sample)
                label var `stub'll "log likelihood"
                return local llvar `stub'll
        }
        return local  nvarl `nvarl'
        return local  xvarl `x'
        return scalar k = `k'
end

[Antonio Cabrita]

Thank you very much, Maarten! And greetings to Konstanz, I have studied there a few years ago

So, I think to start with I'll use the bootstrap.

Could you please tell me what the usual checks are done after a bootstrap?
I already inspected that the coefficients loose significance when I run an apparently too complicated model with too many regressors.

So, is it enough to look at the changing p-value or do you normally also inspect summary statistics like the mean or percentiles of the replicated coefficients?
And should I also check the change in the overall chi square for the model or is it negligible when my focus is on causal analysis and thus the individual regressors?

Thanks a lot!
Antonio

[Maarten Buis]

You started out by saying that you were interested in the effects of the regressors, so that is what you look at. They need to be "stable" in the different replications, that is, they should not differ "too much" from the parameter in the original model. The difficulty is determining how much is too much. My rule of thumb is that if I were to verbalize that effect and I would use different words to describe that effect, then it is too much. Another issue is that if you to a 1000 replications you may be willing to tolerate "some" unusual replications before you call the model unstable, but that leaves the question of how many is too many. Some models are clearly stable and others clearly unstable, the hard part are those models in between. There is no hard rule for those.

[Antonio Cabrita]

Thanks Maarten! It makes sense to me that there is no hard rule.

And in practical terms, I use tab or sum, detail to investigate on the bootstraped regressors or is there a better approach to do this?