Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Writing results to file when model fails to converge

    Hello list users. I am hoping someone has tired to do something all the lines of the following. I am running unadjusted (exploratory) analyses, estimating the associations between many predictors and one outcome of interest. It's a matched case-control study (although that in itself is not important), and all predictors (for now) are dichotomous. No other confounders are initially in the model. I'm writing code to loop through all the predictors, fitting each in turn and storing the estimated effects and standard error in a results file for later inspection.

    As is to be expected, not every model will converge, and so where this happens the model currently being fitted loops indefinitely. Rather than having to identify each of these in turn, and restart, I'd like to amend the code so that, if convergence is not found within eg 100 iterations, the model fitting process moves onto the next predictor. The estimated coefficients in the output file could simply be left as missing or an error message written to the outfile file in another variable (e.g. "Non convergence"). Is this a relatively straightforward process? I've posted some sample code below which may help-it will run where every model converges, but goes into an infinite loop once non-convergence appears. Many thanks for any advice offered.


    Code:
    use test_dataset, clear
    describe
        
    /*    
    Contains data from test_dataset
      obs:         6,312                          
     vars:           1002                        
                      
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
                  storage   display    value
    variable name   type    format     label      variable label
    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    case_control_no float   %9.0g              Case control status (1=with outcome, 2=outcome)
    case_studyid                                        matched sets identifier
    exposure1       float   %9.0g                 1 exposure (binary Y/N)
    exposure2       float   %9.0g                 2 exposure (binary Y/N)
    exposure3       float   %9.0g                 3 exposure (binary Y/N)
    exposure4       float   %9.0g                 4 exposure (binary Y/N)
    ETC
    */
        
    postutil clear
    postfile temp    exposure  logor se   using results_dataset, replace
        
    forvalues i=1(1)1000{
        local a `i'
        
        /*unadjusted analyses-any  exposure*/
        clogit case_control_no i.exposure`i'  , group(case_studyid)  
        local logor=_b[bi_ingred_tally_med`i']
        matrix A=e(V)
        local se=sqrt(A[1,1])
        matrix drop A
            
        post temp  (`a') (`logor') (`se')
        postclose temp    
    }



  • #2
    Originally posted by Ronald McDowell View Post
    ... I'd like to amend the code so that, if convergence is not found within eg 100 iterations, the model fitting process moves onto the next predictor. The estimated coefficients in the output file could simply be left as missing or an error message written to the outfile file in another variable (e.g. "Non convergence"). ...

    Code:
    ...
    
    postutil clear
    postfile temp exposure logor se using results_dataset, replace
    
    forvalues i=1(1)1000{
    local a `i'
    
    /*unadjusted analyses-any exposure*/
    clogit case_control_no i.exposure`i' , group(case_studyid) iterate(100)
    local logor=_b[bi_ingred_tally_med`i']
    matrix A=e(V)
    local se=sqrt(A[1,1])
    matrix drop A
    local converged = e(converged)
    
    post temp (`a') (`logor') (`se') (`converged')
    postclose temp
    }
    See if the above code snippets (in bold) will do it for you. As you must know, all Stata programs spit out a list of scalars and matrices prefixed by e() or r() when they finish running (i.e. they converge, or they hit the maximum number of iterations specified in iterate(##)). It turns out that one of those scalars is called e(converged). It's meaning should be obvious.
    Be aware that it can be very hard to answer a question without sample data. You can use the dataex command for this. Type help dataex at the command line.

    When presenting code or results, please use the code delimiters format them. Use the # button on the formatting toolbar, between the " (double quote) and <> buttons.

    Comment


    • #3
      Hello-thanks for your reply. Everything is working fine now-I had come across the e(converged) macro but hadn't figured out adding the iterate option. Thanks again.

      Comment


      • #4
        On a followup point, sometimes running a model may result in an error message, rather than non-convergence, leading the looping to stop e.g. redundant or inconsistent constraints
        r(412). It is possible to amend the code so that in this instance the error message is outputted and the looping continues? Thank you

        Comment


        • #5
          Try the built-in -capture- prefix. It suppresses output, traps any error, and allows you to obtain the error code and continue:

          Code:
          forval i = 1/1000 {
             capture MyModelCommand ...
             if (_rc != 0) {
                display "error " _rc  " occurred; going to next rep"
             }
          }

          Comment


          • #6
            Thanks, that's great!

            Comment

            Working...
            X