Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • zinb model not converging. gradient value near zero.

    Dear All

    I am running below model on my dataset

    Code:
     zinb dv control1 control2 control3 control4 iv, inflate(control3 control4 iv)
    However, it does not converges after 300 iteration. Value of log likelihood does not change from iteration 11 onwards.

    I got below information from Stata help manual

    If a “not concave” message appears at the last step, there are two possibilities. One is that the result is valid, but there is collinearity in the model that the command did not otherwise catch. Stata checks for obvious collinearity among the independent variables before performing the maximization, but strange collinearities or near collinearities can sometimes arise between coefficients and ancillary parameters. The second, more likely cause for a “not concave” message at the final step is that the optimizer entered a flat region of the likelihood and prematurely declared convergence.

    If a “backed up” message appears at the last step, there are also two possibilities. One is that Stata found a perfect maximum and could not step to a better point; if this is the case, all is fine, but this is a highly unlikely occurrence. The second is that the optimizer worked itself into a bad concave spot where the computed gradient and Hessian gave a bad direction for stepping.

    If either of these messages appears at the last step, perform the maximization again with the gradient option. If the gradient goes to zero, the optimizer has found a maximum that may not be unique but is a maximum. From the standpoint of maximum likelihood estimation, this is a valid result. If the gradient is not zero, it is not a valid result, and you should try tightening up the convergence criterion, or try ltol(0) tol(1e-7) to see if the optimizer can work its way out of the bad region.

    If you get repeated “not concave” steps with little progress being made at each step, try specifying the difficult option. Sometimes difficult works wonderfully, reducing the number of iterations and producing convergence at a good (that is, concave) point. Other times, difficult works poorly, taking much longer to converge than the default stepper.
    Based on this information, I changed my code to following

    Code:
     zinb dv control1 control2 control3 control4 iv, inflate(control3 control4 iv) difficult gradient
    Under 300th iteration, I got gradient vector (length: .0088744) with iv under the inflate part of model having value of -.0027402. (This is the largest value, not considering sign)

    My question.
    Can I treat gradient as zero and my results are valid or gradient values are expected to be much lower (like e-06/e-07) etc.

    Thanks

  • #2
    1. Do you experience convergence problems if you estimate the model with standard negative binomial regression, i.e.
    Code:
    nbreg dv control1 control2 control3 control4 iv
    There are well known reasons that standard negative binomial regression does not converge (e.g. underdispersion relative to a Poisson) and it's possible that this is what's causing non-convergence for you.

    2. Others may differ in their opinions but for me a convergence problem like the one you describe is a signal that the estimation method may not be well suited to your sample, like trying to put a square peg into a round hole. While this may not always be the case I feel it is a possibility usually worth considering.

    Comment


    • #3
      Once the change in the log likelihood (and, I believe, the second derivative of the LL) gets under a specified threshold, basically meaning essentially no change as you're at the maximum, Stata will declare convergence. Here, Stata didn't declare victory. So, I wouldn't be comfortable saying that the gradient for one coefficient seems small, so I manually declare victory - not unless I knew what all this meant, anyway.

      A zero-inflated model is, I think, a special case of a finite mixture model. I am familiar with latent class models, which are definitely a special case of FMMs. If I read right, your LL seems to hit an asymptote, but Stata keeps not declaring convergence, and it's probably also stating that the likelihood function is not concave. I have a feeling that one of the predictors of membership in the structural zero class (i.e. the class whose DV is always = 0) may have an unusual effect, like maybe it's perfectly predictive of membership in that class or something. You could check how many iterations the LL takes before its global value stabilizes, then tell Stata to do that many iterations, and then examine the results to see if one those predictors is problematic. (NB: If you do this, you are doing it only for error checking! The results are not final!!!) I guess you could consider deleting that predictor. I haven't run many zero inflated models, and I did not have this issue the one or two times I ran such a model.

      Note that this is a guess; I see this phenomenon in LCA not infrequently, and I can explain why that happens in LCAs. I am pretty sure the example doesn't translate exactly into FMMs. Hence, I only have a hunch about why this is happening, and unfortunately the only thing I can immediately think of is eliminating a predictor of structural zero class membership if it's problematic.

      More fundamentally, though, if you have no strong a priori reason to believe that there's a group of people who return structural zeroes, then it doesn't seem like you must fit a zero-inflated model. It seems like a non-zero inflated count model of some sort would likely be enough to answer whatever question you are posing.
      Be aware that it can be very hard to answer a question without sample data. You can use the dataex command for this. Type help dataex at the command line.

      When presenting code or results, please use the code delimiters format them. Use the # button on the formatting toolbar, between the " (double quote) and <> buttons.

      Comment


      • #4
        Hello John & Weiwen

        Thanks a lot for your responses.

        Two quick updates
        1. if I run nbreg, the model converges just fine.
        2. I, however, think that nbreg is not the right estimation tool for my data. It is because I have a lot of zeros in my DV. Primary reason for that is my IV (which is also a count variable) also have lot of zeros. Now if IV is zero, DV is zero by default but even if IV is non zero, DV can be zero.
        3. If I run a model using nbreg, where I avoid IV == 0, then my model runs fine and it also looks like good estimation but since I am doing mediation test, I (think) need should use full dataset.

        I am sharing below data sample, code and results.

        model variables: success, reci, backed
        control variables: staffpick, catgid, goalk


        Overall I have a mediation model.

        1. reci -> success (logit model, runs fine)
        2. backed -> reci (zinb model, does not converges)
        3. backed, reci -> success (logit model, runs fine)


        Also, if I change reci, which is currently a count data to a binary data (0,1) and use logit model to test backed -> reci relationship, model runs fine and results are as expected. I however, want to retain the data as collected, if possible

        Code:
        * Example generated by -dataex-. To install: ssc install dataex
        clear
        input byte(staffpick catgid) int backed byte reci int lyear byte success float goalk
        0  9 27 5 2019 1 12.5
        1 15  9 0 2018 1  2.2
        0  9  6 0 2018 0   26
        0  7  0 0 2017 0   10
        0  9 18 0 2013 0   72
        0 14  0 0 2018 0  250
        1  5  8 0 2018 1   20
        0  2  0 0 2018 0   .8
        1 14  0 0 2018 1   15
        0  3 28 0 2017 1   .8
        0 13 16 1 2018 1    5
        0 11  1 0 2018 1   20
        0 14  2 0 2018 0   30
        0 11  1 0 2018 0 1.59
        0  3  0 0 2018 0    5
        0  6  1 0 2018 1  .05
        0 13  0 0 2018 0   25
        0  9  0 0 2017 1   90
        0  1  0 0 2015 1    2
        1 11  4 0 2018 1  .35
        0  8  0 0 2018 0    8
        0  8  0 0 2018 0   12
        0 12  0 0 2018 0   .5
        0  1  0 0 2018 1 .577
        0 11  0 0 2018 0   .5
        end
        Code:
        zinb reci i.lyear i.catgid goalk staffpick backed, inflate(goalk staffpick backed)

        Code:
        . zinb reci i.lyear i.catgid goalk staffpick backed, inflate(goalk staffpick backed)
        
        Fitting constant-only model:
        
        Iteration 0:   log likelihood = -504.15597  (not concave)
        Iteration 1:   log likelihood =  -461.7766  
        Iteration 2:   log likelihood = -432.71256  
        Iteration 3:   log likelihood = -401.44469  
        Iteration 4:   log likelihood = -375.10903  
        Iteration 5:   log likelihood = -356.67088  
        Iteration 6:   log likelihood = -347.10027  
        Iteration 7:   log likelihood =  -335.1863  
        Iteration 8:   log likelihood = -333.67329  
        Iteration 9:   log likelihood = -332.96814  
        Iteration 10:  log likelihood = -332.85121  
        Iteration 11:  log likelihood = -332.81986  
        Iteration 12:  log likelihood = -332.81401  
        Iteration 13:  log likelihood = -332.81268  
        Iteration 14:  log likelihood = -332.81236  
        Iteration 15:  log likelihood = -332.81229  
        Iteration 16:  log likelihood = -332.81228  
        
        Fitting full model:
        
        Iteration 0:   log likelihood = -332.81228  
        Iteration 1:   log likelihood = -302.52987  (not concave)
        Iteration 2:   log likelihood = -300.97032  (not concave)
        Iteration 3:   log likelihood = -300.40752  (not concave)
        Iteration 4:   log likelihood = -300.12482  (not concave)
        Iteration 5:   log likelihood = -299.80705  (not concave)
        Iteration 6:   log likelihood = -299.79929  (not concave)
        Iteration 7:   log likelihood = -299.79622  (not concave)
        Iteration 8:   log likelihood = -299.79377  (not concave)
        Iteration 9:   log likelihood = -299.79353  (not concave)
        Iteration 10:  log likelihood = -299.79343  (not concave)
        Iteration 11:  log likelihood = -299.79342  (not concave)
        Iteration 12:  log likelihood = -299.79342  (not concave)
        Iteration 13:  log likelihood = -299.79342  (not concave)
        Iteration 14:  log likelihood = -299.79342  (not concave)
        Iteration 15:  log likelihood = -299.79342  (not concave)
        Iteration 16:  log likelihood = -299.79342  (not concave)
        Iteration 17:  log likelihood = -299.79342  (not concave)
        Iteration 18:  log likelihood = -299.79342  (not concave)
        Iteration 19:  log likelihood = -299.79342  (not concave)
        Iteration 20:  log likelihood = -299.79342  (not concave)
        Iteration 21:  log likelihood = -299.79342  (not concave)
        Iteration 22:  log likelihood = -299.79342  (not concave)
        Iteration 23:  log likelihood = -299.79342  (not concave)
        Iteration 24:  log likelihood = -299.79342  (not concave)
        Iteration 25:  log likelihood = -299.79342  (not concave)
        Iteration 26:  log likelihood = -299.79342  (not concave)
        Iteration 27:  log likelihood = -299.79342  (not concave)
        Iteration 28:  log likelihood = -299.79342  (not concave)
        Iteration 29:  log likelihood = -299.79342  (not concave)
        Iteration 30:  log likelihood = -299.79342  (not concave)
        Iteration 31:  log likelihood = -299.79342  (not concave)
        Iteration 32:  log likelihood = -299.79342  (not concave)
        Iteration 33:  log likelihood = -299.79342  (not concave)
        Iteration 34:  log likelihood = -299.79342  (not concave)
        Iteration 35:  log likelihood = -299.79342  (not concave)
        Iteration 36:  log likelihood = -299.79342  (not concave)
        Iteration 37:  log likelihood = -299.79342  (not concave)
        Iteration 38:  log likelihood = -299.79342  (not concave)
        Iteration 39:  log likelihood = -299.79342  (not concave)
        Iteration 40:  log likelihood = -299.79342  (not concave)
        Iteration 41:  log likelihood = -299.79342  (not concave)
        Iteration 42:  log likelihood = -299.79342  (not concave)
        Iteration 43:  log likelihood = -299.79342  (not concave)
        Iteration 44:  log likelihood = -299.79342  (not concave)
        Iteration 45:  log likelihood = -299.79342  (not concave)
        Iteration 46:  log likelihood = -299.79342  (not concave)
        Iteration 47:  log likelihood = -299.79342  (not concave)
        Iteration 48:  log likelihood = -299.79342  (not concave)
        Iteration 49:  log likelihood = -299.79342  (not concave)
        Iteration 50:  log likelihood = -299.79342  (not concave)
        Iteration 51:  log likelihood = -299.79342  (not concave)
        Iteration 52:  log likelihood = -299.79342  (not concave)
        Iteration 53:  log likelihood = -299.79342  (not concave)
        Iteration 54:  log likelihood = -299.79342  (not concave)
        Iteration 55:  log likelihood = -299.79342  (not concave)
        Iteration 56:  log likelihood = -299.79342  (not concave)
        Iteration 57:  log likelihood = -299.79342  (not concave)
        Iteration 58:  log likelihood = -299.79342  (not concave)
        Iteration 59:  log likelihood = -299.79342  (not concave)
        Iteration 60:  log likelihood = -299.79342  (not concave)
        Iteration 61:  log likelihood = -299.79342  (not concave)
        Iteration 62:  log likelihood = -299.79342  (not concave)
        Iteration 63:  log likelihood = -299.79342  (not concave)
        Iteration 64:  log likelihood = -299.79342  (not concave)
        Iteration 65:  log likelihood = -299.79342  (not concave)
        Iteration 66:  log likelihood = -299.79342  (not concave)
        Iteration 67:  log likelihood = -299.79342  (not concave)
        Iteration 68:  log likelihood = -299.79342  (not concave)
        Iteration 69:  log likelihood = -299.79342  (not concave)
        Iteration 70:  log likelihood = -299.79342  (not concave)
        Iteration 71:  log likelihood = -299.79342  (not concave)
        Iteration 72:  log likelihood = -299.79342  (not concave)
        Iteration 73:  log likelihood = -299.79342  (not concave)
        Iteration 74:  log likelihood = -299.79342  (not concave)
        Iteration 75:  log likelihood = -299.79342  (not concave)
        Iteration 76:  log likelihood = -299.79342  (not concave)
        Iteration 77:  log likelihood = -299.79342  (not concave)
        Iteration 78:  log likelihood = -299.79342  (not concave)
        Iteration 79:  log likelihood = -299.79342  (not concave)
        Iteration 80:  log likelihood = -299.79342  (not concave)
        Iteration 81:  log likelihood = -299.79342  (not concave)
        Iteration 82:  log likelihood = -299.79342  (not concave)
        Iteration 83:  log likelihood = -299.79342  (not concave)
        Iteration 84:  log likelihood = -299.79342  (not concave)
        Iteration 85:  log likelihood = -299.79342  (not concave)
        Iteration 86:  log likelihood = -299.79342  (not concave)
        Iteration 87:  log likelihood = -299.79342  (not concave)
        Iteration 88:  log likelihood = -299.79342  (not concave)
        Iteration 89:  log likelihood = -299.79342  (not concave)
        Iteration 90:  log likelihood = -299.79342  (not concave)
        Iteration 91:  log likelihood = -299.79342  (not concave)
        Iteration 92:  log likelihood = -299.79342  (not concave)
        Iteration 93:  log likelihood = -299.79342  (not concave)
        Iteration 94:  log likelihood = -299.79342  (not concave)
        Iteration 95:  log likelihood = -299.79342  (not concave)
        Iteration 96:  log likelihood = -299.79342  (not concave)
        Iteration 97:  log likelihood = -299.79342  (not concave)
        Iteration 98:  log likelihood = -299.79342  (not concave)
        Iteration 99:  log likelihood = -299.79342  (not concave)
        Iteration 100: log likelihood = -299.79342  (not concave)
        Iteration 101: log likelihood = -299.79342  (not concave)
        Iteration 102: log likelihood = -299.79342  (not concave)
        Iteration 103: log likelihood = -299.79342  (not concave)
        Iteration 104: log likelihood = -299.79342  (not concave)
        Iteration 105: log likelihood = -299.79342  (not concave)
        Iteration 106: log likelihood = -299.79342  (not concave)
        Iteration 107: log likelihood = -299.79342  (not concave)
        Iteration 108: log likelihood = -299.79342  (not concave)
        Iteration 109: log likelihood = -299.79342  (not concave)
        Iteration 110: log likelihood = -299.79342  (not concave)
        Iteration 111: log likelihood = -299.79342  (not concave)
        Iteration 112: log likelihood = -299.79342  (not concave)
        Iteration 113: log likelihood = -299.79342  (not concave)
        Iteration 114: log likelihood = -299.79342  (not concave)
        Iteration 115: log likelihood = -299.79342  (not concave)
        Iteration 116: log likelihood = -299.79342  (not concave)
        Iteration 117: log likelihood = -299.79342  (not concave)
        Iteration 118: log likelihood = -299.79342  (not concave)
        Iteration 119: log likelihood = -299.79342  (not concave)
        Iteration 120: log likelihood = -299.79342  (not concave)
        Iteration 121: log likelihood = -299.79342  (not concave)
        Iteration 122: log likelihood = -299.79342  (not concave)
        Iteration 123: log likelihood = -299.79342  (not concave)
        Iteration 124: log likelihood = -299.79342  (not concave)
        Iteration 125: log likelihood = -299.79342  (not concave)
        Iteration 126: log likelihood = -299.79342  (not concave)
        Iteration 127: log likelihood = -299.79342  (not concave)
        Iteration 128: log likelihood = -299.79342  (not concave)
        Iteration 129: log likelihood = -299.79342  (not concave)
        Iteration 130: log likelihood = -299.79342  (not concave)
        Iteration 131: log likelihood = -299.79342  (not concave)
        Iteration 132: log likelihood = -299.79342  (not concave)
        Iteration 133: log likelihood = -299.79342  (not concave)
        Iteration 134: log likelihood = -299.79342  (not concave)
        Iteration 135: log likelihood = -299.79342  (not concave)
        Iteration 136: log likelihood = -299.79342  (not concave)
        Iteration 137: log likelihood = -299.79342  (not concave)
        Iteration 138: log likelihood = -299.79342  (not concave)
        Iteration 139: log likelihood = -299.79342  (not concave)
        Iteration 140: log likelihood = -299.79342  (not concave)
        Iteration 141: log likelihood = -299.79342  (not concave)
        Iteration 142: log likelihood = -299.79342  (not concave)
        Iteration 143: log likelihood = -299.79342  (not concave)
        Iteration 144: log likelihood = -299.79342  (not concave)
        Iteration 145: log likelihood = -299.79342  (not concave)
        Iteration 146: log likelihood = -299.79342  (not concave)
        Iteration 147: log likelihood = -299.79342  (not concave)
        Iteration 148: log likelihood = -299.79342  (not concave)
        Iteration 149: log likelihood = -299.79342  (not concave)
        Iteration 150: log likelihood = -299.79342  (not concave)
        Iteration 151: log likelihood = -299.79342  (not concave)
        Iteration 152: log likelihood = -299.79342  (not concave)
        Iteration 153: log likelihood = -299.79342  (not concave)
        Iteration 154: log likelihood = -299.79342  (not concave)
        Iteration 155: log likelihood = -299.79342  (not concave)
        Iteration 156: log likelihood = -299.79342  (not concave)
        Iteration 157: log likelihood = -299.79342  (not concave)
        Iteration 158: log likelihood = -299.79342  (not concave)
        Iteration 159: log likelihood = -299.79342  (not concave)
        Iteration 160: log likelihood = -299.79342  (not concave)
        Iteration 161: log likelihood = -299.79342  (not concave)
        Iteration 162: log likelihood = -299.79342  (not concave)
        Iteration 163: log likelihood = -299.79342  (not concave)
        Iteration 164: log likelihood = -299.79342  (not concave)
        Iteration 165: log likelihood = -299.79342  (not concave)
        Iteration 166: log likelihood = -299.79342  (not concave)
        Iteration 167: log likelihood = -299.79342  (not concave)
        Iteration 168: log likelihood = -299.79342  (not concave)
        Iteration 169: log likelihood = -299.79342  (not concave)
        Iteration 170: log likelihood = -299.79342  (not concave)
        Iteration 171: log likelihood = -299.79342  (not concave)
        Iteration 172: log likelihood = -299.79342  (not concave)
        Iteration 173: log likelihood = -299.79342  (not concave)
        Iteration 174: log likelihood = -299.79342  (not concave)
        Iteration 175: log likelihood = -299.79342  (not concave)
        Iteration 176: log likelihood = -299.79342  (not concave)
        Iteration 177: log likelihood = -299.79342  (not concave)
        Iteration 178: log likelihood = -299.79342  (not concave)
        Iteration 179: log likelihood = -299.79342  (not concave)
        Iteration 180: log likelihood = -299.79342  (not concave)
        Iteration 181: log likelihood = -299.79342  (not concave)
        Iteration 182: log likelihood = -299.79342  (not concave)
        Iteration 183: log likelihood = -299.79342  (not concave)
        Iteration 184: log likelihood = -299.79342  (not concave)
        Iteration 185: log likelihood = -299.79342  (not concave)
        Iteration 186: log likelihood = -299.79342  (not concave)
        Iteration 187: log likelihood = -299.79342  (not concave)
        Iteration 188: log likelihood = -299.79342  (not concave)
        Iteration 189: log likelihood = -299.79342  (not concave)
        Iteration 190: log likelihood = -299.79342  (not concave)
        Iteration 191: log likelihood = -299.79342  (not concave)
        Iteration 192: log likelihood = -299.79342  (not concave)
        Iteration 193: log likelihood = -299.79342  (not concave)
        Iteration 194: log likelihood = -299.79342  (not concave)
        Iteration 195: log likelihood = -299.79342  (not concave)
        Iteration 196: log likelihood = -299.79342  (not concave)
        Iteration 197: log likelihood = -299.79342  (not concave)
        Iteration 198: log likelihood = -299.79342  (not concave)
        Iteration 199: log likelihood = -299.79342  (not concave)
        Iteration 200: log likelihood = -299.79342  (not concave)
        Iteration 201: log likelihood = -299.79342  (not concave)
        Iteration 202: log likelihood = -299.79342  (not concave)
        Iteration 203: log likelihood = -299.79342  (not concave)
        Iteration 204: log likelihood = -299.79342  (not concave)
        Iteration 205: log likelihood = -299.79342  (not concave)
        Iteration 206: log likelihood = -299.79342  (not concave)
        Iteration 207: log likelihood = -299.79342  (not concave)
        Iteration 208: log likelihood = -299.79342  (not concave)
        Iteration 209: log likelihood = -299.79342  (not concave)
        Iteration 210: log likelihood = -299.79342  (not concave)
        Iteration 211: log likelihood = -299.79342  (not concave)
        Iteration 212: log likelihood = -299.79342  (not concave)
        Iteration 213: log likelihood = -299.79342  (not concave)
        Iteration 214: log likelihood = -299.79342  (not concave)
        Iteration 215: log likelihood = -299.79342  (not concave)
        Iteration 216: log likelihood = -299.79342  (not concave)
        Iteration 217: log likelihood = -299.79342  (not concave)
        Iteration 218: log likelihood = -299.79342  (not concave)
        Iteration 219: log likelihood = -299.79342  (not concave)
        Iteration 220: log likelihood = -299.79342  (not concave)
        Iteration 221: log likelihood = -299.79342  (not concave)
        Iteration 222: log likelihood = -299.79342  (not concave)
        Iteration 223: log likelihood = -299.79342  (not concave)
        Iteration 224: log likelihood = -299.79342  (not concave)
        Iteration 225: log likelihood = -299.79342  (not concave)
        Iteration 226: log likelihood = -299.79342  (not concave)
        Iteration 227: log likelihood = -299.79342  (not concave)
        Iteration 228: log likelihood = -299.79342  (not concave)
        Iteration 229: log likelihood = -299.79342  (not concave)
        Iteration 230: log likelihood = -299.79342  (not concave)
        Iteration 231: log likelihood = -299.79342  (not concave)
        Iteration 232: log likelihood = -299.79342  (not concave)
        Iteration 233: log likelihood = -299.79342  (not concave)
        Iteration 234: log likelihood = -299.79342  (not concave)
        Iteration 235: log likelihood = -299.79342  (not concave)
        Iteration 236: log likelihood = -299.79342  (not concave)
        Iteration 237: log likelihood = -299.79342  (not concave)
        Iteration 238: log likelihood = -299.79342  (not concave)
        Iteration 239: log likelihood = -299.79342  (not concave)
        Iteration 240: log likelihood = -299.79342  (not concave)
        Iteration 241: log likelihood = -299.79342  (not concave)
        Iteration 242: log likelihood = -299.79342  (not concave)
        Iteration 243: log likelihood = -299.79342  (not concave)
        Iteration 244: log likelihood = -299.79342  (not concave)
        Iteration 245: log likelihood = -299.79342  (not concave)
        Iteration 246: log likelihood = -299.79342  (not concave)
        Iteration 247: log likelihood = -299.79342  (not concave)
        Iteration 248: log likelihood = -299.79342  (not concave)
        Iteration 249: log likelihood = -299.79342  (not concave)
        Iteration 250: log likelihood = -299.79342  (not concave)
        Iteration 251: log likelihood = -299.79342  (not concave)
        Iteration 252: log likelihood = -299.79342  (not concave)
        Iteration 253: log likelihood = -299.79342  (not concave)
        Iteration 254: log likelihood = -299.79342  (not concave)
        Iteration 255: log likelihood = -299.79342  (not concave)
        Iteration 256: log likelihood = -299.79342  (not concave)
        Iteration 257: log likelihood = -299.79342  (not concave)
        Iteration 258: log likelihood = -299.79342  (not concave)
        Iteration 259: log likelihood = -299.79342  (not concave)
        Iteration 260: log likelihood = -299.79342  (not concave)
        Iteration 261: log likelihood = -299.79342  (not concave)
        Iteration 262: log likelihood = -299.79342  (not concave)
        Iteration 263: log likelihood = -299.79342  (not concave)
        Iteration 264: log likelihood = -299.79342  (not concave)
        Iteration 265: log likelihood = -299.79342  (not concave)
        Iteration 266: log likelihood = -299.79342  (not concave)
        Iteration 267: log likelihood = -299.79342  (not concave)
        Iteration 268: log likelihood = -299.79342  (not concave)
        Iteration 269: log likelihood = -299.79342  (not concave)
        Iteration 270: log likelihood = -299.79342  (not concave)
        Iteration 271: log likelihood = -299.79342  (not concave)
        Iteration 272: log likelihood = -299.79342  (not concave)
        Iteration 273: log likelihood = -299.79342  (not concave)
        Iteration 274: log likelihood = -299.79342  (not concave)
        Iteration 275: log likelihood = -299.79342  (not concave)
        Iteration 276: log likelihood = -299.79342  (not concave)
        Iteration 277: log likelihood = -299.79342  (not concave)
        Iteration 278: log likelihood = -299.79342  (not concave)
        Iteration 279: log likelihood = -299.79342  (not concave)
        Iteration 280: log likelihood = -299.79342  (not concave)
        Iteration 281: log likelihood = -299.79342  (not concave)
        Iteration 282: log likelihood = -299.79342  (not concave)
        Iteration 283: log likelihood = -299.79342  (not concave)
        Iteration 284: log likelihood = -299.79342  (not concave)
        Iteration 285: log likelihood = -299.79342  (not concave)
        Iteration 286: log likelihood = -299.79342  (not concave)
        Iteration 287: log likelihood = -299.79342  (not concave)
        Iteration 288: log likelihood = -299.79342  (not concave)
        Iteration 289: log likelihood = -299.79342  (not concave)
        Iteration 290: log likelihood = -299.79342  (not concave)
        Iteration 291: log likelihood = -299.79342  (not concave)
        Iteration 292: log likelihood = -299.79342  (not concave)
        Iteration 293: log likelihood = -299.79342  (not concave)
        Iteration 294: log likelihood = -299.79342  (not concave)
        Iteration 295: log likelihood = -299.79342  (not concave)
        Iteration 296: log likelihood = -299.79342  (not concave)
        Iteration 297: log likelihood = -299.79342  (not concave)
        Iteration 298: log likelihood = -299.79342  (not concave)
        Iteration 299: log likelihood = -299.79342  (not concave)
        Iteration 300: log likelihood = -299.79342  (not concave)
        convergence not achieved
        
        Zero-inflated negative binomial regression      Number of obs     =        766
                                                        Nonzero obs       =        112
                                                        Zero obs          =        654
        
        Inflation model = logit                         LR chi2(27)       =      66.04
        Log likelihood  = -299.7934                     Prob > chi2       =     0.0000
        
        ------------------------------------------------------------------------------
                reci |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
        -------------+----------------------------------------------------------------
        reci         |
               lyear |
               2010  |          0  (base)
               2011  |  -.5697149    1.61732    -0.35   0.725    -3.739604    2.600174
               2012  |  -1.251007   1.397654    -0.90   0.371    -3.990359    1.488345
               2013  |  -1.445455   1.413003    -1.02   0.306    -4.214891     1.32398
               2014  |  -.9132214   1.351842    -0.68   0.499    -3.562783     1.73634
               2015  |  -1.363519   1.366603    -1.00   0.318    -4.042011    1.314974
               2016  |  -.9699652    1.33526    -0.73   0.468    -3.587027    1.647096
               2017  |    -1.1248   1.348204    -0.83   0.404    -3.767232    1.517632
               2018  |   -1.02616   1.305085    -0.79   0.432    -3.584079    1.531759
               2019  |  -.3138871    1.45639    -0.22   0.829    -3.168359    2.540585
               2020  |   28.57122          .        .       .            .           .
                     |
              catgid |
                  1  |          0  (base)
                  2  |   .7389907   .4404267     1.68   0.093    -.1242297    1.602211
                  3  |  -.2855641   .7311172    -0.39   0.696    -1.718528    1.147399
                  4  |  -4.396396   6.909079    -0.64   0.525    -17.93794     9.14515
                  5  |  -.2751012    .433541    -0.63   0.526    -1.124826    .5746235
                  6  |  -.0048664   .4746102    -0.01   0.992    -.9350853    .9253526
                  7  |   .3717786    .433117     0.86   0.391    -.4771152    1.220672
                  8  |   .1682736   .5304682     0.32   0.751     -.871425    1.207972
                  9  |   .1577414   .3720065     0.42   0.672     -.571378    .8868607
                 10  |   .7142468   1.306129     0.55   0.584    -1.845719    3.274212
                 11  |  -.2344147   .4898845    -0.48   0.632    -1.194571    .7257413
                 12  |   .4798022   .8049038     0.60   0.551     -1.09778    2.057385
                 13  |   .0491881   .4832787     0.10   0.919    -.8980207    .9963969
                 14  |   -.351027   .5814064    -0.60   0.546    -1.490563    .7885087
                 15  |  -4.485073   5.760406    -0.78   0.436    -15.77526    6.805116
                     |
               goalk |   .0012595   .0040594     0.31   0.756    -.0066968    .0092158
           staffpick |   .8313827   .2534079     3.28   0.001     .3347123    1.328053
              backed |   .0221881   .0061052     3.63   0.000     .0102221     .034154
               _cons |   .2489412   1.332415     0.19   0.852    -2.362544    2.860426
        -------------+----------------------------------------------------------------
        inflate      |
               goalk |  -.0084885   .0212762    -0.40   0.690     -.050189    .0332121
           staffpick |   1.316157   1.168394     1.13   0.260    -.9738527    3.606167
              backed |  -23.60358   46.78656    -0.50   0.614    -115.3036    68.09639
               _cons |   22.83846    46.8194     0.49   0.626    -68.92588    114.6028
        -------------+----------------------------------------------------------------
            /lnalpha |  -.5218386   .2926176    -1.78   0.075    -1.095359    .0516815
        -------------+----------------------------------------------------------------
               alpha |   .5934285   .1736476                      .3344197     1.05304
        ------------------------------------------------------------------------------
        convergence not achieved
        r(430);

        Comment


        • #5
          Originally posted by Chandresh Baid View Post
          Hello John & Weiwen

          Thanks a lot for your responses.

          Two quick updates
          1. if I run nbreg, the model converges just fine.
          2. I, however, think that nbreg is not the right estimation tool for my data. It is because I have a lot of zeros in my DV. Primary reason for that is my IV (which is also a count variable) also have lot of zeros. Now if IV is zero, DV is zero by default but even if IV is non zero, DV can be zero.
          3. If I run a model using nbreg, where I avoid IV == 0, then my model runs fine and it also looks like good estimation but since I am doing mediation test, I (think) need should use full dataset.

          I am sharing below data sample, code and results.

          model variables: success, reci, backed
          control variables: staffpick, catgid, goalk


          Overall I have a mediation model.

          1. reci -> success (logit model, runs fine)
          2. backed -> reci (zinb model, does not converges)
          3. backed, reci -> success (logit model, runs fine)

          ...
          Zero-inflated negative binomial regression Number of obs = 766
          Nonzero obs = 112
          Zero obs = 654

          Inflation model = logit LR chi2(27) = 66.04
          Log likelihood = -299.7934 Prob > chi2 = 0.0000

          ------------------------------------------------------------------------------
          reci | Coef. Std. Err. z P>|z| [95% Conf. Interval]
          -------------+----------------------------------------------------------------
          reci |
          lyear |
          2010 | 0 (base)
          2011 | -.5697149 1.61732 -0.35 0.725 -3.739604 2.600174
          2012 | -1.251007 1.397654 -0.90 0.371 -3.990359 1.488345
          2013 | -1.445455 1.413003 -1.02 0.306 -4.214891 1.32398
          2014 | -.9132214 1.351842 -0.68 0.499 -3.562783 1.73634
          2015 | -1.363519 1.366603 -1.00 0.318 -4.042011 1.314974
          2016 | -.9699652 1.33526 -0.73 0.468 -3.587027 1.647096
          2017 | -1.1248 1.348204 -0.83 0.404 -3.767232 1.517632
          2018 | -1.02616 1.305085 -0.79 0.432 -3.584079 1.531759
          2019 | -.3138871 1.45639 -0.22 0.829 -3.168359 2.540585
          2020 | 28.57122 . . . . .
          |
          catgid |
          1 | 0 (base)
          2 | .7389907 .4404267 1.68 0.093 -.1242297 1.602211
          3 | -.2855641 .7311172 -0.39 0.696 -1.718528 1.147399
          4 | -4.396396 6.909079 -0.64 0.525 -17.93794 9.14515
          5 | -.2751012 .433541 -0.63 0.526 -1.124826 .5746235
          6 | -.0048664 .4746102 -0.01 0.992 -.9350853 .9253526
          7 | .3717786 .433117 0.86 0.391 -.4771152 1.220672
          8 | .1682736 .5304682 0.32 0.751 -.871425 1.207972
          9 | .1577414 .3720065 0.42 0.672 -.571378 .8868607
          10 | .7142468 1.306129 0.55 0.584 -1.845719 3.274212
          11 | -.2344147 .4898845 -0.48 0.632 -1.194571 .7257413
          12 | .4798022 .8049038 0.60 0.551 -1.09778 2.057385
          13 | .0491881 .4832787 0.10 0.919 -.8980207 .9963969
          14 | -.351027 .5814064 -0.60 0.546 -1.490563 .7885087
          15 | -4.485073 5.760406 -0.78 0.436 -15.77526 6.805116
          |
          goalk | .0012595 .0040594 0.31 0.756 -.0066968 .0092158
          staffpick | .8313827 .2534079 3.28 0.001 .3347123 1.328053
          backed | .0221881 .0061052 3.63 0.000 .0102221 .034154
          _cons | .2489412 1.332415 0.19 0.852 -2.362544 2.860426
          -------------+----------------------------------------------------------------
          inflate |
          goalk | -.0084885 .0212762 -0.40 0.690 -.050189 .0332121
          staffpick | 1.316157 1.168394 1.13 0.260 -.9738527 3.606167
          backed | -23.60358 46.78656 -0.50 0.614 -115.3036 68.09639
          _cons | 22.83846 46.8194 0.49 0.626 -68.92588 114.6028
          -------------+----------------------------------------------------------------
          /lnalpha | -.5218386 .2926176 -1.78 0.075 -1.095359 .0516815
          -------------+----------------------------------------------------------------
          alpha | .5934285 .1736476 .3344197 1.05304
          ------------------------------------------------------------------------------
          convergence not achieved
          r(430);
          [/CODE]

          Here's what I think I see in the model. The inflate section of the output models the probability of being a structural zero. I'm pretty sure this is basically a logistic model, so if you exponentiate the coefficients, you get odds ratios.

          For the variable backed, in the example you gave, this takes on integer values that can include zero. That's a very very negative coefficient there. If you exponentiate the coefficient, I am pretty sure that's the odds ratio for being in the structural zero class. That means that if you have positive values of backed, you're almost guaranteed to not be a structural zero (i.e. you fall into the standard count model). It's a little unusual for a coefficient to be that predictive. That's an odds ratio that's almost zero.

          Under the count part of the model, the coefficient for year = 2020 is extremely high, and the standard error for this category is missing. That sounds like whatever your DV is, in 2020, the counts for that DV were a lot higher than in other years. Remember, if you exponentiate the raw coefficients in a count model, you get an incidence rate ratio. e^28.5 = an incidence rate ratio of 2.38 * 10^12.

          I haven't used zero-inflated models that much. If you haven't already checked that the data are free of error (within reason), then I'd do that. Otherwise, I have a hunch that those extremely high coefficients might be problematic. We sometimes see numerical instability in other types of regression, e.g. linear regression, when the scales of some variables are extremely high. It could be that you're seeing something like that. I'm not exactly sure what to do about it. You could run two separate analyses, first showing what IVs influence the probability of having a positive count of reci, and then a truncated Poisson or negative binomial regression showing what IVs influence the rate of reci. Truncated count regressions are for cases when a zero value cannot occur. tnbreg and tpoisson are the two commands.
          Be aware that it can be very hard to answer a question without sample data. You can use the dataex command for this. Type help dataex at the command line.

          When presenting code or results, please use the code delimiters format them. Use the # button on the formatting toolbar, between the " (double quote) and <> buttons.

          Comment

          Working...
          X