Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • set seed problem,different seed the results are different

    Hi,
    I know that we must set the same seed for reproducible. For examples,
    Code:
    . sysuse auto,clear
    (1978 Automobile Data) 
    . set seed 1234
    . sqreg price weight length foreign, quantile(.25 .5 .75) reps(100)
    (fitting base model)
    (bootstrapping ....................................................................................................)
    
    Simultaneous quantile regression                     Number of obs =        74
      bootstrap(100) SEs                                 .25 Pseudo R2 =    0.1697
                                                         .50 Pseudo R2 =    0.2347
                                                         .75 Pseudo R2 =    0.3840
    
    ------------------------------------------------------------------------------
                 |              Bootstrap
           price |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
    q25          |
          weight |   1.831789   1.063354     1.72   0.089    -.2890044    3.952582
          length |   2.845558   19.77565     0.14   0.886    -36.59573    42.28685
         foreign |   2209.925   1095.034     2.02   0.047     25.94851    4393.902
           _cons |  -1879.775   2858.265    -0.66   0.513    -7580.405    3820.856
    -------------+----------------------------------------------------------------
    q50          |
          weight |   3.933588   2.477513     1.59   0.117    -1.007656    8.874832
          length |  -41.25191   63.16333    -0.65   0.516    -167.2272    84.72338
         foreign |   3377.771   1270.405     2.66   0.010     844.0279    5911.514
           _cons |   344.6494   5481.411     0.06   0.950    -10587.68    11276.98
    -------------+----------------------------------------------------------------
    q75          |
          weight |    9.22291   2.190276     4.21   0.000     4.854543    13.59128
          length |  -220.7833   75.01802    -2.94   0.004     -370.402   -71.16457
         foreign |   3595.133   1124.303     3.20   0.002     1352.782    5837.484
           _cons |    20242.9   8416.244     2.41   0.019     3457.236    37028.57
    ------------------------------------------------------------------------------
    
    . set seed 1234
    
    . sqreg price weight length foreign, quantile(.25 .5 .75) reps(100)
    (fitting base model)
    (bootstrapping ....................................................................................................)
    
    Simultaneous quantile regression                     Number of obs =        74
      bootstrap(100) SEs                                 .25 Pseudo R2 =    0.1697
                                                         .50 Pseudo R2 =    0.2347
                                                         .75 Pseudo R2 =    0.3840
    
    ------------------------------------------------------------------------------
                 |              Bootstrap
           price |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
    q25          |
          weight |   1.831789   1.063354     1.72   0.089    -.2890044    3.952582
          length |   2.845558   19.77565     0.14   0.886    -36.59573    42.28685
         foreign |   2209.925   1095.034     2.02   0.047     25.94851    4393.902
           _cons |  -1879.775   2858.265    -0.66   0.513    -7580.405    3820.856
    -------------+----------------------------------------------------------------
    q50          |
          weight |   3.933588   2.477513     1.59   0.117    -1.007656    8.874832
          length |  -41.25191   63.16333    -0.65   0.516    -167.2272    84.72338
         foreign |   3377.771   1270.405     2.66   0.010     844.0279    5911.514
           _cons |   344.6494   5481.411     0.06   0.950    -10587.68    11276.98
    -------------+----------------------------------------------------------------
    q75          |
          weight |    9.22291   2.190276     4.21   0.000     4.854543    13.59128
          length |  -220.7833   75.01802    -2.94   0.004     -370.402   -71.16457
         foreign |   3595.133   1124.303     3.20   0.002     1352.782    5837.484
           _cons |    20242.9   8416.244     2.41   0.019     3457.236    37028.57
    ------------------------------------------------------------------------------
    When the different seed is used, the results are significant difference. For example,
    Code:
    . set seed 125874124
    
    . sqreg price weight length foreign, quantile(.25 .5 .75) reps(100)
    (fitting base model)
    (bootstrapping ....................................................................................................)
    
    Simultaneous quantile regression                     Number of obs =        74
      bootstrap(100) SEs                                 .25 Pseudo R2 =    0.1697
                                                         .50 Pseudo R2 =    0.2347
                                                         .75 Pseudo R2 =    0.3840
    
    ------------------------------------------------------------------------------
                 |              Bootstrap
           price |      Coef.   Std. Err.      t    P>|t|     [95% Conf. Interval]
    -------------+----------------------------------------------------------------
    q25          |
          weight |   1.831789   1.662877     1.10   0.274    -1.484715    5.148293
          length |   2.845558   35.83785     0.08   0.937    -68.63078     74.3219
         foreign |   2209.925   1226.628     1.80   0.076    -236.5061    4656.357
           _cons |  -1879.775   3741.564    -0.50   0.617    -9342.088    5582.539
    -------------+----------------------------------------------------------------
    q50          |
          weight |   3.933588   2.864569     1.37   0.174    -1.779615    9.646791
          length |  -41.25191   73.95145    -0.56   0.579    -188.7434    106.2396
         foreign |   3377.771   1211.897     2.79   0.007     960.7191    5794.823
           _cons |   344.6494   6359.924     0.05   0.957    -12339.82    13029.12
    -------------+----------------------------------------------------------------
    q75          |
          weight |    9.22291   2.473674     3.73   0.000     4.289322     14.1565
          length |  -220.7833    82.6403    -2.67   0.009    -385.6042    -55.9624
         foreign |   3595.133   1114.196     3.23   0.002     1372.939    5817.328
           _cons |    20242.9   8790.816     2.30   0.024     2710.174    37775.63
    ------------------------------------------------------------------------------
    From the above results we can see that the coefficient of foreign is significant at 5% level(p = 0.047) when the seed is 1234.
    However, the coefficient of foreign is not significant at 5% level(p = 0.076) when the seed is 125874124.

    I understand that we cannot see the P value only. Since this question are raised by a reviewer.

    However, how could I solve this problem in a more reasonable? Could I solve this problem by increasing the rep number?

    Thanks very much for everyone kindly suggestions!

    Bests,
    wanhaiyou


  • #2
    I would assume that you could try a few different seed values and make a logical case for disregarding one set of results that is inconsistent with the other results as an artifact of random sampling. Basically if you find the same substantive results with several different seed values and encounter a single case where the interpretation is different it may just be an artifact of using pseudorandom number generators.

    Comment


    • #3
      Thanks very much for your greatly suggestion, wbuchanan! I will run the results following your suggestion.
      In addition, is there any article related to this issue(for reference)?


      Bests,
      wanhaiyou

      Comment


      • #4
        What version of Stata are you running? I just ran your code in 14.2 and the differences across seeds were trivial. This may be because Stata periodically changes the random number generator. But even in your results the effect in question just changes from borderline significant (t = 2.02) to borderline insignificant (t = 1.80).

        Also, I was able to replicate your results in 14.2 by specifying

        set rng kiss32

        I then upped the reps to 500 and the differences in T values were again trivial.

        In any event it doesn't surprise me that an effect that is right on the edge between being significant and insignificant sometimes shows up as borderline significant and sometimes shows up as borderline insignificant when you change the seed. I'd probably feel more comfortable increasing the number of reps than I would fishing around for seeds that worked.
        -------------------------------------------
        Richard Williams, Notre Dame Dept of Sociology
        StataNow Version: 19.5 MP (2 processor)

        EMAIL: [email protected]
        WWW: https://academicweb.nd.edu/~rwilliam/

        Comment


        • #5
          Thanks very much for your help, professor Richard.

          Bests,
          wanhaiyou

          Comment

          Working...
          X