Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • iteratively extracting fitstat values to new file

    Hello
    I am using Stata 17 on Windows 10.

    I have dataset of approximately 265,000 observations where I need to calculate McKelvey and Zavoina’s R2 (MZR2) from fitstat for everyday over a 18 month period and extract this with the relevant date to a new dBase.
    Previously I was using statsby with cnreg which prefectly extracted the coefficients I was interested to a new file. But fitstat doesn't run with statsby.

    Code:
    statsby _b nlc=(e(N_lc)) nrc=(e(N_rc)) total=e(N), by(date) sa(finaldlf): cnreg log10flow [aw=pop], censored(censorRED)
    Using runby I can calculate the MZR2 for each date but I can’t find a way of iteratively extracting the MZR2 into a new file (which statsby did so simply).

    Code:
    clear all
    program my_prog
                    cnreg log10flow cases [aw=pop], censored(censorRED)
                    estadd fitstat
                    end
    use "cfcRED.dta"
    runby my_prog, by (date) verbose
    I would like to end up with a file that looks like this (there will be days with missing values in the data):
    date mzr2
    01dec2021 .130215
    02dec2021 .2354
    03dec2021 .365987
    04dec2021 .236597
    05dec2021 .985524
    06dec2021 .32654
    07dec2021
    08dec2021 .32654
    09dec2021 .56841
    Any suggestions for iteratively extracting the MZR2 into a new file would be appreciated.
    Many thanks

  • #2
    estadd fitstat
    makes no sense being that fitstat is a command. You need to store the statistic itself. You are using quite a number of community-contributed commands, so see the requirement to state the provenance of these commands in FAQ Advice #12.

    Comment


    • #3
      Apologies for not including the sources of the community commands
      fitstat (SSC) (from spost by Scott Long and Jeremy Freese)
      runby (SSC) (by Kit Baum and Clyde Schechter)
      estadd (SSC) (from SPost by Scott Long and Jeremy Freese)

      Comment


      • #4
        If this was my problem I would calculate the measure directly without using fitstat at all. What does cnreg leave behind and what is the recipe you need? A quick Google indicates some confusion about what M and Z R-square is, or should be, and whether the details depend on the kind of regression used.

        Comment


        • #5
          If you can implement the formula, do it directly as Nick suggests. Otherwise, I would recommend installing fitstat from the SPost13 website as the SSC version is outdated.

          Code:
          ssc uninstall fitstat
          *INSTALL FROM
          findit spost13_ado
          The procedure to access fitstat results is involved, you need to specify the -save- option and pick up the value from a matrix. I would do it with a loop and frames.

          Code:
          webuse news2, clear
          cnreg date lncltn famown, censored(cnsrd)
          fitstat,  save
          mat l fs_0
          di fs_0[1, "r2_mz"]

          Res.:

          Code:
          . fitstat,  save
          
          Measures of Fit for cnreg of date
          
          Log-Lik Intercept Only:       -620.294   Log-Lik Full Model:           -519.747
          D(96):                        1039.494   LR(2):                         201.095
                                                   Prob > LR:                       0.000
          McFadden's R2:                   0.162   McFadden's Adj R2:               0.156
          ML (Cox-Snell) R2:               0.866   Cragg-Uhler(Nagelkerke) R2:      0.866
          McKelvey & Zavoina's R2:         0.904                              
          Variance of y*:            3843452.933   Variance of error:          369645.260
          AIC:                            10.475   AIC*n:                        1047.494
          BIC:                           597.397   BIC':                         -191.884
          BIC used by Stata:            1057.914   AIC used by Stata:            1047.494
          
          (Indices saved in matrix fs_0)
          
          .
          . mat l fs_0
          
          fs_0[1,28]
                          N        ll_0          ll         dev      dev_df        lrx2     lrx2_df      lrx2_p       r2_mf    r2_mfadj       r2_ml       r2_cu       r2_mz       r2_ef     v_ystar
          cnreg         100  -620.29416  -519.74678   1039.4936          96   201.09475           2   2.152e-44   .16209628   .15564772   .86613821   .86614176   .90382469          .x   3843452.9
          
                    v_error       r2_ct    r2_ctadj          r2      r2_adj         aic       aic_n         bic       bic_p    statabic    stataaic      n_parm       n_rhs
          cnreg   369645.26          .x          .x          .x          .x   10.474936   1047.4936   597.39723  -191.88441   1057.9142   1047.4936           4           2
          
          .
          . di fs_0[1, "r2_mz"]
          .90382469
          
          .
          This is not tested and may contain typos and errors:

          Code:
          levelsof date, local(dates)
          frame create wanted
          frame wanted{
              set obs `=wordcount("`dates'")'
              gen date=.
              gen r2_mz=.
          }
          local i 1
          foreach date of local dates{
              capture{
                  cnreg log10flow cases if date==`date' [aw=pop], censored(censorRED)
                  fitstat, save
                  frame wanted: replace date= `date' in `i'
                  frame wanted: replace r2_mz= fs_0[1, "r2_mz"] in `i'
                  mat drop fs_0
                  local ++i
             }
          }
          
          frame change wanted
          format date %td
          l, sep(0)
          Last edited by Andrew Musau; 01 Dec 2022, 09:46.

          Comment


          • #6
            Thank you both for your time.

            Unfortunately Nick Cox the R2_MZ doesn't work with cnreg (which I do appreciate is an older command), but its good to know about it.

            Andrew Musau This has worked perfectly. Thank you so much. I've spent many hours trying to get this to work, I should have asked on here sooner! I really appreciate it.

            Comment


            • #7
              As I understand it, you want to calculate this measure after cnreg and fitstat will do it for you. So, that should mean that you could calculate it directly. I too would be surprised if cnreg provides it directly, but cnreg should leave in memory the ingredients for the calculation.

              Comment


              • #8
                Hello.

                I'm now trying to run this loop with a string variable instead of a date variable.

                I've replaced the references to date with the string name but I just getting an empty file. I wondered if I was missing something with quotation marks? Everything I've read says foreach is designed to be used with strings and integers, but it just won't work!

                The strings I have are quite complex number, letter and symbol (- and _ )combinations, but they don't have spaces.

                The command I'm using is the same as the one Andrew suggested , except for the addition of clean as it was failing before that was added in.

                Code:
                clear all
                use "cfcRED.dta"
                levelsof catch, clean local(catchms)
                frame create wanted
                frame wanted{
                    set obs `=wordcount("`catchms'")'
                    gen catch=.
                    gen r2_mz=.
                    }
                local i 1
                foreach catch of local catchms{
                    capture{
                        cnreg cases log10flow  if catch==`catch' [aw=pop], censored(censorRED)
                        fitstat, save
                        frame wanted: replace catch= `catch' in `i'
                        frame wanted: replace r2_mz= fs_0[1, "r2_mz"] in `i'
                        mat drop fs_0
                        local ++i
                   }
                }
                
                frame change wanted
                l, sep(0)
                Any advice gratefully received

                Comment


                • #9
                  estadd (SSC) (from SPost by Scott Long and Jeremy Freese)
                  Actually, estadd is by Ben Jann.

                  estadd fitstat is a legitimate command, but you have to use the antiquated spost9 version of fitstat. fitstat can be tricky, because there are different versions available on different sites. I don't know which version you are using, but if you use the spost9 version perhaps your original code would work.
                  -------------------------------------------
                  Richard Williams, Notre Dame Dept of Sociology
                  Stata Version: 17.0 MP (2 processor)

                  EMAIL: [email protected]
                  WWW: https://www3.nd.edu/~rwilliam

                  Comment


                  • #10
                    Originally posted by Natalia Jones View Post

                    I'm now trying to run this loop with a string variable instead of a date variable.
                    On why the code in #8 is failing, see

                    Code:
                    h quotes
                    Assuming the string contains no spaces, you want

                    Code:
                    clear all
                    use "cfcRED.dta"
                    levelsof catch, clean local(catchms)
                    frame create wanted
                    frame wanted{
                        set obs `=wordcount("`catchms'")'
                        gen catch=""
                        gen r2_mz=.
                        }
                    local i 1
                    foreach catch of local catchms{
                        capture{
                            cnreg cases log10flow  if catch=="`catch'" [aw=pop], censored(censorRED)
                            fitstat, save
                            frame wanted: replace catch= "`catch'" in `i'
                            frame wanted: replace r2_mz= fs_0[1, "r2_mz"] in `i'
                            mat drop fs_0
                            local ++i
                       }
                    }
                    
                    frame change wanted
                    l, sep(0)
                    If there are spaces in this string, start with

                    Code:
                    replace catch = strtoname(catch)
                    and then proceed with the code. For details

                    Code:
                    h strtoname()

                    Comment


                    • #11
                      Thanks Andrew Musau . I thought I'd tried every iteration of "" but I missed the gen catch =""
                      Your code worked perfectly. Many thanks

                      Comment

                      Working...
                      X