Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • New on SSC: smcl2do and splitlog, commands for postprocessing log files

    Thanks to Kit Baum, the smcl2do package is available from the SSC.

    The package has 2 new commands:

    smcl2do converts log files to do-files in a robust fashion.

    By default, smcl2do strips all commands which resulted in errors or were interrupted. It can also strip out all commands (such as help or edit) which do not do anything on their own, but which do open windows. It can also keep all commands if you wish. It can handle Mata commands, loops, and do commands.

    splitlog is made for custom postprocessing of log files.

    By default, splitlog separates commands, their associated results, and orphaned results arising from commands not in the log file by placing special comments around each. splitlog can be used to make files of commands, output, or a mixture of the two. The style of the text surrounding the commands, results, and orphans can be customized in case they are needed for other markup languages.

    The two commands are bundled into one package because smcl2do uses splitlog to do its work.

    ssc install smcl2do will install the package for you.

  • #2
    I am not getting the expected results from smcl2do. When run on the attached .scml file, the following command --
    smcl2do using "derbylog.smcl", saving("post_position.do") all replace
    -- creates a .do file, also attached, which does not contain the commands in the .smcl file. Instead, most lines in the .do file look like this:
    do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
    Attached Files

    Comment


    • #3
      splitlog, called by smcl2do, sees the commands from your interactive do file as results, not commands. Include the log command in the do file, and generate a clean log/smcl file?

      I agree, not what I would have expected from reading the help!
      Doug Hemken
      SSCC, Univ. of Wisc.-Madison

      Comment


      • #4
        I guess this means you didn't save the do file you were in the process of building? Bummer!
        Doug Hemken
        SSCC, Univ. of Wisc.-Madison

        Comment


        • #5
          I inherited this smcl file from someone else. Forgive me for not understanding, but what command should I run to convert it to a do file?

          Comment


          • #6
            There are at least two distinct questions here. This is your SMCL file and it makes sense that any processing produces lines like

            Code:
            do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            because these were Stata commands used by Stata during a session. Such commands can be seen in the log. They look like chunks of code that were run from the do-file editor.

            But the commands concerned follow in the log. If you select lines that start with

            Code:
            {com}
            and zap the mark-up you can get there fairly quickly.

            Code:
            {smcl}
            {com}{sf}{ul off}{txt}{.-}
                  name:  {res}<unnamed>
                   {txt}log:  {res}/Users/ashleefrandell/Desktop/derby-log.smcl
              {txt}log type:  {res}smcl
             {txt}opened on:  {res}27 Apr 2018, 09:50:53
            
            {com}. egen field_size = count(pp), by(year)
            {err}variable {bf}year{sf} not found
            (error in option {bf:by()})
            {txt}{search r(111), local:r(111);}
            
            {com}. rename date year
            {res}
            {com}. egen field_size = count(pp), by(year)
            
            . save derby_corrected, replace
            {txt}(note: file derby_corrected.dta not found)
            file derby_corrected.dta saved
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), legend(off)
            {res}{txt}
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(All Races) xtitle(postposition) ytitle(place at finish) saving(scatter.with.all.gph) legend(off)
            {res}{txt}(file scatter.with.all.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. graph export scatter.with.all.jpg
            {err}{p 0 4 2}
            output-file suffix "with.all.jpg" not recognized{break}
            specify correct suffix or specify as() option
            {p_end}
            {txt}{search r(198), local:r(198);}
            
            {com}. help graph export
            
            . graph export scatter_with_all.png
            {txt}(file scatter_with_all.png written in PNG format)
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 7
            {txt}(1,651 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 7) xtitle(postposition) ytitle(place at finish) saving(scatter.with.7.gph) legend(off)
            {res}{txt}(file scatter.with.7.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. graph export scatter_7.png
            {txt}(file scatter_7.png written in PNG format)
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 10
            {txt}(1,619 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 10) xtitle(postposition) ytitle(place at finish) saving(scatter.with.10.gph) legend(off)
            {res}{txt}(file scatter.with.10.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 13
            {txt}(1,562 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 13) xtitle(postposition) ytitle(place at finish) saving(scatter.with.13.gph) legend(off)
            {res}{txt}(file scatter.with.13.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 14
            {txt}(1,581 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 14) xtitle(postposition) ytitle(place at finish) saving(scatter.with.14.gph) legend(off)
            {res}{txt}(file scatter.with.14.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 15
            {txt}(1,499 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 15) xtitle(postposition) ytitle(place at finish) saving(scatter.with.15.gph) legend(off)
            {res}{txt}(file scatter.with.15.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 16
            {txt}(1,551 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 16) xtitle(postposition) ytitle(place at finish) saving(scatter.with.16.gph) legend(off)
            {res}{txt}(file scatter.with.16.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 19
            {txt}(1,470 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 19) xtitle(postposition) ytitle(place at finish) saving(scatter.with.19.gph) legend(off)
            {res}{txt}(file scatter.with.19.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 20
            {txt}(1,379 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 20) xtitle(postposition) ytitle(place at finish) saving(scatter.with.20.gph) legend(off)
            {res}{txt}(file scatter.with.20.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. use derby_corrected, clear
            {txt}
            {com}. keep if field_size == 21
            {txt}(1,616 observations deleted)
            
            {com}. sort pp fin
            {txt}
            {com}. collapse (count) year, by(pp fin)
            {txt}
            {com}. rename year years
            {res}{txt}
            {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), title(Field Size 21) xtitle(postposition) ytitle(place at finish) saving(scatter.with.21.gph) legend(off)
            {res}{txt}(file scatter.with.21.gph saved)
            
            {com}. 
            {txt}end of do-file
            
            {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
            {txt}
            {com}. graph combine scatter.with.7.gph scatter.with.10.gph scatter.with.13.gph scatter.with.14.gph scatter.with.15.gph scatter.with.16.gph scatter.with.19.gph scatter.with.20.gph scatter.with.21.gph
            {res}{txt}
            {com}. 
            {txt}end of do-file
            
            {com}. graph export scatter_fieldsizes.png
            {txt}(file scatter_fieldsizes.png written in PNG format)
            
            {com}. log close
                  {txt}name:  {res}<unnamed>
                   {txt}log:  {res}/Users/ashleefrandell/Desktop/derby-log.smcl
              {txt}log type:  {res}smcl
             {txt}closed on:  {res}27 Apr 2018, 10:32:17
            {txt}{.-}
            {smcl}
            {txt}{sf}{ul off}

            Comment


            • #7
              Originally posted by paulvonhippel View Post
              I am not getting the expected results from smcl2do. When run on the attached .scml file, the following command --
              smcl2do using "derbylog.smcl", saving("post_position.do") all replace
              -- creates a .do file, also attached, which does not contain the commands in the .smcl file. Instead, most lines in the .do file look like this:
              do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
              Your log file came from someone repeatedly submitting commands from the do-file editor, which resulted in all the do /var/folders/.... command, so smcl2do recreated those do commands.

              This is the expected behavior of smcl2do, because it is meant to recreate the commands given to the command window... it's purpose isn't to recreate commands from do-files which were done during the session. (This would end up embedding the contents of every do-file called by every do command in the log, which is not a good thing.)

              This being said, it might not be a bad idea to add an option which allows yanking the commands from those called do-files which do temporary files. I'll have to think and play around with this a bit.

              As for other commands to convert smcl files: there is log2do2, but it would behave in the same fashion, because it has the same purpose.

              Comment


              • #8
                You can recover all the commands submitted from the do-file editor (not interactively) by

                splitlog using derby-log.smcl, res saving(derby1.txt)

                /* here, open "derby1.txt, and add a {smcl} tag as the first line, then do the following: */

                smcl2do using derby1.txt

                As a final step, interleave these new results with your original results - this will have to be done by hand.
                Doug Hemken
                SSCC, Univ. of Wisc.-Madison

                Comment


                • #9
                  Sorry for the confusion. What bothers me was not that the "do" statements in the smcl file were being copied to the do file, but that other statements in the smcl file are not. For example, the following statements were in the smcl file but smcl2do did not put them in the do file. Why not?
                  . use derby_corrected, clear
                  . sort pp fin
                  . collapse (count) year, by(pp fin)
                  . rename year years
                  . twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), legend(off)

                  Comment


                  • #10
                    Bill wrote this so that everything *nested* within a `do` command would be ignored. So for the block

                    Code:
                    {com}. do "/var/folders/92/f5f2j3xs1m5_4s498dqkn9bw0000gn/T//SD28323.000000"
                    {txt}
                    {com}. use derby_corrected, clear
                    {txt}
                    {com}. sort pp fin
                    {txt}
                    {com}. collapse (count) year, by(pp fin)
                    {txt}
                    {com}. rename year years
                    {res}{txt}
                    {com}. twoway (scatter fin pp [fweight=years], msymbol(oh) msize(small)) (lfit fin pp [fweight=years] ), legend(off)
                    {res}{txt}
                    {com}.
                    {txt}end of do-file
                    all of the nested `{com}` lines are "results" to splitlog/smcl2do.

                    It makes perfect sense if you assume that people will save their do files and not orphan their smcl files. Sadly, your case has violated his assumptions!
                    Doug Hemken
                    SSCC, Univ. of Wisc.-Madison

                    Comment


                    • #11
                      Thanks, now I get it. I'm not sure I agree with the assumption that the .do file will be available. If I had a.do file, why would I be trying to convert a .smcl file?

                      Comment


                      • #12
                        I think he just didn't anticipate the tragedy of orphans... he was originally thinking about situations where *he* produces smcl files without do files, i.e. interactively.
                        Doug Hemken
                        SSCC, Univ. of Wisc.-Madison

                        Comment


                        • #13
                          The goal of smcl2do is indeed to act like an after-the-fact recovery of the Review window. In this way it follows in the footsteps of the user-written log2do (defunct) and log2do2 (current). Doug is correct that I never thought about the current example, namely recovering code created by editing and resending code from the Do-file Editor.

                          Suppose now that the goal would be expanded to include such a case.

                          It would be possible to add an option to flatten the nested do commands, so that every command called by all do commands ended up in the resulting do-file. This would be dangerous, however, because it would be like treating do commands like include commands, so that local macros would no longer be local to their do-files. This could then result in a do-file which runs with incorrect results (the worst possible bug). So: bad idea.

                          Suppose instead that smcl2do tried to recreate every do-file which showed up in the smcl file, possibly creating a boatload of do-files. The problem that would arise here would be how to handle do-files which resulted in errors, as part of the purpose of smcl2do is to not recreate errors. There are probably other nuances I haven't thought of yet.

                          So, I'll think about the create-a-boatload-of-do-files idea further, and see if/how I could implement it. If I come up with something reasonable, I'll be sure to update smcl2do.

                          Bill


                          Comment


                          • #14
                            I haven't ever used `include` (but might, now that I know about it). Could one separate the "include" blocks the way you separate the "do" blocks? And provide an **option** to flatten things?

                            Alternatively, someone could develop a separate tool that just grabs everything regardless.
                            Doug Hemken
                            SSCC, Univ. of Wisc.-Madison

                            Comment

                            Working...
                            X