Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • "program foo not found" followed instantly with "program foo already defined"

    master.do:
    Code:
    include outer.do
    outer
    outer
    outer.do:
    Code:
    capture noisily program drop outer
    program outer
        include inner.do
        di "This is the outer program."
        inner
    end
    inner.do:
    Code:
    capture noisily program drop inner
    program inner
        di "This is the inner program."
    end
    Result:
    Code:
    . do master.do
    
    . include outer.do
    
    . capture noisily program drop outer
    program outer not found
    
    . program define outer
      1.         include inner.do
      2.         di "This is the outer program."
      3.         inner
      4. end
    
    . 
    . 
    . outer
    
    . capture noisily program drop inner
    program inner not found
    
    . program define inner
      1.         di "This is the inner program."
      2. end
    
    . This is the outer program.
    This is the inner program.
    
    . outer
    
    . capture noisily program drop inner
    program inner not found
    
    . program define inner
    program inner already defined
    r(110);
    r(110);
    
    end of do-file
    
    r(110);
    I did not expect -program drop inner- to fail the second time.

    I notice that -program dir- shows a program named outer.inner. My best guess is that -program drop inner- tries to drop program "inner", which fails because there is no program "inner" in memory. Then -program define inner- fails because it's actually trying to define program "outer.inner", which is already defined. The error message "program inner already defined" actually means "program outer.inner already defined". (This would require that that -program define- and -program drop- behave asymmetrically, but I don't have a better idea.)

    I tried dropping outer.inner, but I get "subprograms not allowed".

    I can work around the issue by moving -include inner.do- from outer.do to master.do, but this comes at the expense of managing outer.do's dependencies (i.e. inner.do) on its behalf, and I don't feel like I should be responsible for them. Am I missing a more elegant solution?

  • #2
    I don't see how it may work. You are calling inner from outer creating nested program definitions.

    Substituting inner include produces:
    Code:
    clear all
    
    capture noisily program drop outer
    program outer
        capture noisily program drop inner
        program inner
           di "This is the inner program."
        end
        di "This is the outer program."
        inner
    end
    (I added clear all).
    So you have a program in a program

    Perhaps you just want to do or run the file that you have include'd (removing the program/end statements.)

    Last edited by Sergiy Radyakin; 03 Aug 2015, 17:39.

    Comment


    • #3
      Can you explain more? Are nested program definitions problematic? Isn't that what subprograms are?

      Comment


      • #4
        In the .do file (anything.do) you should write:
        Code:
        clear all
        
        program define header
          display "Hello. Look at my cool report below!"
        end
        
        program define report
          header
          summarize
        end
        
        sysuse auto
        report
        In an .ado file (report.ado) you should write:
        Code:
        program define report
          header
          summarize
        end
        
        program define header
          display "Hello. Look at my cool report below!"
        end
        Then you can call:
        Code:
        clear all
        sysuse auto
        report
        I would probably prefer two .ado files (report.ado and header.ado), which roughly corresponds StataCorp's paradigm, as far as I can deduce it from the graphics subsystem. The user can then override the default header.ado if necessary, leaving the report.ado intact. To override an ado file, redirect sysdir with a path of higher priority and place the modified file there.

        Best, Sergiy

        Comment


        • #5
          Your solution bundles outer/inner (report/header) together. I use inner (header) for a bunch of tasks, not all of which are in outer (report). A ton of do-files in my project require inner, but very few require outer. I could stuff all my programs into the same file, but it seems ugly to have every do-file in my project load all these useless programs. It's also more difficult to see exactly which programs each do-file is going to use.

          If I use two ado files, then I require the user to move those ado files into their personal ado directories, or they will not be accessible from anywhere in the project. I would prefer to avoid that, as I like my projects to be self-contained and portable.

          Comment


          • #6
            As I wrote in #2 the simplest is probably to use separate .do files:

            master.do:
            Code:
            clear all
            sysuse auto
            do report.do
            report.do:
            Code:
            do header.do
            summarize
            header.do:
            Code:
            display "Hello. Look at my cool report below!"

            You wrote:
            it seems ugly to have every do-file in my project load all these useless programs
            Stata's program loading facility automatically tracks program definitions and loads new program definitions automatically when an unknown command is encountered and a corresponding definition file (*.ado) is found along the search path. This has been standard all the time I am using Stata, and (ask for confirmation of the older guys) perhaps since version 2.1.

            Also, you wrote:
            I like my projects to be self-contained and portable
            Just create a .pkg file which describes your project, then let the users install the package and Stata will sort out the rest. Check if Robert Picard 's -project- creates a corresponding package file automatically (if not, request a feature).

            Perhaps this question is important then: How did you intend to ship multiple include files originally?

            Best, Sergiy

            Comment


            • #7
              Ah, you added information to #2. Is nesting -program define- in -program define- not permitted? It seems to work OK the first time outer is run. I assumed that that nesting is what the outer.inner program syntax was built for.

              Unfortunately, I do really need these to be programs, not just straightforward do-files. My use case is more complicated than the simplified example. For example, it needs to take an argument.

              I understand that it's not much of a performance hit to load a zillion programs in each do-file; it just seems ugly. For example, they'll all show up in traces, and, as I said, "it's also more difficult to see exactly which programs each do-file is going to use." This might not be a dealbreaker, though.

              Can I install a local pkg on the user's behalf via a do-file? I'm not familiar with this.

              I do love -project-, but I don't think it has this feature.

              My project is an entire folder hierarchy. I don't need it to be a single file, if that's what you're asking. I just want somebody to be able to fork it from version control and run the master do-file.

              Thanks!

              Comment


              • #8
                Anything that ends with an end may not be nested into anything else that ends with an end. Meaning, among other things, that programs may not contain input/end and mata/end blocks.

                Subprograms are defined as separate programs. They are nested within the same ado file, but not into each other. See clear.ado for an example:
                Code:
                  ado      146  clear.clearreturn
                  ado      507  clear._clear_9
                  ado     1406  clear
                      --------
                My use case is more complicated than the simplified example. For example, it needs to take an argument.
                Your complicated case still fits the standard use pattern. See do:

                Code:
                display "Hello, `1'. Look at my cool report below!"
                Code:
                do header.do "Sergiy"
                summarize
                Code:
                clear all
                sysuse auto
                do report.do
                Can I install a local pkg on the user's behalf via a do-file?
                Technically yes. Rumors are that in some countries they will cut off your hand for doing this. But yes, you can.

                I just want somebody to be able to fork it from version control and run the master do-file.
                Those 2.5 users that can "fork it from vcs" can also set up SYSDIR.

                "it's also more difficult to see exactly which programs each do-file is going to use."
                How is include abc.do more obvious than do abc.do?

                On file/folder structure donno, need to see it to make recommendations. In general - go with one file only.
                On deployment - Stata is too primitive for complex projects deployment. You will have to compromise.
                On trace: seriously?

                This probably goes beyond my spare time allowance, but if anyone is interested in advanced Stata programming - let me know.

                Best, Sergiy

                Comment


                • #9
                  Maybe it's late but I don't quite understand all the program gymnastics here. If it's a program, put it in an ado and put it where the user will find it. If you are using project (from SSC), then create a subdirectory within the master directory, call it "ado" and put all you ado programs there. Here's what I put in my master do-files these days

                  Code:
                  * This project uses user-written commands that are not part of the
                  * official Stata installation. To ensure replication, we include local
                  * copies in the "ado" subdirectory and put "ado" in front of the ado path
                  * so that its content has priority over other versions installed
                  * elsewhere.
                  
                      adopath ++ "`c(pwd)'/ado"
                      
                      // install and add dependencies for each program used in this project
                      project, do("ado/user-written_programs.do")
                      
                      // rebuild the index so that Stata can find Mata libraries in ado dir
                      mata: mata mlib index
                  As you can see, I have a "user-written_programs.do" that takes care of declaring all dependencies. Here's its content for the project I'm currently working on

                  Code:
                  /* 
                  -------------------------------------------------------------------------------
                  
                  To ensure replicability and make this project portable, we use local copies of
                  user-written programs.
                  
                  -------------------------------------------------------------------------------
                  */
                  
                      version 14
                      
                      
                  * If you are unable to install -project- from SSC via -ssc install project-,
                  * you can install it using the content of the following zip archive.
                  
                      project, relies_on("project.zip")
                      
                  
                  * Dependencies for programs not installed using Stata packages (i.e. the files
                  * are just saved to this directory)
                  
                      which stringsum    
                      project, original("stringsum.ado")
                      project, relies_on("stringsum.dlg")
                      project, relies_on("stringsum.hlp")
                      
                      // this is a substitute for the standard -replace- command that adds
                      // a rollback copy of the variable and lists some random examples.
                      which replacel
                      project, original("replacel.ado")
                      
                      which package_dependencies
                      project, original("package_dependencies.ado")
                  
                      which unicode_dependencies
                      project, original("unicode_dependencies.ado")
                      
                      
                  * If we use user-written programs installed in Stata's default location (PLUS),
                  * this project will not run on another computer until these programs are installed.
                  * If the programs are updated, the project may not replicate anymore. Here's
                  * what's currently installed
                      
                      ado
                      
                  
                  * Install user-written programs that are distributed as Stata packages. We only
                  * need to install/uninstall/update packages once so uncomment the code as needed
                  
                      net set ado "."    // packages will now be installed in this directory
                      
                  *    ssc install leftalign
                  *    ssc install listsome
                  *    ssc install randomtag
                  *    ssc install filelist
                      
                      
                  * If we want to uninstall programs, here's a sample of the correct syntax
                  
                  *    ado uninstall leftalign, from(".")
                  
                  
                  * If we want to check for updates,
                  
                  *    adoupdate, dir(".") update
                      
                      
                  * Show what packages are installed in this directory
                  
                      ado dir, from(".")
                      
                      
                  * Add dependencies for all files installed using Stata packages in this directory.
                  * This functionality will probably be rolled into -project- in a future version.
                  
                      package_dependencies
                  This is the content of "package_dependencies.ado"

                  Code:
                  *! version 1.0.2  29mar2015  Robert Picard, picard@netbox.com
                  program define package_dependencies
                  
                      version 9
                  
                  
                  * Stata tracks what it installs in this directory via the -net- command
                  * using the following files
                  
                      project, original("stata.trk")
                      cap project, relies_on("backup.trk")    // capture in case not there
                      
                      
                  * Loop over all installed files from Stata packages and add dependencies
                  
                      tempname trk
                      file open `trk' using "stata.trk", read
                      
                      file read `trk' line
                      
                      
                      while !r(eof) {
                      
                          if regexm("`line'", "^f (.+)") {
                          
                              local fname = regexs(1)
                              
                              if regexm("`fname'","\.(ado|mlib)$") project, original("`fname'") preserve
                              else project, relies_on("`fname'") preserve
                              
                          }
                          
                          file read `trk' line
                      }
                      
                      file close `trk'
                      
                      
                  end
                  The content of "unicode_dependencies.ado" is

                  Code:
                  *! version 1.0.0  08jul2015  Robert Picard, picard@netbox.com
                  program define unicode_dependencies
                  
                      version 14
                  
                  
                  * attach all backup files to the project
                  
                      filelist , dir(bak.stunicode)
                      local nobs = _N
                      if `nobs' {
                          
                          forvalues i = 1/`nobs' {
                              
                              project, relies_on("`=dirname[`i']'/`=filename[`i']'") preserve
                              
                          }
                      }
                      
                  end
                  I'm long overdue to update the demo projects that come with project. The next update will probably include some/all of the above although I hesitate to post something that's too complicated for the uninitiated to understand; I'm a bit concerned that it will scare people away. I'm busy these days on this project so this will probably have to wait until I'm done with it.

                  Comment


                  • #10
                    Aside from the above, note the Warning given in the help for include.

                    Do not use command include in the body of a Stata program:
                    Code:
                                    program ...
                                            ...
                                            include ...
                                            ...
                                    end
                    The include will not be executed, as you might have hoped, when the program is compiled. Instead, the include will be stored in your program and executed every time your program is run. The result will be the same as if the lines had been included at compile time, but the execution will be slower.
                    Best
                    Daniel

                    Comment


                    • #11
                      Anything that ends with an end may not be nested into anything else that ends with an end. Meaning, among other things, that programs may not contain input/end and mata/end blocks.
                      Interesting. program define ... program define ... end ... end fails, but program define ... include [program define ... end] ... end doesn’t. Maybe it should fail. edit: See below.

                      Your complicated case still fits the standard use pattern.
                      I didn’t know do could pass arguments. Thanks for this.

                      How is include abc.do more obvious than do abc.do?
                      I was referring to stuffing multiple programs into a single file. In other words, include bop.do, include twist.do, include pull.do is more obvious than include programs.do.

                      On file/folder structure donno, need to see it to make recommendations. In general - go with one file only.
                      Don’t look now, but this project includes about 500 do-files. :-s

                      On trace: seriously?
                      Yes. Is this unusual? I use trace extensively. I think it’s my favorite debugging tool in Stata.

                      adopath ++
                      Perfect. I looked up “17.5 Where does Stata look for ado-files?” in the manual. It said “Stata looks for ado-files in seven places”, and made no mention any method of modifying the search path. I took this to mean that there was no way to change this behavior. I am delighted that adopath exists. I think it will solve all of my problems.

                      Aside from the above, note the Warning given in the help for include.
                      Aha! This warning would have been nice to see when I, er, called include from within program define. I think this explains why it is possible (if inadvisable) to include a program (which ends with an end) inside another program: when the parent program is called, it’s no longer defining a program, so it’s happy to define a child program. Sadly, the explanation for why not to do this seems incomplete. The only reason given is a performance hit, which is my case is minimal. Logically, it still seems like it should work. The behavior that the manual guesses I might have hoped for is actually not what I expected; rather the actual behavior is what I expected. I do not see how this warning explains the “program inner not found”/“program inner already defined” error.

                      TL;DR

                      I don't understand this error, but regardless I'm going to refactor my code to use adopath, which sidesteps the issue.

                      Comment


                      • #12
                        Glad to hear that a local and project-specific "ado" directory added to the adopath will solve your problems. It's a useful strategy in terms of program development because you compartmentalize your code, without fear of messing up unrelated projects (e.g. each project has their own local version of a utility you are working on).

                        With respect to being able to track which do-file uses which program, you can have your ado-program self-declare its dependency. Here's a silly example

                        Code:
                        *! version 1.0.0, 04aug2015, Robert Picard, picard@netbox.com      
                        program hello_world
                        
                            version 13
                            
                            dis "Hello World"
                            
                            // automatically track dependencies for this program
                            findfile "hello_world.ado"
                            project, original("`r(fn)'")
                            
                        end
                        You can then use

                        Code:
                        project xyz, list(concordance)
                        to find all do-files that used the hello_world command.
                        Last edited by Robert Picard; 04 Aug 2015, 10:26.

                        Comment

                        Working...
                        X