"program foo not found" followed instantly with "program foo already defined"

[Nils Enevoldsen]

master.do:

include outer.do
outer
outer

outer.do:

capture noisily program drop outer
program outer
    include inner.do
    di "This is the outer program."
    inner
end

inner.do:

capture noisily program drop inner
program inner
    di "This is the inner program."
end

Result:

. do master.do

. include outer.do

. capture noisily program drop outer
program outer not found

. program define outer
  1.         include inner.do
  2.         di "This is the outer program."
  3.         inner
  4. end

. 
. 
. outer

. capture noisily program drop inner
program inner not found

. program define inner
  1.         di "This is the inner program."
  2. end

. This is the outer program.
This is the inner program.

. outer

. capture noisily program drop inner
program inner not found

. program define inner
program inner already defined
r(110);
r(110);

end of do-file

r(110);

I did not expect -program drop inner- to fail the second time.

I notice that -program dir- shows a program named outer.inner. My best guess is that -program drop inner- tries to drop program "inner", which fails because there is no program "inner" in memory. Then -program define inner- fails because it's actually trying to define program "outer.inner", which is already defined. The error message "program inner already defined" actually means "program outer.inner already defined". (This would require that that -program define- and -program drop- behave asymmetrically, but I don't have a better idea.)

I tried dropping outer.inner, but I get "subprograms not allowed".

I can work around the issue by moving -include inner.do- from outer.do to master.do, but this comes at the expense of managing outer.do's dependencies (i.e. inner.do) on its behalf, and I don't feel like I should be responsible for them. Am I missing a more elegant solution?

[Sergiy Radyakin]

I don't see how it may work. You are calling inner from outer creating nested program definitions.

Substituting inner include produces:

clear all

capture noisily program drop outer
program outer
    capture noisily program drop inner
    program inner
       di "This is the inner program."
    end
    di "This is the outer program."
    inner
end

(I added clear all).
So you have a program in a program

Perhaps you just want to do or run the file that you have include'd (removing the program/end statements.)

[Nils Enevoldsen]

Can you explain more? Are nested program definitions problematic? Isn't that what subprograms are?

[Sergiy Radyakin]

In the .do file (anything.do) you should write:

clear all

program define header
  display "Hello. Look at my cool report below!"
end

program define report
  header
  summarize
end

sysuse auto
report

In an .ado file (report.ado) you should write:

program define report
  header
  summarize
end

program define header
  display "Hello. Look at my cool report below!"
end

Then you can call:

clear all
sysuse auto
report

I would probably prefer two .ado files (report.ado and header.ado), which roughly corresponds StataCorp's paradigm, as far as I can deduce it from the graphics subsystem. The user can then override the default header.ado if necessary, leaving the report.ado intact. To override an ado file, redirect sysdir with a path of higher priority and place the modified file there.

Best, Sergiy

[Nils Enevoldsen]

Your solution bundles outer/inner (report/header) together. I use inner (header) for a bunch of tasks, not all of which are in outer (report). A ton of do-files in my project require inner, but very few require outer. I could stuff all my programs into the same file, but it seems ugly to have every do-file in my project load all these useless programs. It's also more difficult to see exactly which programs each do-file is going to use.

If I use two ado files, then I require the user to move those ado files into their personal ado directories, or they will not be accessible from anywhere in the project. I would prefer to avoid that, as I like my projects to be self-contained and portable.

[Sergiy Radyakin]

As I wrote in #2 the simplest is probably to use separate .do files:

master.do:

clear all
sysuse auto
do report.do

report.do:

do header.do
summarize

header.do:

display "Hello. Look at my cool report below!"

You wrote:

it seems ugly to have every do-file in my project load all these useless programs

Stata's program loading facility automatically tracks program definitions and loads new program definitions automatically when an unknown command is encountered and a corresponding definition file (*.ado) is found along the search path. This has been standard all the time I am using Stata, and (ask for confirmation of the older guys) perhaps since version 2.1.

Also, you wrote:

I like my projects to be self-contained and portable

Just create a .pkg file which describes your project, then let the users install the package and Stata will sort out the rest. Check if Robert Picard 's -project- creates a corresponding package file automatically (if not, request a feature).

Perhaps this question is important then: How did you intend to ship multiple include files originally?

Best, Sergiy

[Nils Enevoldsen]

Ah, you added information to #2. Is nesting -program define- in -program define- not permitted? It seems to work OK the first time outer is run. I assumed that that nesting is what the outer.inner program syntax was built for.

Unfortunately, I do really need these to be programs, not just straightforward do-files. My use case is more complicated than the simplified example. For example, it needs to take an argument.

I understand that it's not much of a performance hit to load a zillion programs in each do-file; it just seems ugly. For example, they'll all show up in traces, and, as I said, "it's also more difficult to see exactly which programs each do-file is going to use." This might not be a dealbreaker, though.

Can I install a local pkg on the user's behalf via a do-file? I'm not familiar with this.

I do love -project-, but I don't think it has this feature.

My project is an entire folder hierarchy. I don't need it to be a single file, if that's what you're asking. I just want somebody to be able to fork it from version control and run the master do-file.

Thanks!

[Sergiy Radyakin]

Anything that ends with an end may not be nested into anything else that ends with an end. Meaning, among other things, that programs may not contain input/end and mata/end blocks.

Subprograms are defined as separate programs. They are nested within the same ado file, but not into each other. See clear.ado for an example:

  ado      146  clear.clearreturn
  ado      507  clear._clear_9
  ado     1406  clear
      --------

My use case is more complicated than the simplified example. For example, it needs to take an argument.

Your complicated case still fits the standard use pattern. See do:

display "Hello, `1'. Look at my cool report below!"

do header.do "Sergiy"
summarize

clear all
sysuse auto
do report.do

Can I install a local pkg on the user's behalf via a do-file?

Technically yes. Rumors are that in some countries they will cut off your hand for doing this. But yes, you can.

I just want somebody to be able to fork it from version control and run the master do-file.

Those 2.5 users that can "fork it from vcs" can also set up SYSDIR.

"it's also more difficult to see exactly which programs each do-file is going to use."

How is include abc.do more obvious than do abc.do?

On file/folder structure donno, need to see it to make recommendations. In general - go with one file only.
On deployment - Stata is too primitive for complex projects deployment. You will have to compromise.
On trace: seriously?

This probably goes beyond my spare time allowance, but if anyone is interested in advanced Stata programming - let me know.

Best, Sergiy

[Robert Picard]

Maybe it's late but I don't quite understand all the program gymnastics here. If it's a program, put it in an ado and put it where the user will find it. If you are using project (from SSC), then create a subdirectory within the master directory, call it "ado" and put all you ado programs there. Here's what I put in my master do-files these days

* This project uses user-written commands that are not part of the
* official Stata installation. To ensure replication, we include local
* copies in the "ado" subdirectory and put "ado" in front of the ado path
* so that its content has priority over other versions installed
* elsewhere.

    adopath ++ "`c(pwd)'/ado"
    
    // install and add dependencies for each program used in this project
    project, do("ado/user-written_programs.do")
    
    // rebuild the index so that Stata can find Mata libraries in ado dir
    mata: mata mlib index

As you can see, I have a "user-written_programs.do" that takes care of declaring all dependencies. Here's its content for the project I'm currently working on

/* 
-------------------------------------------------------------------------------

To ensure replicability and make this project portable, we use local copies of
user-written programs.

-------------------------------------------------------------------------------
*/

    version 14
    
    
* If you are unable to install -project- from SSC via -ssc install project-,
* you can install it using the content of the following zip archive.

    project, relies_on("project.zip")
    

* Dependencies for programs not installed using Stata packages (i.e. the files
* are just saved to this directory)

    which stringsum    
    project, original("stringsum.ado")
    project, relies_on("stringsum.dlg")
    project, relies_on("stringsum.hlp")
    
    // this is a substitute for the standard -replace- command that adds
    // a rollback copy of the variable and lists some random examples.
    which replacel
    project, original("replacel.ado")
    
    which package_dependencies
    project, original("package_dependencies.ado")

    which unicode_dependencies
    project, original("unicode_dependencies.ado")
    
    
* If we use user-written programs installed in Stata's default location (PLUS),
* this project will not run on another computer until these programs are installed.
* If the programs are updated, the project may not replicate anymore. Here's
* what's currently installed
    
    ado
    

* Install user-written programs that are distributed as Stata packages. We only
* need to install/uninstall/update packages once so uncomment the code as needed

    net set ado "."    // packages will now be installed in this directory
    
*    ssc install leftalign
*    ssc install listsome
*    ssc install randomtag
*    ssc install filelist
    
    
* If we want to uninstall programs, here's a sample of the correct syntax

*    ado uninstall leftalign, from(".")


* If we want to check for updates,

*    adoupdate, dir(".") update
    
    
* Show what packages are installed in this directory

    ado dir, from(".")
    
    
* Add dependencies for all files installed using Stata packages in this directory.
* This functionality will probably be rolled into -project- in a future version.

    package_dependencies

This is the content of "package_dependencies.ado"

*! version 1.0.2  29mar2015  Robert Picard, [email protected]
program define package_dependencies

    version 9


* Stata tracks what it installs in this directory via the -net- command
* using the following files

    project, original("stata.trk")
    cap project, relies_on("backup.trk")    // capture in case not there
    
    
* Loop over all installed files from Stata packages and add dependencies

    tempname trk
    file open `trk' using "stata.trk", read
    
    file read `trk' line
    
    
    while !r(eof) {
    
        if regexm("`line'", "^f (.+)") {
        
            local fname = regexs(1)
            
            if regexm("`fname'","\.(ado|mlib)$") project, original("`fname'") preserve
            else project, relies_on("`fname'") preserve
            
        }
        
        file read `trk' line
    }
    
    file close `trk'
    
    
end

The content of "unicode_dependencies.ado" is

*! version 1.0.0  08jul2015  Robert Picard, [email protected]
program define unicode_dependencies

    version 14


* attach all backup files to the project

    filelist , dir(bak.stunicode)
    local nobs = _N
    if `nobs' {
        
        forvalues i = 1/`nobs' {
            
            project, relies_on("`=dirname[`i']'/`=filename[`i']'") preserve
            
        }
    }
    
end

I'm long overdue to update the demo projects that come with project. The next update will probably include some/all of the above although I hesitate to post something that's too complicated for the uninitiated to understand; I'm a bit concerned that it will scare people away. I'm busy these days on this project so this will probably have to wait until I'm done with it.

[daniel klein]

Aside from the above, note the Warning given in the help for include.

Do not use command include in the body of a Stata program:

                program ...
                        ...
                        include ...
                        ...
                end

The include will not be executed, as you might have hoped, when the program is compiled. Instead, the include will be stored in your program and executed every time your program is run. The result will be the same as if the lines had been included at compile time, but the execution will be slower.

Best
Daniel

[Nils Enevoldsen]

Anything that ends with an end may not be nested into anything else that ends with an end. Meaning, among other things, that programs may not contain input/end and mata/end blocks.

Interesting. program define ... program define ... end ... end fails, but program define ... include [program define ... end] ... end doesn’t. Maybe it should fail. edit: See below.

Your complicated case still fits the standard use pattern.

I didn’t know do could pass arguments. Thanks for this.

How is include abc.do more obvious than do abc.do?

I was referring to stuffing multiple programs into a single file. In other words, include bop.do, include twist.do, include pull.do is more obvious than include programs.do.

On file/folder structure donno, need to see it to make recommendations. In general - go with one file only.

Don’t look now, but this project includes about 500 do-files. :-s

On trace: seriously?

Yes. Is this unusual? I use trace extensively. I think it’s my favorite debugging tool in Stata.

adopath ++

Perfect. I looked up “17.5 Where does Stata look for ado-files?” in the manual. It said “Stata looks for ado-files in seven places”, and made no mention any method of modifying the search path. I took this to mean that there was no way to change this behavior. I am delighted that adopath exists. I think it will solve all of my problems.

Aside from the above, note the Warning given in the help for include.

Aha! This warning would have been nice to see when I, er, called include from within program define. I think this explains why it is possible (if inadvisable) to include a program (which ends with an end) inside another program: when the parent program is called, it’s no longer defining a program, so it’s happy to define a child program. Sadly, the explanation for why not to do this seems incomplete. The only reason given is a performance hit, which is my case is minimal. Logically, it still seems like it should work. The behavior that the manual guesses I might have hoped for is actually not what I expected; rather the actual behavior is what I expected. I do not see how this warning explains the “program inner not found”/“program inner already defined” error.

TL;DR

I don't understand this error, but regardless I'm going to refactor my code to use adopath, which sidesteps the issue.

[Robert Picard]

Glad to hear that a local and project-specific "ado" directory added to the adopath will solve your problems. It's a useful strategy in terms of program development because you compartmentalize your code, without fear of messing up unrelated projects (e.g. each project has their own local version of a utility you are working on).

With respect to being able to track which do-file uses which program, you can have your ado-program self-declare its dependency. Here's a silly example

*! version 1.0.0, 04aug2015, Robert Picard, [email protected]      
program hello_world

    version 13
    
    dis "Hello World"
    
    // automatically track dependencies for this program
    findfile "hello_world.ado"
    project, original("`r(fn)'")
    
end

You can then use

project xyz, list(concordance)

to find all do-files that used the hello_world command.