Something fishy with a tempdir

[Sergiy Radyakin]

Dear All,

I am using Stata's tempfile command to create a temporary directory. Consider the following example (which works pretty much as I expect):

clear all

program define tmptst
    version 18.0
    tempfile tmp
    local tmp `"`tmp'z"'  // important here
    mkdir `"`tmp'"'
    findfile "auto.dta"
    local dst `"`tmp'\auto.dta"'
    copy `"`r(fn)'"' `"`dst'"'
    global lastfile=`"`dst'"'
    global lastdir=`"`tmp'"'
end

tmptst
display `"$lastdir"'
display `"$lastfile"'
dir "$lastdir"
dir "$lastfile"

/* END OF FILE */

Once the program tmptst executes, it leaves after itself a folder and a file, which are saved into the globals and the outer program (the caller) may decide whether to retain or discard that output.

But if the code is modified as to remove the augmentation of the temporary folder name (in other words if one removes or comments out the line marked as important here) then the behavior is radically different. I would expect that, perhaps, Stata could decide to delete the temporary folder in a similar manner like it does with the temp files, but instead it renames the temporary folder into a different (and rather unpredictable) name. For example, after several runs with the temporary directory being:

C:\Users\MYUSERNAME\AppData\Local\Temp\ST_2c38_000001.tmp

I end up with the following artefacts:

Notice here that:
- "ST_" changes to "STU" ,
- some index is added at the end "45", "48", "4b", etc, and
- a couple of zeroes are eliminated from the name (probably to keep the fixed width of the name).

Correspondingly the outer program can't locate the temporary folder created by the inner program, nor can it pick up the results or do a cleanup.

Since I am working with sensitive data, I want to make sure that no copy of it accidentally remains on the machine that is doing the analysis, but the above process seems to automatically create such snapshots of my temporary directories (in which the results are staged) and it is not clear whether they will be deleted automatically at some later point and under what conditions.

I am currently (depending on other factors) using as a workaround either:

1. modifying the tempfile name (as shown), or
2. generating the temporary folder by the outer program and passing it as a parameter to the inner program.

Both seem to work, but I am puzzled by the above behavior observed in Stata (v18.0, Windows 10, if that matters) and not sure if that was intended.

Here is some additional information:

1. notice that the above code does not modify the current directory and no file there is open by the current process (or any other process, as I currently believe), so it must be possible to delete the folder with the temporary file's name that is created here. And if the folder can be renamed, it should be possible to delete it, right?

2. the use of tempfile command to generate a temporary directory name is inspired by the advice by Hua Peng (StataCorp) here.

3. I am aware of Bill Gould's advice to me from 2007 : "You are passing a tempfile name out of a program for use by a calling program. (That is inappropriate use of -tempfile-.
Each program, do-file, and ado-file knows only to erase tempfiles that it itself obtained from -tempfile-)." As a matter of fact I do want to take the responsibility to cleanup myself, but it seems I can't do it, because the name changes to an unpredictable one, and I have no instrument to detect or follow this renaming.

4. I can't use fixed locations, since there may be multiple instances running in parallel, so I need to rely on Stata's mechanisms to avoid naming collisions (described here).

Thank you, Sergiy Radyakin

[Hua Peng (StataCorp)]

I will look into it next week (out of office this week).