🐛 Working with temporary frames

[Sergiy Radyakin]

Dear All,

consider the following code demonstrating the problem:

version 16.0
clear all
frame dir
pwf

frame create _000000
frame dir
pwf
frame change _000000

frame create __000000
frame dir
pwf
frame change __000000

When executed in Stata 16.1 it stops on the last command of the brown block, where there is seemingly nothing different from the previous (green) block of code, just the name of the frame is different:

. frame change __000000
frame __000000 not found
r(111);

Despite the previous frame create command has not resulted in any errors, the requested frame is NOT created, as evidenced by the output of the frame dir command:

. frame create __000000

. frame dir
  _000000  0 x 0
  default  0 x 0

I believe Stata has 2 defects, which are:
1) failure to create the requested frame;
2) failure to report that the requested frame can not or may not be created (if there is a naming convention or anything that justifies #1).

I believe the problem may be due to the specific pattern of the name which coincides with temporary names which may be generated by Stata internally, which is what I am using in my actual code:

tempname WSPACES
        frame create `WSPACES'
        frame change `WSPACES'

The exact Stata version is:
Stata/MP 16.1 for Windows (64-bit x86-64)
Revision 21 Jan 2021
Compile number 873

I tend to think that Stata 16.0 didn't have this problem, or it was not showing up in my runs of equivalent code, (though I can't be 100% sure). If anyone has a way of confirming the problem in Stata 16.0/Stata16.1 this may be helpful.

The green paragraph of code is not essential for provoking the problem. It only illustrates that the problem does occur only for some names of frames (same result for __000001, 000002, etc as for the __000000).
Note also that it is important that you run the code in freshly open Stata only! (not reusing any old session, as it seems that there is something that clear all doesn't clear). The example code will run if you type tempvar tmp in the command line before running the code, even if the variable with that name doesn't exist in the current data/frame.


Note also, that Stata's manual guides us on the use of the temporary naming generators:

tempvar assigns names to the specified local macro names that may be used as temporary variable names in a dataset. When the program or do-file concludes, any variables with these assigned names are dropped.

tempname assigns names to the specified local macro names that may be used as temporary scalar or matrix names. When the program or do-file concludes, any scalars or matrices with these assigned names are dropped.

tempfile assigns names to the specified local macro names that may be used as names for temporary files. When the program or do-file concludes, any datasets created with these assigned names are erased.

But doesn't mention any special way of generating temporary names for frames. The only reference I can think of was my own earlier post:
https://www.statalist.org/forums/for...eneral/1568820
and an earlier post by Dave Airey
https://www.statalist.org/forums/for...eneral/1519740

both of these examples used tempname for the same purpose and hasn't received any objection of such use from StataCorp's staff. Based on reply of Alan Riley (StataCorp) there were some changes scheduled to take place between version 16.0 and 16.1 to the relevant Stata code.

It would be helpful if I could get a clarification/guidance on the allocation of the temporary frames:
- what command should be used to generate the temporary name, any of existing 3 mentioned above or yet another one is needed for this purpose and
- whether this is safe to use in all versions of Stata or only from a certain version number, as well as
- behavior under the version control.

Thank you, Sergiy Radyakin

[Antonio Alfonso]

I don´t have error with it code. sorry

[Sebastian Kripfganz]

I can confirm Sergiy's observations.

[Sergiy Radyakin]

Antonio Alfonso , Sebastian Kripfganz ,

thank you for taking the time to run the example code.
Since you are posting opposite results, would you mind sharing the exact Stata version numbers/platforms you are running it in?

Thank you, Sergiy

[Alan Riley (StataCorp)]

Sergiy Radyakin has found a bug which we will fix in a future update.

To create a temporary frame, the command tempname should be used to obtain the name. I have not tested all possibilities yet, but I believe if tempname had been used rather than a hard-coded name which happens to look like a temporary name, then everything would have been ok.

We need to state in our documentation that tempname is to be used for obtaining names for temporary frames just as with temporary macros, scalars, and the like.

We also need to make clear in our documentation that, while not prohibited, hardcoded names which look like temporary names but which were not obtained from tempname, tempvar, or tempfile should not be used. Stata doesn't prevent you from creating names beginning with double underscores, but these should be viewed like reserved system variables and only obtained or used with official facilities.

[Leonardo Guizzetti]

Edit: Crossed with #5. Thanks for confirming the bug, Alan.

I can confirm Sergiy's observation on my Stata, on a fresh instance, setting version to be 16.0 or 16.1:

Stata/MP 16.1 for Windows (64-bit x86-64)
Revision 21 Jan 2021

I also note from some quick experimentation that frame names that begin with -__[0-9]- and have at least one letter or number following seem to not be created. I tend to agree with Sergiy that it probably has to do with some preserved namespace for the -tempname- facility on the basis of that pattern. It would make sense that if this is protected namespace, then -tempname- has is exempted.

[Antonio Alfonso]

stata/MP 16.0 for win32

[Sergiy Radyakin]

Dear Alan,

thank you very much for confirming the bug status and continuing to improve Stata!

You wrote "I believe if tempname had been used rather than a hard-coded name which happens to look like a temporary name, then everything would have been ok.".
Unfortunately my experiments do not confirm that. I've encountered the problem with it in the actual code, which was allocating temporary names with the tempname statement as you describe (see first post above mentioning WSPACES).
Even with that, there is no such problem for matrices (where temporary names are also expected to be allocated with tempname):

clear all
matrix __000000=1,2,3
matrix list __000000

The minimalistic example was using hardcoded names purely to isolate the issue and to avoid any ambiguities of how the names were allocated. I do utilize standard Stata's facilities to generate objects of temporary nature to the fullest extent possible, but where there is no such a possibility have to resort to other means. Where Stata lacks this (imho) is:

1. generating a temporary folder for example:
   1. to safely unzip a zip archive, knowing that I am not overwriting anything else, or
   2. to be able to flush unpacked zip later with a single command, not worrying to go by some file index, or
   3. to stage the content of an XLSX or other similar file format,
   4. to prepare a session folder for another external program which produces a set of files with various nature.
2. generating a temporary file with a particular extension (e.g. a temporary *.tex file, or a temporary *.xlsx file) which can be executed using the default OS associated program.

I believe I was in communication earlier regarding both of these issues. See for example here:
https://www.statalist.org/forums/for...eneral/1553253
where Hua Peng and Zhao Xu from StataCorp have advised on how to deal with the issue, but still, native methods for dealing with temporary folders and temporary files with particular extensions would be very much welcomed, at least for the tasks that encounter commonly.

You've also written that documentation will be enhanced to clarify these issues. Thank you, that would be a very welcomed enhancement.

Finally, a somewhat related, but also somewhat unrelated issue. Consider the following two fragments:

First:

clear all
sysuse auto
tempvar len
generate `len'=strlen(make)
save "C:\temp\delme_tmp.dta"

And second:

clear all
use "C:\temp\delme_tmp.dta"
tempvar ttt
generate `ttt'=price+weight
list

Running first in one Stata session and second in a different Stata session (subsequently) will result in an error, although both of the procedures appear to be legit.

In short, would it be possible to have the second code 'realize' that the candidate name that it is about to allocate is already taken, and generate a next suitable name in this case?

Longer description - I commonly work with datasets, which are being produced as part of some automation procedure, which cuts data by years, countries, etc and contains a bunch of artefacts of such manipulation in the form of __###### variables. While mostly they could be harmlessly dropped after opening the data, their mere presence there is not noticeable until the collision occurs, (and usually it occurs not on the first run, and not even in the first round of interaction with the consultant or client). It takes a bit of back and forth to find out all the open/import/convert commands that can bring such data into memory under different scenarios, before the names allocated with tempvar later in my code no longer collide with any artefacts in the data. And yet there are datasets where useful data happened to be stored in those variables, so mechanically adding drop _* to every data open statement is an overkill. When I am thinking of a temporary object - I'd like to think of it as "something that I don't care where it is stored or what it is called, but something that definitely does not coincide with whatever exists there already, and will be surely deleted after I leave the scope". Stata seems to define it as "everything that starts with __".

For the temporary files, I'd probably prefer the opposite way - a garbage collect when Stata session is started. For example, when Stata initiates session 17, then all files in the temporary folder that are marked as belonging to session 17 are to be flushed (they may remain there due to e.g. power failure and hence no deletion in that originally started Stata session. This will guarantee that all the temporary files attributable to the current session are indeed produced within it. (and if those files can't be deleted, then perhaps it's a bigger problem, and one may find out about it a couple of hours or days later, depending on the luck and the size of data).

Sincerely and best regards, Sergiy

[Sergiy Radyakin]

Antonio Alfonso ,
did you manage to run Stata 16 on Win32 somehow?
According to these references win-64 is required for Windows platform: first and second.
Thanks for taking time to look into this issue. Best, Sergiy