Bug? Unexpected behavior with -by-, -file-, and an expression

[daniel klein]

I cannot even explain this in words:

clear all

input str3 foo
"foo"
"bar"
end

tempname fh
tempfile fn

file open `fh' using "`fn'" , write

sort foo
assert foo[1] == "bar" // <- "bar" is now the first obs.

by foo : generate x = 42

file write `fh' (foo[1]) // <- foo[1] == "bar", right?

file close `fh'

type "`fn'" // <- should be "bar", but we get "foo" instead

The above code should result in writing "bar" to a temporary file. Yet, it writes "foo" to the file:

(output omitted)
. type "`fn'" // <- but we get "foo" instead
foo

If I leave the by out of the code, I get the expected result.

(output omitted)
. * by foo : generate x = 42
(output omitted)
. type "`fn'" // <- but we get "foo" instead
bar

Also, if I refer to foo[1] before writing to the file, it also produces the correct result.

(output omitted)
. by foo : generate x = 42
. display foo[1]
bar
(output omitted)
. type "`fn'" // <- but we get "foo" instead
bar

And, I can produce the correct result with in-line expressions:

(output omitted)
. file write `fh' "`=foo[1]'" // <- remember that we should get "bar"
(output omitted)
. type "`fn'" // <- but we get "foo" instead
bar

I have intentionally omitted a version statement because the same strange behavior occurs in Stata 16 to 18.

What is going on here?

[Clyde Schechter]

I have no insights into why this is happening. All I can say is that I can replicate your results on my system (Stata MP4, Windows) in both version 18 and version 17. I think you need to bounce this to Tech Support.

[daniel klein]

Thanks for confirming. I believe it has something to do with by. It appears as if by somehow messes up the indices.

FWIW, another way you can get the correct result is insering a simple list

. by foo : generate x = 42

. 
. list

     +----------+
     | foo    x |
     |----------|
  1. | bar   42 |
  2. | foo   42 |
     +----------+
(output omitted)
. type "`fn'" // <- should be "bar", but we get "foo" instead
bar

This is puzzling.

I will notify tech-support, although I am still having trouble putting the setup into verbal description. I will point tech-support to this thread.

[daniel klein]

I should mention that the best workaround is probably to run by before opening the file. That is, code

(code omitted)
sort foo
by foo : generate x = 42

file open `fh' using "`fn'" , write
(code omitted)

The reason I cannot do this (easily) is that the problem appears in a much longer ado-file in which I am opening (and closing) the file in one program and writing to the file in multiple other (sub-)programs. Think of the setup as something like:

program main
    
    tempname fh
    tempfile fn
    
    file open `fh' using "`fn'" , write
    
    write_stuff `fh'
    
    file close `fh'
    
end


program write_stuff
    
    args fh
    
    by ...
    
    file write `fh' "stuff" _newline
    
end

Anyway, the post in #1 suffices to illustrate the problem. And, I am pretty sure that this is a problem, i.e., a bug, not a feature.

[Daniel Schaefer]

This is really very strange behavior. I think we might be seeing some kind of a race condition here. Reads and writes to the file system are sometimes done asynchronously because the cable to the hard drive represents a major bottleneck. You wouldn't necessarily need a multithreaded version of Stata to have this kind of implementation. It is possible that the two file commands are actually executed before the sort command finishes depending on the details of the underlying implementation of the interpreter. Maybe by and sort are on a separate thread from the file commands and the file commands use a cashed version of the data if by and sort haven't finished executing first? Then by makes sort run just a little bit longer, loosing the race against the file commands...

Note, I don't even need to refer to foo[1] before writing the file to get the correct result in Stata 18. I can print anything to the console to get the correct value at the end of the script.

clear all

input str3 foo
"foo"
"bar"
end

tempname fh
tempfile fn

file open `fh' using "`fn'" , write

sort foo
assert foo[1] == "bar" // <- "bar" is now the first obs.

by foo : generate x = 42

di "wait and write to the console."

file write `fh' (foo[1]) // <- foo[1] == "bar", right?

file close `fh'

type "`fn'" // <- should be "bar", but we get "foo" instead

[Clyde Schechter]

Dan Schaefer's theory about a race condition is intriguing. But I have some evidence that contravenes this. Even though writing a message to the Results window seems to "correct" the problem, I don't think that's attributable to delaying the -file write- commands, because if, instead, I put a -sleep- command in that place, we still get the incorrect "foo". This happens with both long delays, -sleep 10000-, and short ones -sleep 1-.

. clear all

.
. input str3 foo

           foo
  1. "foo"
  2. "bar"
  3. end

.
. tempname fh

. tempfile fn

.
. file open `fh' using "`fn'" , write

.
. sort foo

. assert foo[1] == "bar" // <- "bar" is now the first obs.

.
. by foo : generate x = 42

.
. sleep 10000

.
. file write `fh' (foo[1]) // <- foo[1] == "bar", right?

.
. file close `fh'

.
. type "`fn'" // <- should be "bar", but we get "foo" instead
foo

[Daniel Schaefer]

Hmmm, interesting. I suppose in a multithreading situation it depends on which thread is set to sleep, but then again, it seems like temporarily blocking the main thread with -sleep- should have the same effect as blocking the main thread with -display-. So probably not a race condition then.

I thought there might be something special about -display- since it does some IO work by writing to the console, but it seems like any old command after by (except, apparently, -sleep-) avoids the bug.

clear all

input str3 foo
"foo"
"bar"
end

tempname fh
tempfile fn

file open `fh' using "`fn'" , write

sort foo
assert foo[1] == "bar" // <- "bar" is now the first obs.

by foo : generate x = 42

generate y = 43

file write `fh' (foo[1]) // <- foo[1] == "bar", right?

file close `fh'

type "`fn'" // <- should be "bar", but we get "foo" instead

As you might expect, I can reproduce with -bysort- as well.

[Hua Peng (StataCorp)]

This is a bug in file write, which does not properly reset the range of observations after a by. Instead of using the whole range, it uses the range of the last by group. Calling any commands reset the range after by and before file write circumvents the issue.

We will fix the bug in a future update.

[daniel klein]

Wow, that was a fast diagnosis of the problem. Impressive. Thanks.

[daniel klein]

StataCorp has fixed the problem in the 20dec2023 update. Thanks.