Read-write access in large number of loops

[Cydney Bruce]

Hi Everyone,

So I am running a simulation study with a lot of different factors to look at, which means we end up with a lot of loops.
Each program runs independently without issue.

The simulation itself will run without issue for quite a while (around 2 hrs which saves around 11,000 results) and will trip over at a different point each time.

I can't give you the exact code just because the program is quite large, but a simplification that I think will highlight the problem is as below:

foreach i of numlist 142 142.5 143 143.5 144 144.5 145 145.5 {
    foreach j of numlist 146 146.5 147 147.5 148 148.5 149 {
        foreach k of numlist 30 31 32 33 34 {
            foreach l of numlist 35 36 37 38 {    
                forvalues m = 1/10 {

                    clear
                    sysuse surface.dta
                    
                    keep if `i' < latitude & latitude < `j'
                    local n = _n
                    gen ref = _n
                    save temp.dta, replace
                    
                    clear
                    set obs `n'
                    gen ref = _n
                    merge 1:1 ref using temp.dta                
                    
                    keep if `k' < longitude & longitude < `l'

                    gen closestate = c(seed)

                    save "results_`i'_`j'_`k'_`l'_`m'.dta"
                }
            }
        }
    }
}

Now I realise no one would ever want to do the above, however it highlights the main issue. I have minimised the number of replace saves the program does as I was finding that the error would be read write access which I think was caused by the size of the dataset, meaning Stata hadn't quite done saving before it was trying to overwrite on the next loop. The final results are now being saved line by line to combat this problem. However, due to the program I am using, that merge step is unavoidable meaning at a point the program will trip when trying to save this temporary file.

Short of labelling every single temp file separately so that there can't be a read write issue, I have reached the point that I am out of ideas for how to get this thing to run through.

Any advice would be appreciated either on how I can stop Stata tripping over read write access, or on how to go about restarting at the point of tripping (I am saving the state in each results dataset with the thought this could be used to "pick up where I left off"?)

Thanks and best wishes,
Cydney

[Mike Lacy]

It can be true in Stata that repeated attempts to use -save- will lead to competition for access, in which case inserting the -sleep- command right before the relevant save command can help by pausing execution for some number of milliseconds. You might try -sleep 100-, and increase or decrease the 100 as necessary.

That being said, because you haven't shown us the actual error message you received, I am not able to tell for sure what that might have been. For example, I might guess that you are receiving some message to the effect that such and such file is busy or can't be accessed, and I'd presume from what you say that you think (know?) it's the -save temp.dta, replace- where problems occur, but you might be wrong and without more information about your error message, I can't be sure about that.

I also don't know what you mean by "final results are now being saved line by line," so if that ploy is indeed necessary, knowing exactly what you mean by that would be helpful.

[Benno Schoenberger]

Can you perhaps briefly explain what the loop over m = 1/10 is supposed to do ? Since the macro 'm' isn't used anywhere except when saving your results, it looks like the same results are saved 10 times each time. In this respect, you could save running time and thus also file write operations if you omit the useless loop m=1/10.

[Cydney Bruce]

Thank you both for your responses.

Mike, so i have reran this morning and the error that the program gets is:

 file stratified.dta is read-only; cannot be modified or erased

The file stratified is the equivalent of temp above in my program, (extract from actual program included below).

This program ran up until result (1_2_1_4_10), but I have also seen it break with the same error on (1_6_1_6_7) so it has made it past this point previously without error.

egen StratID = group(`varlist')
save "temp1.dta", replace

sum StratID
local stratno = r(max)


forvalues z = 1/`stratno' {
            clear
            use "temp1.dta"
            
            keep if StratID == `z'
            
            simple_ra
            
            gen rando = .
            replace rando = 1 if assignment == 0
            replace rando = 2 if assignment == 1
     
            save "StratTrial`z'.dta", replace
        }

        forvalues z = 1/`stratno' {
            
            if `z' == 1 {
                clear
                use "StratTrial`z'.dta"
                
                save "stratified.dta", replace
            }

            
            if `z' != 1 {
                clear
                use "stratified.dta"
                append using "StratTrial`z'.dta"
                sort randoid
                
                save "stratified.dta", replace
            }
}

As for how the results save, previously the program appended the memory to a results dataset after each iteration

append using "output\Main_Results.dta

However this dataset was experiencing the read write error much earlier as the dataset was growing as the program went on. At this point it would run for around 30 minutes (and produced about 2000 results) when the program would fail. I did try the sleep command in this iteration of the program but sleep 1000 didn't slow it down enough (although did mean the program ran longer and produced more results).



Benno, the m=1/10 loop is how I am implementing the simulation aspect. It is currently set to 10 because I was trying to get a feel for how long the program will take to run but currently it cant even handle 10 runs (that is why it is not doing anything but holding a place in the results). I do know that there is a simulate command, however I had a lot of trouble getting this to work as it does not seem to be compatible with the user command RALLOC which cannot be substituted in the program.

Best wishes,
Cydney

[Mike Lacy]

I don't have any great insights here, but here are some thoughts:

1) I would not use -sort- inside the loop. In the example code, I don't see it as necessary, and it will be slow, and *possibly* cause timing issues.

2) Rather than -append- the files one by one, I would create a list of files and append them in one command outside the loop. That might help Stata with avoiding conflicts.

append using StratTrial1 StratTrial2 StratTrial3 ....

3) In my experience, -sleep 1000- is quite long, and the fact that that didn't solve your problem makes me suspicious about the nature of the problem. Are you perhaps using a server where others' uses could be competing?

4) I'd insert a bunch of display echos to be sure about exactly where the errors were occurring. Perhaps you've already done this.

5) Can you try your simulation with the same # of reps but smaller (sample) files?

[Daniel Feenberg]

What does ls -l or attrib say about the file permisions for the file? What about setting the permissions explicitly with a shell command after each save? If Stata doesn't wait for the save to complete before executing the next command, that seems like a bug for Stata tech support to address.

[Cydney Bruce]

Hi Mike,

I appreciate your thoughts.

I have had a go at a few of your suggestions, previously I was running on a server but I am now making sure both Stata and where the files save is local.
I also didn't realise that append could be used in that way and will definitely be taking it forward!
I also realised that while the sort is necessary (the records need to be in order for the next step) that this does not need to happen after each append and has been moved to the end.

As an additional solution from someone in my institution, all statements that read data (either as use, merge or save) I have wrapped in a while loop with a capture statement so that the program will keep attempting the line until it is successful, e.g.

local rc = 2
while `rc' != 0 {
                capture {
                    save "StratTrial`z'.dta", replace
                }
            local rc = _rc
            }

Certain parts in the loop take a long time to run but so far this does seem to stop it crashing (and give the particularly intensive parts of the program more time to run while some results can still produce quickly!).


Daniel, I had not thought about setting permissions and am not familiar with shell commands so will need to look into this.
I agree it seems odd to me that Stata would continue with the next step before the previous one has finished processing but it is the only explanation I have for the program crashing (with the above changes we have now made it 45,000 files where previously it had not made it to 12,000).

Thank you for all of the feedback it has been very useful!

Best wishes,
Cydney

[Mike Lacy]

Over the years, I've heard about or experienced problems like yours, bit I can't ever recall seeing a situation in which something like that capture/while thing or -sleep- with a large value or some other "trick" was necessary to keep Stata from running into problems with disk/file access. I'm certainly not an expert here, but I'm thinking there's some odd interplay of Stata/your machine/OS/dataset(s) that's going on that shouldn't be happening. I'd encourage you submit your problem to Stata Tech Support and report back on what they say. (You could just point them to this thread.) What they say would be interesting and useful to others.