Pausing a loop while running

[Doris Rivera]

Dear Statalist, I am running several loops for different files that might take weeks (so many obs). However, I have a limited access in the work place, so, I am wondering if pausing a loop while it is running will make me lose all the progress. If not, would I have to run in the command section “pause on”?
Lets say that a loop will last 5 days, but I only have two days of availability. Is it possible to tell Stata to start working Monday at a given hour and pause on Tuesday at a given hour. And then the next Monday start again in the same loop until Tuesday and so on? If this is possible, can anyone give a hint on how to do it? I mean, I need to tell this to Stata since the very beginning in the do.file (before starting all the loops?)

Any help is much appreciated.

[Mike Lacy]

I have two ideas here, which might be helpful, but which would benefit from comments by someone else more knowledgeable here:

1) All operating systems I have worked with provide a way to change the so-called "priority" of a running program. Programs with low priority will continue running "in the background" but the operating system will (more or less) only let them do work when other processes aren't using the computer. My idea here, then, would be that you start your Stata program running, and then change its priority to a low value when you want it pause, and then reset its priority to a higher level when you want it to run again. I have personally only done this "by hand" on Windows and UNIX systems, but I'd presume that someone with the right skills could do this with some kind of batch control program so you don't have to do that.

2) If your program is such that a -pause- and some other commands can be inserted into your loop, something like the following could be done. My sketch below is clumsy, but I hope illustrative. The stop times were just what I used while experimenting and you'd replace them with what is relevant to you.

cap prog drop newstop
prog newstop
   // You could type in this second stop time by hand at the pause prompt,
   // but having it defined as a command is easier.  A program with multiple
   // stop times is possible, too.
   scalar stop = clock("11oct2022 10:05:00", "DMYhms")
end
//
//
clear
// Data to work with.
sysuse auto
drop make
//
pause on
// Initial time at which to pause
scalar stop = clock("11oct2022 10:00:00", "DMYhms")
//
forval i = 1/10000 { // just an example loop
   local cdate = c(current_date)
   local ctime = c(current_time)
   if clock("`cdate'  `ctime'",  "DMYhms") <= stop {
      pwcorr * // for example
   }
   else {
      pause   
      // Type "newstop" and then "end" at the pause prompt to resume execution
      // or "BREAK" to completely halt.
   }   
}

[Daniel Schaefer]

If I understand correctly, both of the solutions that Mike provides in #2 will require the Stata application to continue executing in the background (even if it's not processing any data) and all of the associated data will have to stay in RAM. If it is okay to keep Stata running while you don't have access, then these solutions should work well for you - but in that case, I wonder why you wouldn't just leave Stata executing the loop while you aren't there?

If you have to stop Stata completely at the end of the day, or you can't consume what is apparently a large amount of memory when you aren't there, or if you are worried someone will close Stata when you aren't there, then you will probably need to save your progress to the disk and write some code that is smart enough to pick up where you left off. The implementation will depend on the details of what you are doing. If possible, I might subset the data into smaller batches that will execute while you have access.

[Doris Rivera]

Dear Mike Lacy and Daniel Schaefer, thanks a lot for your answers. Following the code provided by Mike, I assume then that the pause is done between commands? Just to see if I understood well how this works. My loops have to be inside Mike's loop? Let say I have several steps in one loop, then, this code would allow me to pause the process if going from step1 to step2 (or from loop1 to loop2), right? But, what if step1 is long enough that takes more than my hours of work? In such a case, it wont stop until step2, right?

If I understood well what Daniel says, then, even doing something like Mike's suggestion, it will continue to use memory from the system? That would be a problem to me, since, the point is that the process I run is so intensive in memory that it takes almost all the memory from the server, reason why I can only work on the server two days per week (even though my loops take more than two days to run).

Sorry for bothering you with my questions, but this is beyond my basic level of Stata

[Daniel Schaefer]

If I understood well what Daniel says, then, even doing something like Mike's suggestion, it will continue to use memory from the system?

This is my understanding. Basically, in order to "remember" where you are in the loop, Stata will need to store the current state of the program in memory until you need to use it again. This is why I suggest that you may need to programmatically "remember" the state of the loop by writing some data to the hard drive so that you can clear the data out of RAM without loosing your place. As I said, the right way to do this will depend on the details of what you are doing.

Of the two of us, Mike is the more experienced Stata programer. If he has an alternate suggestion, there is a good chance that he is right. That being said, I'm fairly confident that "remembering your place" will mean "keeping the data in memory" unless you explicitly write your progress to the hard drive and stop execution of Stata.

[Mike Lacy]

Daniel, I might be experienced, but I'm not sure I know more :-}. Yes, the Stata solution I proposed would keep control of memory. Regarding Doris's comments/question about step1 taking a long time, yes, the pause I'm proposing would have to be between steps, so it sounds like the Stata pause wouldn't help here.

Regarding the idea of remembering where the program is and restarting later: I believe this kind of thing goes by the name "saving the program state." I can't imagine that a Stata user could do this, as there are all sorts of aspects of a program that I don't think a user's program could access to save. (The exception to this would be if the procedure in question were completely user-written, so that the relevant variables/matrices would be known.)

One more thought: I have a vague memory from a few decades past of some kind of background/pause option on a UNIX system that I think involved the OS saving the program state and then later resuming later at the user's request, but that might be a false memory. This makes me think that any answer is going to require knowing what OS is being used. Or, perhaps better, perhaps whomever manages the server Doris uses will have some advice.

[Doris Rivera]

Dear Mike and Daniel, thanks a lot for your ideas. At least, I learned a couple of new things very interesting. I think that as suggested by Mike I should talk with the person in charge of the server to seek a solution. Thanks again!