General questions: Set up time to run a do file, "Live" data, and Model implementation

[Marvin Aliaga]

Hello,

I have a couple of general questions regarding Stata that I hope you can answer or at least give some information.

1. Is there a way to automatically run a do file at a certain time? For example every day at 9 am?

2. Can Stata read live data? For example, let's say I have a system that tracks the numbers of violent incidents in a mental institution. Staff enter data regarding the incident into the system as soon as they occur. My questions is, can Stata be linked to that system continuously? The point is to avoid the step of downloading the data from his system and then imported into Stata.

3. Let's say that I develop a model that predicts which student will be involved in acts of indiscipline. Let s say that I know that 12-15 years old, men and students with previous act of indiscipline are more likely to be involve in acts of indiscipline. I know that there are some commands such as margin where you can enter specific values of the model and you get the predicted y. But is there a software that I can use to develop a "Dashboard" where Teachers can enter info of their students and see which one are more likely to be involve in indiscipline based n my model? Or Stata can do this?

I know these are very general questions but I would appreciate any responses.

Thank you,
Marvin

[Carole J. Wilson]

Regarding question 1, see Sergiy Radyakin's post here

[Clyde Schechter]

With regard to #1, it depends on your operating system. Stata can be run in batch mode, and if your operating system allows you to set up regularly scheduled tasks, you can do it that way. See http://www.stata.com/support/faqs/windows/batch-mode/ for details relating to Windows.

With regard to #2, I believe the answer is no. Stata keeps its active data in memory. I am not aware of any feature that would allow your system to notify a running instance of Stata that there is new data waiting to be -use-d and have Stata read it in in real time. Somebody else may know of a way to do this, but I don't.

With regard to #3, one could imagine a way of getting Stata to do this. But I wouldn't think Stata is the best tool for that kind of job. That's more the kind of thing that you would set up in a relational data base structured around a query language, I think. Also since you're dealing with information that sounds potentially sensitive, there may be privacy and confidentiality issues. Stata is not really set up to handle issues of access and data security, so again it seems the wrong tool for the job.

[wbuchanan]

Some of what you want is probably addressed with the Numerics By Stata license where you would essentially be embedding Stata inside of another application.

With regards to #2, it depends on the DB architecture and permissions. Basically you would set up an on commit trigger that would fire a call to the system to execute a do file that would run the process, but there wouldn't be a good way of having the functionality directly implemented within the DB environment aside from the option mentioned above.

For the third question, I would go with @Clyde Schecter. Once you have all the coefficients for any effects of interest you could push them into a database table and use it to look up the values. Refitting the model again would be costly and probably bog down the system, but it's fairly easy to build look up tables that could be used by other applications.

And for #1, see man cron.

[Marvin Aliaga]

Thanks everyone and I am sorry for the late reply. So basically the answer is No for all my 3 points.

My first 2 questions are because we want to know when an staff member have 3 incidents in a period if 6 months, I can do this manually too every day but I just was wondering if there is an option to read live data or to execute a do daily. I think it will become to complicated if I do the batch mode thing.

Regarding question 3. Can anyone recommend me a good software/application to deliver this? I am completely new with this.

thank you,
Marvin

[ben earnhart]

Are you in a Unix/Linux environment or a Windows environment?

If it's Windows, ASP.net and SQL Server with Active Directory users and groups managing security permissions seems the way to go.

If it's 'nix, PHP with MySQ is a popular combination. Security permissions, I'm not sure.

Not sure you want to go down the road of becoming a web developer, though.

It might be simplest if you ran Stata in batch mode, kicked off as a cron job ('nix) or scheduled task (Windows). Having the Stata job query for new information once a day should be good enough, right? Then it dumps the results into a file that has restricted permissions. No live "what if" type queries this way, but it gets you most of what you want in a reasonably simple manner. You might anticipate likely "what if" scenarios, and spit those out as part of the .do file. It might seem so at first, but batch mode isn't all that complicated.

[Marvin Aliaga]

Thanks Ben.. do you have any comments for my question 3?

[wbuchanan]

Marvin Aliaga the answer to your three questions is not no, but you need to use the right tool for the job. Trying to make Stata do this is analogous to trying to measure the dimensions of a room with a screwdriver (wrong tool for the job).

My first 2 questions are because we want to know when an staff member have 3 incidents in a period if 6 months, I can do this manually too every day but I just was wondering if there is an option to read live data or to execute a do daily. I think it will become to complicated if I do the batch mode thing.

To do this, you create a TRIGGER with the ON COMMIT property set which would run a query along the lines of:

SELECT EMP_ID, COUNT(*) AS incidents
FROM TABLE_NAME
WHERE CREATED_DATE BETWEEN GETDATE() - (your SQL instances function for 6 months) AND GETDATE() -- GETDATE() is the SQL Server version to get the current system date
GROUP BY EMP_ID
HAVING incidents >= 3

So every time the table gets updated it runs a query to count the number of incidents within the specified window. (There are more efficient way to do this, but I'm getting ready to head out of the office and just wanted to illustrate a basic idea).

With your last question, you could do this using HTML/JS if you know the coefficients used to score/multiply values and bypass using a dashboard.

I thought this was posted yesterday, but I guess it wasn't. In either case, here were the two cents I was going to throw in yesterday afternoon.

[Marvin Aliaga]

Thank all! i will do a little research and see whats the best option for me.

[ben earnhart]

If you're going all the way and becoming a web developer, then pretty much any scripting/programming language should be able to let people do what you want them to be able to do for #3. I was thinking, though, that if you could anticipate what they might ask it, you could compute likely scenarios in Stata, and avoid needing to become a web developer.

[wbuchanan]

ben earnhart I think the intent was to have a web interface for users to be able to use a webpage analogously to a calculator. That said, if there are a finite number of combinations then I would definitely suggest what I think you are also suggesting (e.g., create a lookup table with coefficients from a model/marginal effects) and do the prediction/classification using user specified values in conjunction with the established coefficients.

[ben earnhart]

wbuchanan -- I wasn't even thinking of anything that fancy. I was envisioning just a fairly long document. If he needs to do lookup tables in SQL, might as well just compute things in real-time, unless the model is complicated to implement.

[wbuchanan]

ben earnhart the problem with the computation in real time would come down to all of the IO required to move data into/out of Stata and to do the same with the user input and results. If you push a look up table into the SQL backend for a website then you just write a SQL function that takes the parameters from the user and does the look up (then you also get the benefit of having the function compiled and not interpreted at run time to save a little bit of resources). In either case, I think we all would agree that short of the Numerics By Stata licensing - to embed Stata in another existing application - it wouldn't really be practical to try using Stata to function in the way that the OP is hoping.

[ben earnhart]

What I was envisioning was something like:

• Once a day, a Stata .do file runs, kicked of as a cron job or scheduled task.
• That .do file grabs the most up-to-date information, presumably via ODBC.
• It generates a nice summary, and a bunch of tables encompassing (at least approximately) anything anybody is likely to want to look up.
• It saves the results to a shared folder (with appropriate permissions), or a folder on a webserver (also locked down with appropriate permissions).

Covers most of the bases, w/out needing to become a web developer. Takes a little interaction with the DB admin to get the ODBC going, and a little interaction with the Sys Admin to get permissions on the shared folder/web folder set up correctly. Stata *can* generate pretty .html.

Alternatively, he becomes a web developer. If the model isn't too complicated, he should be able to make PHP or ASP (or other languages) emulate Stata for the predicted values, He didn't say what model he was trying to use, but most of the time, computing predicted values is something that is straightforward to compute once you have the model parameters.

[wbuchanan]

ben earnhart that could work too. I don't think it would require becoming a full fledged web-developer to do something like a prediction/look up thing in a web page, although having web-dev skills would definitely make for a nicer solution that could be maintained a bit more easily by other devs. If Marvin Aliaga needed to cover all the bases in terms of descriptive statistics and could live with your suggestion of setting up a schedule to run the tasks, my package eda would provide fairly extensive coverage for any types of graphs/descriptive statistics that would typically be a first step in exploratory data analysis.