Creating notebooks from do files

[Mikko Rönkkö]

This may be a feature request rather than a question. I am teaching a quantitative research methods class where the students use either Stata or RStudio. In the computer labs we work on a data analysis problem together. At the end of the class, the students have an analysis file which they use to produce a notebook in Word format. Then they have one week to go through the notebook and write an explanation for each step of the analysis and their interpretation of the results.

Producing a notebook is something that is simple with RStudio, but requires a lot of manual work in Stata. After preparing the analysis file, the RStudio users simply use the RStudio's compile notebook feature. It runs the analysis file and produces one notebook file (PDF, HTML, or Word format) containing the R code and all the output including all plots.

To give an example, the student starts with an analysis file that may look like this

https://www.dropbox.com/s/0vcqu331xa...ise%201.R?dl=1

Then they use the compile notebook feature of RStudio to produce a notebook that looks like this

https://www.dropbox.com/s/jpuunw5z57...raw-R.doc?dl=1

The students then write explanation for each step of the analysis and interpret the results. You can check my model answer here

https://www.dropbox.com/s/w71kqgtfb5...%20R.docx?dl=1

If anyone knows anything similar for Stata, I would be very interested. I know that graphlog package produces PDF logs containing all plots, but I need the logs/notebooks in a format that the students can easily edit. Moreover, I would prefer that the students do not have to user graph export for each plot that they do. To obtain similar output using Stata, I currently suggest the following workflow:

- Set your working directory
- Start your do file with "log using assingment1, replace text"
- End your do file with "log close"
- After each graph add "graph export plotX.pdf"
- Open the Word document template from MyCourses
- Copy-paste the content of assignment1.log to the document template and insert the exported figures into right places.

If there are multiple figures, then preparing the document manually takes time. Here is the output that I would like to get (my model answer):

https://www.dropbox.com/s/8kahu9565t...tata.docx?dl=0

Mikko

[Maarten Buis]

I think graph export is unavoidable: Stata graphs by default overwrite the previous graphs, so if there is more than one graph only the last one remains available. You can use log html to transform a log file into an html file, which can be read in MS Word. This won't include any graphics, but with some use of file you should be able to automate the inclusion of the graphs. Question is: is that any less work than the workflow you propose?

[Nick Cox]

If you name a graph, it remains open in its own window until you close it or the session ends.

Try

sysuse auto, clear
ssc inst aaplot
aaplot mpg weight, name(keep1)
gen gpm = 1000/mpg
aaplot gpm weight, name(keep2)

Gratuitous advertisement for aaplot (SSC) there.

See also log2html (SSC).

[wbuchanan]

The most comparable output across languages could likely be obtained using StatWeave. However, RStudio does not create "notebooks" as much as weaved documents. The Jupyter project is the most versatile for generating notebooks and if everyone is working on Windows you can find some work that James Fielder did to integrate Stata and Python to generate an actual notebook from within Stata.

[Mikko Rönkkö]

Thanks for your quick comments.

I think that the TeX/HTML/Word conversion is not difficult with the existing open source tools. Making the Stata -> Word conversion of logs scripted seems relatively straightforward. My goal would be to providw the students one command that they could use to run a do file and produce notebook. Like

do2word mydofile.do

The hard part is including the graphs in the Word document. There does not seem to be a way to have Stata automatically export all graphics and log the location of the exported files. I can tell the students to use graph export or name their graphs and then export using the graph viewer, but if there are, say 20 graphs that they have produced, including the graphs manually is tedious.

[Nick Cox]

20 graphs? In my most recent teaching that sounds similar, my instructions were to include no more than 5 graphs, and so to think about which were most helpful.

I know that's a quite different issue, but it's just my experience. With that scope, copying and pasting the graphs into MS Word is then the least difficult part and the students seem happy. (They do have means of telling me otherwise.)

[Doug Hemken]

More solutions:
Use R markdown with a Stata engine, see http://www.ssc.wisc.edu/~hemken/Stat...aMarkdown.html
Use the Markdoc package, in Stata

However you will still need to use graph export. Students should perhaps be using graph combine or graph, by() to allow comparison among graphs? which would reduce the number of graph files you have to export.

[haghish]

Mikko Rönkkö

1. markdoc is a general purpose literate programming package for Stata
2. The package offers similar functionality compared to R-Markdown.
3. It recognizes 3 markup languages (Markdown, HTML, LaTeX) for writing the dynamic document, which means it's a combination or R-markdown, R-HTML, and R-Sweave at once.
4. It can export HTML, LaTeX, PDF, Microsoft Word, Open Office, etc.
5. It has a built-in JavaScript syntax highlighter...
6. The link to my website is not complete yet. Please read the help files!
7. installation: ssc install markdoc
8. enjoy! :-)

You might also have a look at Weaver package ssc install weaver

[wbuchanan]

Mikko Rönkkö a notebook is an interactive web-based application that uses a kernel to pass commands to the server interactively and allows the user to change, modify, and/or update previous commands interactively. It seems like what you want is more analogous to what StatWeave and markdoc provide which is a static document in the vein of what Donald Knuth called "literate programming." If you want the students to be able to interact with the commands, you would need to use the Python plugins that a few folks (@James Fielder among them) have developed. After installing, you can spin up an iPython or Jupyter server and have a notebook interface.

[Mikko Rönkkö]

Thanks for the suggestions.

@Nick Cox: I want the students to document how they worked with the data. An analysis session might consist of a couple of scatter plot matrices and kernel density plots, then a few regression models and with the usual post-estimation plots for each model, followed by a marginal effects plot. It can add up to about 20, but the number is not always that high.

haghish The markdoc package looks really promising. If there was a way to automatically export the graphs and include them in the file, that would be exactly what I need. But even without that, it should simplify the process a bit and syntax highlighting is nice to have.

[Nick Cox]

"I want the students to document how they worked with the data"

Me too.

But I will agree that different goals will call for different styles.

It's just my impression that most of these extras for report production are very complicated and difficult to understand unless you wrote the code yourself, which won't be true of your students. I would recommend not recommending anything to your students you've not tried out yourself.

[Attaullah Shah]

haghish, does the markdoc support MS word, if yes, can you give a sample example just like you used an example in the help file using html

[haghish]

Mikko Rönkkö

Both weaver and markdoc do that. However, the graphs most be exported first and then included in the document. weaver, however, makes it easier because it has a command for importing and resizing a graph in document. But I think for students, markdoc is the best package. Here's an example to insert a graph in a pdf document in both markdoc and weaver. I also upload the dofile.

p.s. If you want a single command to create a graph, export it and include it in the document, it shouldn't be hard to do since it only requires putting these three commands in a new program. But I am not sure if it'd be a good practice.

************************************** MarkDoc example 
set linesize 80
qui log using example, replace smcl

//OFF
sysuse auto, clear
histogram price 
graph export price.png, replace width(250)
//ON

/***
Adding a graph using MarkDoc
============================

In this example, __Markdown__ syntax is used for writing the document which is 
easier than __HTML__ or __LaTeX__, particularly for students.

![](price.png)
***/

qui log c
markdoc example, replace exp(pdf) 



************************************** Weaver example 

weave using example, replace 
sysuse auto, clear
histogram price 
graph export price.png, replace

txt *-Adding a graph using MarkDoc-*                                             ///
    In this example, #__Weaver Markup__# syntax is used for writing the         ///
    document which is easier than #__HTML__# or #__LaTeX__#, particularly for     ///
    students.
    
img price.png, width(250) title("The histogram of the #__price__# variable")

weave close​

[haghish]

Attaullah Shah

sure . I repeated the markdoc example above by exporting it to Microsoft Word docx and openoffice odt. see the attached do file.

[Attaullah Shah]

Thanks for your reply and quick action. One more question, is that possible to control the font size and font style of the exporting document. I find that the default for the body text is Cambria (Body) with font size of 12, and for Stata output the font style is Consolas with font size of 11. Can that be change, how? Your comments are always appreciated.