Limitations of Stata 17's New Table Collection System

[Mike Cassidy]

I've been exploring Stata 17's new collect suite of commands for publication-quality tables. It is cleverly designed in many ways and a long-overdue enhancement. But it seems to me there are some surprising limitations, particularly around what I'll call table "prefixes" and "suffixes" for lack of better terminology. For example, there doesn't seem to be a straightforward way to create table titles or notes. By contrast, Ben Jann's excellent estout command allows for customized headers and footers, which is very helpful when creating LaTeX output (e.g., if one wants to use threeparttable or make other formatting changes). In other words, collect seems really strong for the meat of the table, but may have forgotten the bread. Am I missing something?

In concept, it would probably be easy enough to write a wrapper command for collect to address these limitations, but then again it might not be straightforward because collect doesn't allow for appending of exported files (again, unlike estout).

Any thoughts would be appreciated, as it seems to me that these limitations might slow the adoption of a potentially very useful system.

[Leonardo Guizzetti]

Table titles and notes don't go with the table collection itself, which is just the summary data. They instead are added based on the output method. See for example -help putdocx table-.

[Mike Cassidy]

Thanks, putdocx does seem a good solution if the goal is a Word document. Does anyone have any suggestions for LaTeX output? Texdoc is perhaps the best option, but the implementation doesn't seem as clean as putdocx.

In general, it does seem like the ability to generate titles, notes, etc. from within the collection system itself would be a useful upgrade for the future (why not be able to do everything within one command?).

[Leonardo Guizzetti]

It's impossible to speculate why StataCorp made the design decisions they did. -putpdf-, -putdocx- and -putexcel- have their own options to export collections with titles. -collect export- to tex files seems to only allow just the table. Presumably, this design choice was made the give you, the programmer, the maximum flexibility over how you want the table to be presented.

[Mike Cassidy]

Ben Jann Jeff Pitblado (StataCorp) might you have any thoughts about this? (I ask the directed question given you have both previously posted helpful related content.) Thanks!

[Ben Jann]

Ahem ... didn't really look much into -collect- yet. LaTeX files are simple text files so you could just use -file- to add your own stuff. Adding stuff at the bottom is easy:

collect export table1.tex, tableonly replace
file open fh using table1.tex, write append
file write fh "my line 1" _n
file write fh "my line 2" _n
file close fh

Adding stuff at the top is a bit more complicated:

// backup table and open new file
copy table1.tex tmp.tex
file open fh using table1.tex, write replace
// write top lines
file write fh "my line 1" _n
file write fh "my line 2" _n
// now insert the table
file open fh2 using tmp.tex, read
file read fh2 line
while r(eof)==0 {
    file write fh `"`line'"' _n
    file read fh2 line
}
file close fh2
// write bottom lines
file write fh "my line 3" _n
file write fh "my line 4" _n
// clean up
file close fh
erase tmp.tex

ben

[Jeff Pitblado (StataCorp)]

If we were to add option append to collect export, it
could only work for HTML, LaTeX, SMCL, plain text, and Markdown.
We will consider adding such a feature.

collect supports titles and notes for PDF and MS Word documents
because that information must be passed to the put* commands that
collect uses to export tables to those file formats.

Originally we supposed that reports built from LaTeX would start with
LaTeX source files composed outside of Stata. We assumed that these files
would simply consume tabular environments exported by collect
using the LaTeX macro \input.

Alternatively, such a report could be built from Stata do-files. Like Ben's
examples above, I'll use the file command to generate the report, but
I'm still going to use LaTeX macro \input to place the exported tabular
environment in the report.

* open the file and compose the document preamble
file open fh using report.tex, write replace
file write fh "\documentclass{article}" _n
file write fh "\usepackage{multirow}" _n
file write fh "\usepackage{amsmath}" _n
file write fh "\usepackage{ulem}" _n
file write fh "\usepackage[table]{xcolor}" _n
file write fh "\begin{document}" _n

* introduction
file write fh _n "\section{Introduction}" _n
file write fh _n "Introduction paragraphs go here." _n

* analysis
file write fh _n "\section{Analysis}" _n
file write fh _n "Start discussing the analysis here." _n
file write fh _n

* export table and include it in our document
sysuse auto
table rep78 foreign
collect export tab1.tex, tableonly replace
file write fh "\begin{table}[ht]" _n
file write fh "\centering" _n
file write fh "\caption{My two-way table caption.}" _n
file write fh "\input{tab1.tex}" _n
file write fh "\end{table}" _n

* conculsion
file write fh _n "\section{Conclusion}" _n
file write fh _n "Concluding paragraphs go here." _n

* end the document and close the file
file write fh _n "\end{document}" _n
file close fh

I'm also attaching the resulting PDF, which I constructed by running
lualatex report.

[Hua Peng (StataCorp)]

An alternative to file write is to create a "source" tex file with narrative and Stata code to run. Then use dyntext to transform the source tex file to the final tex file with Stata code run and results weaved in. In this case, the result is the tex file produced by collect export which contains the table. The source tex file:

\documentclass{article}
\usepackage{multirow}
\usepackage{amsmath}
\usepackage{ulem}
\usepackage[table]{xcolor}
\begin{document}

\section{Introduction}
We use Stata's \textbf{dyntext} command with \texttt{dd\_do} tag 
to produce the table, then use \textbf{collect export} to save 
the table to a tex file. Then we may either use 
\texttt{dd\_include} tag or LaTeX's input to include 
the table in the document. 

\section{Analysis}
We first include a simple table.

% produce the table with dd_do, the section is in a LaTeX
% comment block since we only want the file it produced 
% when processed through dyntext but not the block itself
\iffalse
<<dd_do>>
sysuse auto
table rep78 foreign
collect export tab1.tex, tableonly replace
<</dd_do>>
\fi

% include the tab1.tex 
\begin{table}[ht]
\centering
\caption{My two-way table caption.}
\input{tab1.tex}
\end{table}

We may also include the table using \texttt{dd\_include} tag. It will 
include the content of tab1.tex in the transformed tex file, hence 
remove future dependency to tab1.tex.  

\begin{table}[ht]
\centering
\caption{My two-way table caption.}
<<dd_include: tab1.tex>>
\caption{Include table with \texttt{dd\_include}.}
\end{table}

\section{Conclusion}
Now we can use \textbf{dyntext} to transform this tex file (example.tex) to 
the desired tex file (\texttt{example\_out.tex}).

\end{document}

Then we can use:

dyntext example.tex, saving(example_out.tex) replace

to produce the example_out.tex which has the tab1.tex included in two ways, one using LaTeX's input as a link to the file, the other using <<dd_include>> with the actual content. The example_out.tex looks like:

\documentclass{article}
\usepackage{multirow}
\usepackage{amsmath}
\usepackage{ulem}
\usepackage[table]{xcolor}
\begin{document}

\section{Introduction}
We use Stata's \textbf{dyntext} command with \texttt{dd\_do} tag 
to produce the table, then use \textbf{collect export} to save 
the table to a tex file. Then we may either use 
\texttt{dd\_include} tag or LaTeX's input to include 
the table in the document. 

\section{Analysis}
We first include a simple table.

% produce the table with dd_do, the section is in a LaTeX
% comment block since we only want the file it produced 
% when processed through dyntext but not the block itself
\iffalse
. sysuse auto
(1978 automobile data)

. table rep78 foreign

------------------------------------------------
                   |          Car origin        
                   |  Domestic   Foreign   Total
-------------------+----------------------------
Repair record 1978 |                            
  1                |         2                 2
  2                |         8                 8
  3                |        27         3      30
  4                |         9         9      18
  5                |         2         9      11
  Total            |        48        21      69
------------------------------------------------

. collect export tab1.tex, tableonly replace
(collection Table exported to file tab1.tex)

\fi

% include the tab1.tex 
\begin{table}[ht]
\centering
\caption{My two-way table caption.}
\input{tab1.tex}
\end{table}

We may also include the table using \texttt{dd\_include} tag. It will 
include the content of tab1.tex in the transformed tex file, hence 
remove future dependency to tab1.tex.  

\begin{table}[ht]
\centering
\caption{My two-way table caption.}
\begin{tabular}{llll}
\cline{1-4}
\multicolumn{1}{c}{} &
  \multicolumn{3}{|c}{Car origin} \\
\multicolumn{1}{c}{} &
  \multicolumn{1}{|r}{Domestic} &
  \multicolumn{1}{r}{Foreign} &
  \multicolumn{1}{r}{Total} \\
\cline{1-4}
\multicolumn{1}{l}{Repair record 1978} &
  \multicolumn{1}{|r}{} &
  \multicolumn{1}{r}{} &
  \multicolumn{1}{r}{} \\
\multicolumn{1}{l}{\hspace{1em}1} &
  \multicolumn{1}{|r}{2} &
  \multicolumn{1}{r}{} &
  \multicolumn{1}{r}{2} \\
\multicolumn{1}{l}{\hspace{1em}2} &
  \multicolumn{1}{|r}{8} &
  \multicolumn{1}{r}{} &
  \multicolumn{1}{r}{8} \\
\multicolumn{1}{l}{\hspace{1em}3} &
  \multicolumn{1}{|r}{27} &
  \multicolumn{1}{r}{3} &
  \multicolumn{1}{r}{30} \\
\multicolumn{1}{l}{\hspace{1em}4} &
  \multicolumn{1}{|r}{9} &
  \multicolumn{1}{r}{9} &
  \multicolumn{1}{r}{18} \\
\multicolumn{1}{l}{\hspace{1em}5} &
  \multicolumn{1}{|r}{2} &
  \multicolumn{1}{r}{9} &
  \multicolumn{1}{r}{11} \\
\multicolumn{1}{l}{\hspace{1em}Total} &
  \multicolumn{1}{|r}{48} &
  \multicolumn{1}{r}{21} &
  \multicolumn{1}{r}{69} \\
\cline{1-4}
\end{tabular}
\caption{Include table with \texttt{dd\_include}.}
\end{table}

\section{Conclusion}
Now we can use \textbf{dyntext} to transform this tex file (example.tex) to 
the desired tex file (\texttt{example\_out.tex}).

\end{document}

I attach the pdf file generated by example_out.tex in overleaf. StataTable.pdf

[Mike Cassidy]

Thanks very much, Ben, Jeff, and Hua. I appreciate you taking the time to think about this issue and offer creative solutions. Building off Ben's code, I wrote a simple command, collectex, which loosely follows the syntax of collect export but allows options to include a prefix and suffix. I'll post it here in case anyone finds it useful, noting that it is very much a rough draft at this point. Feel free to modify/extend!

program define collectex , eclass sortpreserve  
              version 17

********************************************************************************        
*1) command syntax        
********************************************************************************
*get user input        
syntax [anything(name=mytable)] [using]  , PREfix(string asis) SUFfix(string asis) [replace append]
********************************************************************************


********************************************************************************
*2) take raw exported tex table and and prefix and suffix
********************************************************************************
*benn jann code modified

*original raw table
collect export `"`mytable'"', tableonly replace

*top (prefix)
// backup table and open new file
tempname fh // main file handle
tempname fhtemp // temporary file handle
tempfile mytempfile // temporary file

*make a copy of raw table 
copy `"`mytable'"' `mytempfile'
file open `fh' using `"`mytable'"', write replace // open file

// write top lines
local prefix `"`macval(prefix)'"'
if `"`macval(prefix)'"'!="" {
     file write `fh' `macval(prefix)' _n
}

// now insert the table
file open `fhtemp' using `"`mytempfile'"', read
file read `fhtemp' line
while r(eof)==0 {
    file write `fh' `"`line'"' _n
    file read `fhtemp' line
}
file close `fhtemp'

// write bottom lines
local suffix `"`macval(suffix)'"'
if `"`macval(suffix)'"'!="" {
     file write `fh' `macval(suffix)' _n
}

// clean up
file close `fh'

********************************************************************************

end

And here is a simple example (a shorter version of Jeff's example):

sysuse auto
table rep78 foreign
collectex table1.tex , prefix("\documentclass{article}" "\usepackage{multirow}" "\usepackage{amsmath}"  ///
                              "\usepackage{ulem}" "\usepackage[table]{xcolor}" "\begin{document}" ///
                              "\section{Introduction}" "Introduction paragraphs go here." ///
                              "\begin{table}[ht]" "\centering" "\caption{My two-way table caption.}") ///
                      suffix("\end{table}" "\end{document}")