New package -inshell- available on the SSC

[Matthew Hall]

With thanks to Kit Baum my package inshell has been updated on the SSC. Though this package was originally released in December 2021, I wanted to wait until I made some improvements before I announced it here on this forum.

This package is a wrapper of shell that will capture the standard output, standard error, and native shell return codes of commands sent to the operating system's shell through Stata. It works on macOS, Linux, and Microsoft Windows, although it has not been thoroughly tested on the latter.

It is an improvement over other shell wrappers in several ways. For instance, ashell, with all due respect to its author Nikos Askitas and which seems to be the predominant package of its kind, does not properly handle the commands it sends to the shell. It has been in use since 2007 and was last updated in 2009 and I am surprised that no one has noticed this limitation. Because shell code (with the exception of Windows Command shell) uses the same dollar sign alphanumeric notation for its variables that Stata uses for its own global macros, commands sent via a shell wrapper must prevent macro substitution, which inshell does at every level with prodigious use of the macval(lclname) expansion operator.

For an example of this, try (on macOS and Linux systems):

inshell will also capture and display the standard error and the shell's native return code (not to be confused with c(rc) or _rc, Stata's internal return code) if the commands send to it produce an error. It displays these error details using a simple "theme" like so:

The last command and its resulting error in the above screenshot is more than just a silly demonstration; it not only shows that inshell can handle Unicode (UTF-8) strings (with proper display column sizing) but that it can even correctly parse expressions that Stata would otherwise interpret as its own local macros. In testing this package I somewhat randomly discovered that the cal command-line utility outputs its error with this unusual quoting that mirrors Stata's local macro structure, and though this specific case is unlikely to be relevant to any of inshell's users, my discovery of it led to a desire to make inshell that much more robust.

Users of shell in all its forms are encouraged to read the help file, as it is a good primer on the advanced usage of the shell through Stata.

The package also includes (of all things) a wrapper of cd. I added this subprogram primarily to allow users of inshell to cd to directories stored as shell variables. As shell cannot cd to any directories and cd will not reference variables within the shell's environment, this allows the user to use inshell as a more complete solution in their code.

It has also been tested with Microsoft PowerShell and the help file contains information regarding its use:

Because the output of shell commands can often contain ANSI escape sequences, which are meant to be interpreted as display directives and can garble the results that are captured and printed, it also contains a subprogram that will remove these if it detects them. This is important because these control sequences are often invisible and can cause headaches when passed to other commands. For example, if the above PowerShell command is sent through shell, the results might look something like this:

and if the user enters such a command using the GUI version of Stata and selects the text in the Results window they may witness the text visibly shift. As stated above these aberrations can become more than just visual when they are captured into macros and subsequently parsed.

Hopefully some of you will find this program useful. Please don't hesitate to contact me here on this forum or via email with comments or suggestions.

Best,
Matthew Hall

[Matthew Hall]

Though I just released a major revision less than two weeks ago, I have already released an upgrade to my shell wrapper inshell. I have put an enormous amount of testing and experimentation into this (at least on my Intel Mac running macOS 10.15.7 with StataSE 17 and zsh 5.9) in order to figure out all of the limitations and eccentricities of accessing the shell through Stata, and you the user can become the beneficiary of all of this hard work by downloading it from the SSC.

Not only does it have much more robust detection and removal of ANSI escape sequences but it also handles strings with Stata macro-defining characters more reliably than previous versions. I jokingly quipped in the original help file that some of the limitations of Stata macros would only pose a problem to users attempting to inshell ASCII art, but this is exactly what I ended up doing in order to thoroughly test and expand upon its capabilities.

Using the program neofetch (http://github.com/dylanaraps/neofetch), which is a command-line system information tool that not only displays an ASCII art logo for your operating system, but also uses ANSI escape sequences in order to create a highly formatted infographic which is intended to be used in screenshots of your system. Below is an example of what it looks like in a terminal emulator:

Neofetch allows one to alternatively display an ASCII art logo from one of 178 other operating systems, so I wrote a loop to cycle through and test each one of these with inshell. Many contain backticks (`), single (') and double quotes ("), as well as dollar signs ($) and other special characters that prompt Stata to try to interpret the contents of the local macro into which they are captured as one of its internal structures. The version of inshell (1.5) that I released two weeks ago was only successful at capturing about 25% of these; the current version (1.6) can now properly handle 174 out of 178 (98%), one of which is presented below.

I also used the phoon utility (http://www.acme.com/software/phoon/) which shows the current phase of the moon using its own ASCII art but also allows you to specify any date using a (positive) Unix epoch. I wrote a loop to cycle through each and every 12 hour interval starting on January 1st 1970, and the current version is successful with 100% of these, an example of which is presented below.

(Note: In the above screenshots, the global ${ulb} is equal to /usr/local/bin/, the path where these utilities are installed on my machine, which is necessary to specify as the Stata GUI application does not source the shell configuration files necessary to extend the $PATH to include /usr/local/bin, at least on my system. See the help file for more information on this.)

While these specific utilities may not be the most relevant use cases of inshell, the results above show that its code is robust enough to handle almost any strings it may encounter.

Although I have no prior experience programming in it, I've realized that the only way to achieve 100% accuracy with the capture of lines containing special characters is to recode some portions in Mata. I have already started working on this with great results and will hopefully have an upgraded version to release in the next couple of weeks.

As always, don't hesitate to email me, DM me on this forum, or reply to this post with any questions, comments, or suggestions. Hopefully some of you will find this program useful.

[Attaullah Shah]

On windows, the I get the following error, even with the help command

. inshell help
unknown function now()

[Matthew Hall]

On windows, the I get the following error, even with the help command

. inshell help
unknown function now()

Attaullah-- Which version of Stata are you using? It seems like the now()datetime function is new to Stata 17. It's not critical that I use it, so I'll find something to replace it. Thanks for bringing this to my attention.

[Matthew Hall]

The issue that Attaullah Shah brought to my attention has been remediated and the fix is up at the SSC. inshell is now at version 1.7 and is compatible with Stata versions 14 and greater.

As mentioned above, I have begun to recode parts of this in Mata, which I am in the process of learning from the beginning. Despite my inexperience, I already have a working prototype that replaces both of inshell's two subprograms (the "deansification" routine and the cd wrapper) with Mata functions, and I hope to release the whole package soon. The rationale behind recoding in Mata is that in rare cases Stata cannot handle macros that contain certain permutations of special characters. Working in Mata allows less opportunity for macro expansion when strings are passed from one function to another.

As mentioned in (#2) one of the more interesting (and highly effective) methods that I've employed to test inshell involves using command-line utilities that display ASCII art; one that produces renditions of operating system logos (with heavy ANSI formatting) called neofetch and another that displays phases of the moon, called phoon.

The version of inshell (1.5) that I released two weeks ago was only successful at capturing about 25% of these; the current version (1.6) can now properly handle 174 out of 178 (98%), ...

The remaining 4 are the logos for the operating systems Windows7, Debian, Korora, and Refracted Devuan, the latter three of which are Linux distributions.

Comparing the current version of inshell with the upcoming one, and with the output and return list presented side-by-side:

inshell (version 1.7):

inshell (version 2.0, in beta testing):

Lines 6, 16, and 17 are extremely difficult to handle within macros due to unbalanced quoting and the presence of macro-defining characters. As you can see the current version simply replaces the line with "THIS LINE WAS NOT CAPTURED", which it does only after trying three methods of processing the macro, rather than aborting the whole operation.

The beta of inshell 2.0 however has no problem processing these lines on the first try, and I have not yet encountered a line with which it has any difficulty.

The new version will also return its macros for captured standard output in the reverse order, which is to say that they will appear in the return list in the same order in which they are displayed in the standard output for easier readability.

Beyond all of this I have an additional (useful) feature or two that I will be implementing, so stay tuned.

[Matthew Hall]

Version 2.0 (along with a small but important update to 2.1) of inshell is complete and was released on the SSC a couple weeks ago. This is a major revision that includes several new and important features.

inshell now processes output from the shell (to remove ANSI escape sequences and other invisible characters) in Mata, which means that any information processed and captured by it is now impervious to Stata's macro expansion routines (see posts above). And although it doesn't really benefit from the conversion, the cd wrapper has also been transitioned to Mata as well.

One important and little-used feature of shell in Stata that now functions with inshell is the use of the global macro S_SHELL (see page 793 of the complete PDF manual). It is used to specify an alternative shell when using shell and can be quite useful in certain situations.

inshell also now allows several useful options which can be set via a series of global macros. Because GUI instances of Stata often do not source the necessary configuration files to extend the user's PATH, inshell offers the option to add a single path to the beginning of the user's PATH upon every invocation of inshell. This is particularly useful because even though the user may provide an absolute path to a desired executable said executable may need to locate other commands which will otherwise not be found without an extension to the user's PATH. Consider the following example:

inshell now also includes a "diagnostics" program (accessed via inshell diagnostics), which displays information about the various inshell options as well as details about the shell you are using through Stata:

It is important to note that although inshell can capture and store strings containing macro-defining characters and/or unbalanced quoting (via its Mata-coded file processing routine) within returned r() macros, these difficult-to-handle macros cannot be subsequently parsed or even displayed using ado Stata code and will require Mata code in order to work with them. For example, if a user of inshell were to execute the command from the previous post (inshell neofetch --ascii_distro Debian -L) and view the list of returned macros, the user can confirm that they exist exactly as they do in inshell's displayed output, however if one were to display lines 6, 16, or 17, they will be unable to properly do so. In this specific case, using display ""`r(no6)'"" will not display the line but " ' . $$$" instead.

Creating a new local and then attempting to display it using `macval(lclname)' will also fail, as well as various permutations of the two using unbalanced quoting, i.e. display ""`macval(test)'". Similarly, they will be unable to use functions such as strpos() for parsing. The following are some solutions using Mata that will allow the user to effectively parse the content of these strings no matter what they contain:

As always, please don't hesitate to contact me with comments or suggestions. Hopefully some of you will find this program useful.