Inconsistent behavior of view vs. type commands

[Sergiy Radyakin]

It seems that different Stata commands treat the last line of text files differently depending on whether there is a line terminator or not at the end of it.

Is there a guide which would specify how the last line is treated (by command)? Or is it just a bug?

Thank you, Sergiy

[William Lisowski]

Sergiy, those of us intrigued by your question would appreciate it if you would provide the URL of your test file in text that can be copied and pasted, for our experimentation.

[Sergiy Radyakin]

Hello William,

thank you for looking into the issue. The URL for the example file is: http://www.radyakin.org/statalist/st...s/missline.txt .
The exact commands that I typed are:

view http://www.radyakin.org/statalist/statabugs/missline.txt
type http://www.radyakin.org/statalist/statabugs/missline.txt

[William Lisowski]

I agree with you that this difference in the handling of subtly odd files is apparently undocumented and the behavior of view is misleading, which is what I consider the real problem. If there's a reason for view to not display the final line, it seems to me view should report that there is content between the final line end and the end of the file.

I notice that, gratifyingly, import delimited deals correctly with the data. I did my work on Stata for Mac and the results are identical to yours. I seem to have lost the knack of creating a text file with no line delimiter at the end of the last line (my editor of choice adds it if it's not found) so it's not a 100% replication on a Mac, but still.

I think asking Stata Technical Services about this difference is warranted.

[Sergiy Radyakin]

Thank you William, I find this inconsistency in other commands, and sometimes more essential. For example in the do-files, if the last line is not having the terminator it is not executed if the file is run from the network, but if the file is downloaded, and run from the do-editor, it gets executed.

. do http://www.radyakin.org/statalist/statabugs/missline.do

. display 1
1

. display 2
2

If one downloads the same file, opens it in the do-editor and does it (Ctrl+D), 3 lines will appear! (simply do-ing the downloaded file would still produce only 2 lines).

Furthermore, the downloaded file will be modified without user's permission by the do-editor by inserting the line terminator and silently re-saving the file. (File size changes from 31 bytes to 33 bytes for my example case). This is already sufficient to trigger alert on the VCS.

Funny, if I protect the file with RO-attribute, Stata asks me some confirmation:

Answering Yes is equally scary as answering No here.

I will bring this to the attention of the developers, if there is no explanation for it, as it seems.

Best, Sergiy Radyakin

[William Lisowski]

That is an awesome dialog box. I know Stata is sometimes terse in its messages, but this is a new record in brevity. Maybe a

I copied missline.do to my disk and opened it in the do-file editor. I did nothing to it, and the do-file editor ran the saved version that I had opened, and only lines 1 and 2 were run, exactly as if I had issued the command do missline.do in the command window. I edited it, and Stata ran the edited version from a temporary file, and all three lines were run.

[Clyde Schechter]

FWIW, I recall that many, many versions back it was documented behavior of the do-file editor that the last line of the file would not be executed if it did not contain a line terminator. The manuals warned people to routinely end their do-files with the -exit- command, since Stata's behavior would be the same regardless of whether the -exit- command was executed or not. There is no such documentation currently that I can find, and at least on my set up (Windows 7, Stata 15.1 MP2), the final line of a do-file is always executed when run from the do-editor, line terminator or no. But my instinct is that some of what we are seeing here is the ghost of previously documented behavior that has yet been completely interred. Just a hunch.

[Chinh Nguyen (StataCorp)]

I can explain what's going on with the Do-file Editor but I'll have to get back to you on why the end-of-line delimiter is necessary.

When saving a file, the Do-file Editor always appends an end-of-line delimiter if one does not exist at the end of the do-file. This is to avoid the problem with the lack of an end-of-line delimiter on the last line.

If you execute a do-file from the Do-file Editor and the do-file does not already exist on disk, the Do-file Editor appends an end-of-line delimiter if necessary, writes the do-file to a temporary file, and then executes the temporary file.

If the do-file already exists on disks, does not contain any unsaved changes, and contains an end-of-line delimiter at the end of the file, then Stata simply executes the do-file on disk.

If the do-file already exists on disks, contains unsaved changes or does not contain an end-of-line delimiter at the end of the file, the Do-file Editor will append an end-of-line delimiter if necessary then save the do-file to disk if the Do-file Editor preference Always save before do/run is enabled and execute the do-file on disk. If the preference is not enabled, then the do-file is written to a temporary file and Stata executes the temporary file.

Furthermore, the downloaded file will be modified without user's permission by the do-editor by inserting the line terminator and silently re-saving the file.

I think the reason the Do-file Editor attempted to save your read-only file is because you may have the preference Always save before do/run enabled and the do-file did not contain an end-of-line delimiter at the end of the file. I'll look into what Stata should do instead for read-only files but I will note that I did not get the empty confirmation dialog. I got a warning that access to the file was denied. I'm using Windows 10 and it appears you're using an earlier version of Windows but I don't know if that explains the difference in behavior.

I copied missline.do to my disk and opened it in the do-file editor. I did nothing to it, and the do-file editor ran the saved version that I had opened, and only lines 1 and 2 were run, exactly as if I had issued the command do missline.do in the command window. I edited it, and Stata ran the edited version from a temporary file, and all three lines were run.

I downloaded missline.do to my Desktop, opened it in the Do-file Editor, immediately pressed ctrl-d and all three lines were run. I did this in both Stata 14 and Stata 15. I restored missline.do before each attempt since Stata wants to resave the file because of the missing end-of-line delimiter.

[Sergiy Radyakin]

Dear Chinh,

thank you for clarifying that the behavior of the do-editor is controlled by an option. I think the option in question though is not just the "Always save before do/run" which is unchecked in my settings, but also the "Normalize" flag for the endings, which is set in my settings and, imho, is the default option.

Contrary to

If the preference is not enabled, then the do-file is written to a temporary file and Stata executes the temporary file.

it appears that Stata (retested with each Stata 14 and 15) disregards these options for the case of last line. Regardless of the choice of "Normalize" it adds a line end (digit 4 appears on the gutter) and attempts to overwrite the original file regardless of the "Always save before do/run" setting.

The empty dialog for a read-only file is reproducible in Stata 15 and Windows 10 environment.

Exact repro-steps are:

1. download the file from http://www.radyakin.org/statalist/statabugs/missline.do to your drive (I used my desktop folder, shouldn't matter);
2. set file read-only attribute;
3. double-click the file (opens in do-editor with default settings);
4. without any edits, simply press Ctrl+D to do the file.

If no read-only attribute is set, I do see three lines in the output due to the do-editor fix. But this may be platform-dependent. Looks like on Windows the setting is somehow disregarded.
William Lisowski could you please write your platform description?

This is to avoid the problem with the lack of an end-of-line delimiter on the last line.

Imho, rather than fixing the source files, it should be the behavior of the reader to tolerate absence of the CR/LF for the line and impute them from the file end. Kind of what this code (prototype, I didn't bother about macro expansion with macval() which is not the focus here) is doing:

file open fh using "http://www.radyakin.org/statalist/statabugs/missline.do", read text

file read fh oneline
while r(eof)==0 {  
  display `"`oneline'"'
  file read fh oneline
}
file close fh

Output (properly displays all lines):

. file open fh using "http://www.radyakin.org/statalist/statabugs/missline.do", read text

.
. file read fh oneline

. while r(eof)==0 {  
  2.   display `"`oneline'"'
  3.   file read fh oneline
  4. }
display 1
display 2
display 3

. file close fh

1. If all the readers replicate this behavior (do, ado, view, type, import delimited, insheet, etc..) there will be less or no need for the source modification.
2. It is rather confusing that with default settings, if I am telling the user "do this file -> link" I can't be sure what result she will end up with since I can't be sure she will actually type in the command line "do weblink" rather than download and do. Most of the users don't feel the difference, and don't feel the need to elaborate what they actually did when we are looking for a discrepancy. For the case of auto-generated do files, I have to take extra caution each time to include some extra empty lines in the end or a comment line to make sure that regardless HOW the file is executed the result is the same.
3. For do-files there is this workaround of including always the comment, such as "// end of file", but for data files, I always worry whether the user will end up without the last line of data, or with an extra line of missings, or have the output "just right".

Thank you, Sergiy Radyakin

[Stata 14.0 + Win7]
[Stata 15.0 + Win10]

[William Lisowski]

Sergiy Radyakin -

I run Stata 15.1 on macOS 10.13.3 High Sierra, detailed below.

I downloaded missline.do and set it to be read-only by changing my permissions in the file's Get Info window to Read only.

I then double-click on missline.do, which launches Stata and opens missline.do in the Do-file Editor window. I run three commands from the Stata Command window and then click the "Do" button in the Do-file Editor window. I regret that I do not get the terse dialog box. I do only get two of the three lines run, as described in post #6. That is true whether or not the file has been set to read-only.

ADDED IN EDIT: If I modify the file in the Do-file Editor and then run it, it will be run from a temporary file. If I try to save it, a dialog box will tell me the file is locked and give me the option to save it anyway. There is no Do-file Editor option to always save before running that I have been able to find.

. shell sw_vers -productVersion

10.13.3

. about

Stata/SE 15.1 for Mac (64-bit Intel)
Revision 11 Jan 2018
[redacted]
                      

. ls missline.do

-r--r--r--@ 1 lisowskiw  staff  31 Feb  7 14:47 missline.do

. do "/Users/lisowskiw/Downloads/missline.do"

. display 1
1

. display 2
2

.
end of do-file

.

[Chinh Nguyen (StataCorp)]

I run Stata 15.1 on macOS 10.13.3 High Sierra, detailed below.

Sorry, I was referring to Windows. Stata for Mac's behavior is slightly different than Stata for Windows. Stata for Mac only appends an end-of-line delimiter when saving a file either to disk or as a temporary file. It does not check for and append an end-of-line delimiter before executing the file if a file is unmodified. I'll be looking into that but will probably make it consistent with the Windows behavior.

I regret that I do not get the terse dialog box.

...

If I try to save it, a dialog box will tell me the file is locked and give me the option to save it anyway.

In Windows, the Do-file Editor is warning you that the file is read only and asks if you want to make it writable. You'd see the same dialog if you opened a read-only file and started typing. Unfortunately, we forgot to put the text in the confirmation dialog stating what's going on. I'll be looking into what's the most Windows-consistent way of editing a read only file and make that change in a future update so you may or may not see that dialog (with text) in the future.

On the Mac, the file is locked dialog is not coming from Stata directly but from the Mac so we get that nice feature for free. Prompting to make the file writable when saving rather than when editing might be the approach we'll make for Stata for Windows.

There is no Do-file Editor option to always save before running that I have been able to find.

On Stata for Mac, it's in the Do-file Editor's Advanced tab.

[Chinh Nguyen (StataCorp)]

In a future update to Stata 15, here's what we're going to do:

1. On all platforms, we'll fix the issue with the last line of a file not being executed if it doesn't contain an end-of-line delimiter.

2. On all platforms, we'll fix the issue with the last line of a file not being displayed in using view if it doesn't contain an end-of-line delimiter.

3. On Windows, the Do-file Editor will no longer append an end-of-line delimiter to the document in the editor if the document's last line does not contain an end-of-line delimiter. This is consistent with Stata for Mac behavior.

4. On Windows, the Do-file Editor will append an end-of-line delimiter to the file that's saved to disk if that last line does not contain an end-of-line delimiter.This is consistent with Stata for Mac behavior and maintains backwards compatibility with older versions of Stata that still require the last line to contain an end-of-line delimiter.

5. On Windows, you will no longer see an empty dialog appear if you try to edit a read-only file in the Do-file Editor (the dialog is going away). Stata for Windows will be consistent with other Windows applications where attempting to save a read-only file from the Do-file Editor will prompt you for a different filename. If you try to overwrite the existing read-only file, you'll get an access denied error dialog.

6. On all platforms, the Do-file Editor will execute a read-only file that has been modified in the editor by writing out the file to a temporary file and executing the temporary file.

If you find any other instances where Stata is ignoring a line because it does not contain an end-of-line delimiter, please let us know.

[Daniel Bela]

Hi!

Just for the record, Stata 15.1 for Unix, running on Ubuntu Linux 17.10, behaves in the same way that William Lisowski describes for the Mac. Ubuntu features, however, no such automated mechanism of notifying the user that he tries to write to a non-writable file.

I also recall a hint -- somewhere in the PDF manuals, I guess -- to always include a blank line at the end of each do- or ado-file. However, I can't find the very spot by now.

Looking forward to the fix, this caused headache several times.

Regards
Bela

[Chinh Nguyen (StataCorp)]

Ubuntu features, however, no such automated mechanism of notifying the user that he tries to write to a non-writable file.

Although not completely obvious by the meaning, I was presented with an error dialog that said "The file you are trying to save can not be created." when attempting to overwrite a read-only file in Stata for Unix. We can probably improve the text in the dialog as I'm pretty sure it's just a generic error message from not being able to save a file.

[Sergiy Radyakin]

All of the changes listed in post #12 by Chinh Nguyen (StataCorp) are in line with my expectations, and hopefully with others' too. Thank you for looking into this!

I am aware of at least one more command where the end of line matters: infix. A trivial example (executed here in Stata 14) would be something like:

. clear

. type  http://www.radyakin.org/statalist/statabugs/missline.dct
infix dictionary {
 byte x 1-1
}
1
2
3

. infix using http://www.radyakin.org/statalist/statabugs/missline.dct
infix dictionary {
 byte x 1-1
}
(2 observations read)

This can occur when a fixed format data file is prepended with a data dictionary (prepared in advance) on the server to obtain a single readable/downloadable file. The expectation is that 3 observations will be read, since there is valid data on the last line of the file, albeit without a terminator. The use of the fixed format seems to be declining. If anyone encountered this situation, and you don't have control over the process that generates the data portion, just prepare a stub file with a line terminator in it, then concatenate three files (dictionary+data+terminator).

If the same file is downloaded, opened and saved in the do-editor, then infixed, there will be 3 observations read.

The same problem occurs if the dictionary and data are saved in two different files.

If the dictionary file itself is missing the line terminator on the last line, Stata considers it an invalid dictionary:

. clear

. infix using testinfix.dct
infix dictionary using "fixeddata.txt" {
 byte x 1-1     
dictionary invalid
r(614);

Interestingly in Windows and Stata 14 there seems to be no problem with the following files even if there is no line terminator on the last line:

• stata.toc (.toc)
• package (.pkg)
• text (for insheet) (.txt).
• dialog files

Thank you, Sergiy