PSA: Replacing string values does not entirely remove the old values from the saved .dta file
I want to alert people who de-identify, anonymize, or share data to a behavior of Stata's .dta writer. I reported it to StataCorp Technical Support on 7 September. Details of their reply are at the end.
In short: replace name = "" does not remove names from a saved dataset. Neither does replacing names with shorter pseudonyms. The old values stay in the file, minus their first byte. Stata does not show them, but anyone can read them with a text editor or grep.
1. String data
When a str# value is replaced by a shorter one, Stata writes the new value and a null terminator, and leaves the rest of the fixed-width field as it was. The rest of the old value is then saved to disk. Here is a reproduction on StataNow 19.5:
Stata shows id as 1, 2 and 3. On disk, the three 30-byte fields are
Blanking the names with replace id = "" gives the same result. So does -saveold- to formats 117 and 115. -help dta- documents this at the level of the file format: a short string is followed by "random bytes". But those bytes are not random. They are the value you just overwrote.
Dropping the variable removes its contents from that save. However, freed string memory gets reused. Any string variable you create later in the same session, including one in a new dataset after -clear-, can pick up the dropped values and write them into the next file you save.
I scanned a few hundred published datasets from public research archives. About a quarter of those with a str# variable wider than some of its values have readable text in the padding that appears in no visible value in the file. The text includes addresses, place names and free-text responses. In one file, the padding held the full names of several individuals who appear nowhere in the dataset's visible values. The files most at risk are those with wide string variables that hold short values.
2. Metadata
The fixed-width metadata slots (variable names, formats, value label names, variable labels, sort list) are also written without being cleared first. The padding contains whatever memory Stata last used for them. That can include names and labels from a different dataset saved earlier in the same session. Using -clear-, -clear all- or -label drop _all- in between does not help. The padding also survives a load and re-save in a fresh session. About half of the published files I scanned have fragments like this. You can see it in the auto dataset Stata ships with: the slot for -turn- contains "turn\0ng_circle".
StataCorp's response
Technical Support told me that StataCorp does not consider this a bug, but they said it is prompting internal discussion. One option they are considering is zeroing the unused space in strings on -save-, either automatically or through an option, because doing it on every save could slow Stata down.
In my view, the current behavior is a disclosure risk for personally identifiable information (PII).
In short: replace name = "" does not remove names from a saved dataset. Neither does replacing names with shorter pseudonyms. The old values stay in the file, minus their first byte. Stata does not show them, but anyone can read them with a text editor or grep.
1. String data
When a str# value is replaced by a shorter one, Stata writes the new value and a null terminator, and leaves the rest of the fixed-width field as it was. The rest of the old value is then saved to disk. Here is a reproduction on StataNow 19.5:
Code:
version 19 clear input str30 id "Maria Fernanda Oliveira" "João Carlos Pereira" "Ana Beatriz Santos" end replace id = string(_n) save pseudonymised.dta, replace
Code:
1\0ria Fernanda Oliveira\0\0\0\0\0\0\0 2\0ão Carlos Pereira\0\0\0\0\0\0\0\0\0\0 3\0a Beatriz Santos\0\0\0\0\0\0\0\0\0\0\0\0
Dropping the variable removes its contents from that save. However, freed string memory gets reused. Any string variable you create later in the same session, including one in a new dataset after -clear-, can pick up the dropped values and write them into the next file you save.
I scanned a few hundred published datasets from public research archives. About a quarter of those with a str# variable wider than some of its values have readable text in the padding that appears in no visible value in the file. The text includes addresses, place names and free-text responses. In one file, the padding held the full names of several individuals who appear nowhere in the dataset's visible values. The files most at risk are those with wide string variables that hold short values.
2. Metadata
The fixed-width metadata slots (variable names, formats, value label names, variable labels, sort list) are also written without being cleared first. The padding contains whatever memory Stata last used for them. That can include names and labels from a different dataset saved earlier in the same session. Using -clear-, -clear all- or -label drop _all- in between does not help. The padding also survives a load and re-save in a fresh session. About half of the published files I scanned have fragments like this. You can see it in the auto dataset Stata ships with: the slot for -turn- contains "turn\0ng_circle".
Code:
mata: fh = fopen(findfile("auto.dta"), "r"); s = fread(fh, 100000); fclose(fh); strpos(s, "ng_circle")
Technical Support told me that StataCorp does not consider this a bug, but they said it is prompting internal discussion. One option they are considering is zeroing the unused space in strings on -save-, either automatically or through an option, because doing it on every save could slow Stata down.
In my view, the current behavior is a disclosure risk for personally identifiable information (PII).
Comment