What is the structure of the header of a .dta file? Attempting to combine binary datafiles without loading into memory

[Kasper Kotisaari]

Hi everyone!
I would like to ask if anyone knows what is the structure of a .dta datafile's binary header?

My problem is that I would like to save a datafile so big, that I don't have enough memory to save it.
Now, if I split the datafile into two sections, I have enough memory to perform operations and save those sections.
But then again, I will run out of memory if I append the second section to the first and attempt to save the whole datafile.

In order to solve this, it occured to me, that I could remove the binary header of the second section.
Then I could just tell bash or command prompt to put the two files together, as in:

 cat first_section.dta second_section.dta > whole_datafile.dta

This way the whole file would not have to be loaded into memory.
But to achieve this, I would need to know where the header ends.
Let's assume both files have matching number of variables, variable names and variable types.
I wonder will the observation numbers cause a conflict and can I make the second file to start observations' numbers where the first datafile ends?
After looking at a .dta file with a text editor, I'm assuming the data part starts at a point where I see the tag <data> .
Finally, since the structure looks a bit like html, I'm also thinking should I remove the closing </> parts from the first section file, so that the resulting file would have its header start at first_section.dta and end at second_section.dta, if that makes any sense?

I welcome any ideas or better approaches!

Best regards,
Kasper

[Nick Cox]

search dta

leads to the manual entry

[P] File formats .dta -- Description of .dta file format

or find that within

https://www.stata.com/manuals/p.pdf

It's hard for me to imagine that your approach will work easily even with your benign assumptions. Whether you can make progress by

using efficient data types

combining ASCII versions of your data

dropping variables you don't need

is impossible to say.

Find a computer with more memory is a predictable but presumably unhelpful suggestion.

[Felix Bittmann]

You can try to generate a few example files and check the content using a Hexeditor (https://hexed.it/). This might help you identify patterns. I am not sure if your idea is entirely feasible as the .dta surely saves a lot more metadata. If you just append the raw files with the header of one removed, I wonder whether the metadata of the other file will be corrupted. For example, the total number of cases changes. If this value is saved in another place in the file and not updated after the append, the file is probably corrupted.

[Daniel Schaefer]

Neat question. What you are trying to do here is absolutely cursed. You might want to try the compress command first.

Have you tried opening a .dta file as plain text yet? It's uncompressed xml with some compressed or possibly raw base64 binary for values. Here is a header of a test .dta file I made in Stata 18:

clear
set obs 100
gen x = runiform()
gen y = runiform()
gen z = runiform()
save test.dta

<stata_dta><header><release>118</release><byteorder>LSF</byteorder><K> </K><N>d       </N><label>  </label><timestamp>15 Jan 2026 14:53</timestamp></header>

I think Felix and Nick are completely right and trying to delete the header than merge will give you a corrupt file, but why not give it a try anyway?

Edit: Okay, yes I see you've already tried opening it in a text editor. I think the issue is that you're probably to have to be much smarter about how you combine the binary data within the <data> tags. Assuming the <map> is the same across files...

[Daniel Schaefer]

Cat wont make you load the entire file into memory, but the thing is, Stata's -append- is more than likely implemented as some kind of bit stream with a fixed-size buffer too, so the memory requirements are probably similar to what you propose with cat. The xml you are parsing out must account for a small amount of the total memory. Assuming you get this working, I wonder if you run into the same memory issue when you try to load the combined data anyway.

[Kasper Kotisaari]

Thank you for your many good points! I realize now that maybe this will be too risky for example because of what Felix Bittmann pointed out with the number of total cases becoming corrupted, and since I will also not be able to verify the end result. But I will report back here if I decide to test this. Maybe not many usecases, except if someone with a small-memory-computer produces very large datafiles for someone who has access to bigger RAM memory, or possibly some efficiency savings but with a risk of data corruption. Maybe if this was somehow made as a verified tool it could be usable.

[Kasper Kotisaari]

After first test, I see that at least simply copying the inside of <data></data> to another datafile's similar section (assuming both datafiles have matching shape, variable names) will mess up the observation count and cause the resulting file to not open. Supposedly the observation count is also in the header, and so the total resulting count would need to be known and inputed there with the correct (binary?) format.

[Nick Cox]

It's salutary that Stata's own commands for combining datasets all hinge on a specific kind of combination and feature checks that the combination can be done unambigously with the information you supply.

I am left with pointers to

using efficient data types

combining ASCII versions of your data

dropping variables you don't need

finding a computer with more memory

[Maarten Buis]

If what Nick suggests does not work, then you can ask the people at StataCorp: https://www.stata.com/support/tech-support/contact/ . My guess would be that they will advise you against hacking the data format, but they do know a lot more about how Stata handles large dataset internally, and can probably help you find a way to make what you want to achieve doable. Also since that is a private conversation you can get a lot more in the finer details needed to find the find the solution.

As an aside, your question is a classic XY-problem: you actual problem is that you cannot save a large dataset, but you ask about the header of .dta file. You had a hunch that you could do it by hacking Stata's data format, and got lost in that rabbit hole. If you take a step back: it does not matter how you save that big file, as long as it is done. If you take another step back, do you really need to save that big of a file? Maybe, but maybe not. If you can get your work done by side-stepping the entire problem then that works too. So if you ask your question to Stata's techsupport, tell them about your real problem: what is the research problem that you try to solve, how did you end up with those gargantuan datasets, and why do you want to save them. This increases the number of potential angles through which your problem could be solved and thus increases the chance that an answer can be found.

[Mike Lacy]

My first thought was to think of saving the file in a few different parts that share a name stub and open them with -append- on the target machine with more memory:

... code to save pieces of the file as stub1, stub2, etc. ...
local flist: dir . files "stub*.dta"
append using `flist'

This is in the spirit of what Maarten suggests, i.e., focusing on an easy way to accomplish the desired goal. Perhaps, though, there's something about the original problem that precludes my simplistic approach.

[daniel klein]

No need for guessing. Stata's dta format is documented at the byte level.

[Nick Cox]

... a reference given already in #2 of this thread.

[Daniel Feenberg]

I have a basic question - how is it that the OP can create a workspace in memory that Stata doesn't have enough memory to save? In my experience, -save- consumes disk space, not core. I just did a little experiment running

set obs 10000000
forvalues i=1/100 {
    gen double x`i'=_n*`i'
}
save foo

The Linux command top tells me the maximum virtual memory used is 9.48GB, and it doesn't increase during the -save-. The workspace size (from -memory-) is 9.38GB. This is with Stata version 19.5.

I am guessing I maybe don't understand the question as well as the rest of you.

[Clyde Schechter]

Re #13:

On the one hand, I can imagine that when the -save- command is invoked, Stata might request a little bit of additional memory to hold a newly generated file header or something like that. If so, it is possible that a data set that just barely fit into maximum allocated memory with very little or no room to spare might be -use-able but un-save-able. Emphasis on imagine: -use- and -save- are built-in commands, and only the people at StataCorp know if this scenario is really possible.

That said, I don't think that's what's going on here. My interpretation of OP's problem is that he can't even append his two subsets of the data because, combined, they break memory limits. I realize that's not what is literally said in #1, but it strikes me as a more reasonable and likely explanation of what's happening. And I also conjecture that the other responders in this thread interpreted it this way.