java.lang Issue when Generating a Graph

[Blake Eliason]

Hi all, I am attempting to create a simple graph and I am receiving an error I have never received before, being:

java.lang.OutOfMemoryError: Javaheap space

I'm working with a dataset that contains 34 variables and 4 million observations. I am using Stata 17.0 on Windows 10. Here is the code I'm using to generate the graph:

    * Plot Mean Annual Pay
    twoway (line mean_pay_MN yq, lcolor(ltblue)) (scatter mean_pay_MN yq, mcolor(ltblue))                         ///
           (line mean_pay_US yq, lcolor(red)) (scatter mean_pay_US yq, mcolor(red)),                            ///
           xline(`xpos', lcolor(black))                                                                            ///
           text(60000 `xpos' "MN Noncompete" "Ban Enacted", place(right) size(small) orientation(horizontal))    ///
           xtitle("Year-Quarter")                                                                                ///
           ytitle("Mean Annual Pay")                                                                            ///
           legend(pos(6) order(1 "Minnesota Mean Pay" 3 "All Other States Mean Pay"))
    graph export "$figures/sum_stats/mean_annual_pay.jpg", replace

I've read in other forums to increase the java heap size, which I have done to an extreme amount and I still run into issues. When I run

java query

, I receive

set java_heapmax     16384m

The graph generates, but it appears the export is where the program has issues, as the file generated is 0 kb and blank.

I have generated plenty of other graphs with Stata but have never run into this issue before. If you have any advice or questions, let me know.

[Daniel Schaefer]

Hi Blake,

It sounds like you are running out of memory. Doesn't really matter how large you make the heap if there isn't enough physical memory in your computer to support it. Do you know how much RAM your system has? 32 gigs ram is 32000 megabytes, so if you have 32 gigs you'd be allocating about half of that to the Java heap alone. Your OS may also place a hard upper limit on the amount of memory any process can allocate.

The plot will display in Stata, right? I might start by saving the plot manually first using the GUI. Then you might try running the plot line, letting it finish executing, then running graph export separately. Maybe Java garbage collection will free enough space in the interim that you will be able to export the plot?

Edit: Maybe graph export doesn't store data in the Java heap, and so there is no stack memory (or whatever) left for Stata to build the .jpg before writing it to the disk? You aren't getting the heap error any more right? So that suggests it is not a heap problem...

[Blake Eliason]

Daniel: thanks for your response. See my answers below:

1. My system has 256gb of RAM.
2. The plot does display in Stata until the java.lang.OutOfMemoryError: Javaheap space error appears, at which point the window that displays the graph is forced out.
3. I am still receiving the Java heap error, if that changes the analysis. In the original post, I probably should have made clear that the java.lang.OutOfMemoryError: Javaheap space appears after a few hours of running the graph generation code. This is another part of my confusion, in that I have never had to wait hours to generate a single graph in Stata before.

[James Hassell (StataCorp)]

Blake Eliason, the only time Java gets used when graphing is when exporting to JPG, or when exporting to a PNG when using console Stata (no GUI). It sounds like you have a graph window so not using console Stata, and the command you posted is exporting to a JPG, so that explains why Java is being used at all.

A couple of things come to mind that my be causing the java.lang.OutOfMemoryError. Maybe there is some problem with the code or the library used to generate the JPG file from the graph, although I don't think anything of that nature has ever been reported. Or maybe it is related to the large number of points/lines; you mentioned your dataset was 4 million observations.

Please send a reproducible example to [email protected] including the dataset (if possible) and we'll be happy to take a look.

A likely work around is to export to PNG instead.

Best regards,
James Hassell

[Daniel Schaefer]

Okay, so you want to draw about 4 million points twice with scatter. You also want to draw lines between every adjacent pair of points. So you want to draw 16 million objects: 8 million points, then 8 million lines between points. Let's take execution time to be a proxy for memory efficiency. I don't have trouble drawing the 8 mil points. Execution time is on the order of minutes.

clear
set obs 4000000

gen x1 = runiform()
gen y1 = x1
gen x2 = runiform()
gen y2 = (x2 * 2) + 2

graph twoway (scatter x1 y1) (scatter x2 y2)

Lines are still executing for me. Possibly still on the order of minutes (have to wait to find out) but could be much longer.

graph twoway (line x1 y1) (line x2 y2)

Quick sanity check. Are you sure you want line instead of lfit here?

Edit: crossed with #4

[Blake Eliason]

James Hassell (StataCorp) I ended up switching to .png files and that immediately fixed the problem. I think journals are getting picky about image file types these days, so I may end up emailing [email protected] if they require .jpg. Otherwise, thanks for your help!

[Blake Eliason]

Daniel Schaefer it appears that a workaround recommended by James Hassell (switching to .png instead of .jpg) fixed my problem. Thanks for your help!

[Daniel Schaefer]

Glad you found a solution! By the way, it should be pretty trivial to convert from .png to .jpg using whatever software is built into your OS. If you have python, you might consider automating the conversion in your Stata do file with python integration as a workaround. The reverse is not necessarily as straightforward, since .jpg is a lossy format, so you tend to lose image resolution with .jpg.