3900 unable to allocate real <tmp>[1071973170,1]

[ben earnhart]

I have a user who is able to replicate this error on two machines: a Linux box with (I believe) 24 GB RAM and Stata 13 IC, and a Windows 7 box with 16 GB RAM Stata 13 IC *and* SE. All the same error. Based on previous posts, seems to be a memory issue, but I'd expect IC to run into problems before SE, or one system to barf at a slightly different point if he's trying to do the impossible. Programming bug (I can find out what procedure is doing it)? Would upgrading RAM to 32 GB under Windows help?

Thanks for any thoughts.

[Brendan Halpin]

By my calculations that's about 8GB, which is a lot of memory even on big machines. Can you confirm that that sort of memory is free at the time of calling?

How exactly is the error triggered?

[ben earnhart]

I'm *pretty* sure he has that much RAM available; the Linux cluster dedicates that much RAM (24 GB) to any given user, and his Windows box is generally running minimal stuff, maybe 2 GB worth tops. I'll find out tomorrow details on what kind of data he's using and what procedure/.ado files. Odd thing is that it barfs at the same point with both OSs and both flavors of Stata. It takes about four hours to replicate the problem, so I guess I can set up a trace to see how much RAM is available when it errors out.

[Brendan Halpin]

A trace is a good idea, particularly if it takes so long for the problem to emerge (note you can get Stata to check memory with the memory command). Looking at the code, too. It could be that the code has already allocated lots of memory when the crash happens, perhaps by unnecessarily duplicating things in memory.

[ben earnhart]

I verified that Stata could access almost all of his 16 GB or RAM; generating a million cases with 100,000 random variables, it pegged at around 14.5 GB physical memory it was taking up. Interestingly enough, his process that crashed was topping out at a little over 8 GB, and would not go beyond that. He was running a package called "nearstat," and I have pasted the output below. I wish I could rename this posting, since it seems to point to a problem with nearstat, not Stata itself, since we get the same error under Linux with 24 GB RAM, Windows with 16 GB, and both Stata IC and SE. Any clues?

. nearstat lat_11 long_11, near( lat_95_10 long_95_10 ) distvar(DistPV3m) ncount(NrNeigbPV3m) dband (0 3) r (3958.761)

select(): 3900 unable to allocate real <tmp>[1071973170,1]
nearstat_calcdist(): - function returned error
<istmt>: - function returned error

[Brendan Halpin]

nearstat is a user written add-on (it's conventional to identify where you got stuff like that). I've had a quick look at it and I suspect the problem is that it is allocating multiple vectors of this size. The nearstat_calcdist() mata function has multiple calls to create vectors of _N*1, sometimes in pairs.

I'm not sure I have it right, though, as that would suggest your data set size is 1,071,973,170, which is implausible unless there is some sort of expansion going on.

[ben earnhart]

Well, I learned more since posting this afternoon. First, that nearstat will limit itself about 8 GB of RAM when doing most of its computations (which runs about four hours on his PC), then memory usage shoots up to as much it can grab (about 15 GB when it pooped out) in the last few minutes -- then it crashes, on the PC. Second, when he was using the High-Performance-Computing facility which uses Linux, he had been submitting jobs to his login node, which only has 4 GB, not the 24 GB if he had been submitting jobs properly. So tomorrow, we will try submitting his job on the node with 24 GB and see what happens. Bizarre that it gives the same error; I assumed you were right that it was 8 GB, but no, seems to get the same error at 4ish and 15ish GB.

[ben earnhart]

BTW -- do you think it would help if we compressed his data first? I don't know if the data is stored as float, long, or what, but if the size of the original data file matters, it might help?

[Mike Lacy]

If it's relevant to the problem at hand, I'd consider giving the package -geonear- a try (package geonear from http://fmwww.bc.edu/repec/bocode/g) as an alternative to -nearstat- . I once had a problem that using -geonear- (if I remember correctly it was that package) I estimated would have taken several months. On a whim, I tried -geonear- and it ran in less than five minutes. Perhaps -geonear- would manage memory more nicely as well for the current problem..

Regards, Mike

[Robert Picard]

I'm the author of geonear (available from SSC) which finds nearest neighbors using geodetic distances. I can confirm that geonear is orders of magnitude faster than nearstat (also from SSC). The more points in the data, the more geonear smokes the competition. The standard method to find nearest neighbors involves calculating the distance between all points. This is _N*_N distances to calculate. In the example below, it takes nearstat 10 seconds to calculate the nearest neighbors between two sets of 1000 points (1M distances). When redone with two sets of 2000 points (4M distances), run time is close to 60 seconds. By contrast, geonear does the same tasks in .59 and .95 seconds respectively.

I was not familiar with nearstat so I prepared the following test to show how to replicate the nearstat example in #5 using geonear. Unfortunately, there seems to be a problem with how nearstat computes some distances. The example also uses geodist (also from SSC) to recalculate the distances generated by nearstat.

. clear

. timer clear

. set obs 1000
obs was 0, now 1000

. set seed 1234

. 
. * generate points around the globe and save
. gen id_11 = _n

. gen double lat_11 = -90 + 180 * uniform()

. gen double long_11 = -180 + 360 * uniform()

. gen id_95 = _n

. gen double lat_95_10 = -90 + 180 * uniform()

. gen double long_95_10 = -180 + 360 * uniform()

. sum

    Variable |       Obs        Mean    Std. Dev.       Min        Max
-------------+--------------------------------------------------------
       id_11 |      1000       500.5    288.8194          1       1000
      lat_11 |      1000   -2.502279     52.4849  -89.93621   89.77325
     long_11 |      1000    .3267772    104.8923  -179.7503   178.5147
       id_95 |      1000       500.5    288.8194          1       1000
   lat_95_10 |      1000   -1.725084    52.64324   -89.9422   89.99214
-------------+--------------------------------------------------------
  long_95_10 |      1000    4.252457    103.5606  -179.2025   179.4711

. save "geodata", replace
file geodata.dta saved

. 
. * find nearest neighbors within 500 km of each point
. timer on 1

. geonear id_11 lat_11 long_11 using "geodata", n(id_95 lat_95_10 long_95_10) long within(500) ra(6371.009)

-------------------------------------------------------------------------------
Unique base locations   = 1,000          Unique neighbor locations = 1,000     
Bases * Neighbors       = 1,000,000      Number of regions         = 42        
Computed distances      = 158,064        Total run time (seconds)  = .586
-------------------------------------------------------------------------------

. timer off 1

. 
. * count the number of neighbors found and collapse to one obs per id_11
. by id_11: gen Nid_95 = _N

. by id_11: keep if _n == 1
(2653 observations deleted)

. replace Nid_95 = 0 if km_to_id_95 > 500
(208 real changes made)

. 
. * merge back the coordinates for each point
. merge 1:1 id_11 using "geodata", assert(match) keepusing(lat_11 long_11) nogen

    Result                           # of obs.
    -----------------------------------------
    not matched                             0
    matched                             1,000  
    -----------------------------------------

. merge m:1 id_95 using "geodata", keep(match) keepusing(lat_95_10 long_95_10) nogen

    Result                           # of obs.
    -----------------------------------------
    not matched                             0
    matched                             1,000  
    -----------------------------------------

. sort id_11 km_to_id_95 id_95

. list in 1/5

     +----------------------------------------------------------------------------------------+
     | id_11   id_95   km_to_~95   Nid_95       lat_11      long_11    lat_95_10   long_95_10 |
     |----------------------------------------------------------------------------------------|
  1. |     1     964   660.83893        0   -4.1364162    38.089997    -2.927109    32.260084 |
  2. |     2     854    765.7025        0   -30.484203   -73.790075   -37.085015   -76.153354 |
  3. |     3     683   439.91325        1   -27.973676   -127.62344   -26.154621   -123.67774 |
  4. |     4     624   392.19837        1   -25.607773    85.637097   -24.706801    89.404757 |
  5. |     5     495   198.03966        5    79.669831    21.471668    80.512362    30.607669 |
     +----------------------------------------------------------------------------------------+

. 
. * redo using -nearstat-
. use "geodata", clear

. timer on 2

. nearstat lat_11 long_11, near( lat_95_10 long_95_10 ) distvar(DistPV3m) ncount(NrNeigbPV3m) dband (0 500) nid(id_95 closest)

 Distance (in km) calculations completed successfully and/or all requests processed

. timer off 2

. 
. * merge back the coordinates of the nearest id_95 neighbor
. drop id_95    lat_95_10   long_95_10

. rename closest id_95

. merge m:1 id_95 using "geodata", keep(match) keepusing(lat_95_10 long_95_10) nogen

    Result                           # of obs.
    -----------------------------------------
    not matched                             0
    matched                             1,000  
    -----------------------------------------

. sort id_11

. 
. * calculate the distances again and show incorrect distances
. geodist lat_11 long_11 lat_95_10 long_95_10, gen(d) sphere r(6371.009)

. gen delta = d - DistPV3m

. list in 1/5

     +-----------------------------------------------------------------------------------------------------------------+
     | id_11       lat_11      long_11   id_95    DistPV3m   NrNei~3m    lat_95_10   long_9~10           d       delta |
     |-----------------------------------------------------------------------------------------------------------------|
  1. |     1   -4.1364162    38.089997     964   660.83893          0    -2.927109   32.260084   660.83893    5.68e-13 |
  2. |     2   -30.484203   -73.790075     783   212.63751          1   -28.663642   179.05642   9878.0956    9665.458 |
  3. |     3   -27.973676   -127.62344     729   414.58239          3   -30.062668   122.36385   10177.099    9762.517 |
  4. |     4   -25.607773    85.637097     624   392.19837          1   -24.706801   89.404757   392.19837   -2.84e-13 |
  5. |     5    79.669831    21.471668     495   198.03966          5    80.512362   30.607669   198.03966   -3.69e-13 |
     +-----------------------------------------------------------------------------------------------------------------+

. 
. * Timing results
. timer list 1
   1:      0.59 /        1 =       0.5880

. timer list 2
   2:     10.21 /        1 =      10.2130

.

[ben earnhart]

Wow, thanks for the geonear examples. Sounds very promising. Looking at the documentation, we weren't sure it could do what he wanted, showing sites from 2011 relative to sites from 1995 (also other lags/years). I think it's doing just that, right? We'll run stuff here on some smaller datasets where we can get nearstat to work, and compare to the results from geonear.

[Robert Picard]

My sample code shows that in some cases, nearstat does not calculate (or pick) the nearest neighbors correctly. I emailed the author but the email bounced.

I started a new thread here with some additional code that further illustrates the bug in nearstat. You should look at the "brute force" example to fully understand how to find the nearest neighbors by computing distances between all pairwise combinations of sites between 2011 and 1995. If this is what you want, then use geonear instead, it will be orders of magnitude faster.

[ben earnhart]

Anybody who saw my post from a few minutes ago, please ignore! I hadn't realized that switching units to miles, you still needed to get the radius in Km. So my results were totally wonky. Once I found my error, nearstat correlates 1.0 with geonear on the data I was using, and the means and maxes are the same!

[ben earnhart]

ps. big, Big BIG thanks to Robert Picard and Mike Lacy. The example file I was given was a small one that the user said only took a couple of minutes to run (on his I7 with 16 GB). It ran in about 0.31 seconds on my old Core-2 Duo with 4 GB. The potential errors in nearstat's calculations are beyond me, but if we can get this stuff to run with almost no resources, much faster, and presumably more accurate results, it just doesn't get any better than that! THANK YOU

[ben earnhart]

ps. Taking advantage of geonear while in long form has allowed us to increase our capabilities. I can do weighted distances (densities) where closer observations get more weight based on arbitrary formulas! It's really cool. Nearstat gave him exactly what he wanted, no more and no less, whereas geonear is more flexible. Sorry for the multiple (arguably redundant) posts, but with the new forum-style interface, I'm not clogging in-boxes, so I think it's OK to gush?