Cloud computing Stata/BE

[Michael Schuster]

Dear all,

I have a very big geocoded panel dataset of about 21 million observations or 4 million if I restrict to the main years. Yet, I am still a student and my university gives just access to Stata/BE 17.0. I have tried to run a very complex -acreg- regression with spatial and temporal correlation and it took me forever (more than 3 days) for just one regression. I am thinking, therefore, about using cloud computing in order to get access to more memory (RAM). I have some questions left:

1. Would that help me? Does that even make sense with the smallest version of Stata? Are there RAM limitations on this version?

2. Can I use my license in the cloud and how?

I am really not an expert in cloud computing and the technical conditions of Stata.

Thanks for any advice on that topic.

[ericmelse]

Dear Michael,

You could take a look at the website of CODE OCEAN, who are partners of Stata Corp. (see here).

They facilitate running Stata do files using their cloud server, i.e. you can put both your dta and do file there and run it using Stata on their server.
More information about using Stata on/with CODE OCEAN is available here.

Besides the ability to run Stata from a cloud server for your research purposes, you can also 'publish' your project from it so your readers are able to run (replicate) your project and inspect results without having to need to download data etc.
I suppose that could be usefull as well in your case.

Best,
Eric

[Michael Schuster]

Dear Eric,

Thanks for you suggestions. So here is my problem: I have a geocoded (5x5km grid cells) panel data set which ranges from 2000 to 2015 with 5 years gap (meaning that I have the years 2000, 2005, 2010, 2015, see example) over all of Sub-Saharan Africa. I have about 4 million observations. I run the rather new regression command -acreg- GDP growth on HIV prevalence or male circumcision as an instrument, respectively. As I am using Stata/BE 17.0 on a Mac this regression takes forever.

* Example generated by -dataex-. For more info, type help dataex
clear
input long grid_id double(longitude latitude) float(year growth_GDP_PPP_total) double(HIV_mean_prev MC_mean_prev) float ln_gdp_ppp_total
612551 2.72916666678866 13.4791666663955 2000         .  .0113952420651913 .990811169147491 16.076998
612551 2.72916666678866 13.4791666663955 2005  .1706244 .00958700943738222 .994386434555054 16.234533
612551 2.72916666678866 13.4791666663955 2010 .22794993 .00588105153292418         .9955161  16.43988
612551 2.72916666678866 13.4791666663955 2015  .1681127 .00376865570433438         .9969429  16.59527
612552 2.77083333345551 13.4791666663955 2000         .  .0114019885659218 .990886151790619 15.395984
612552 2.77083333345551 13.4791666663955 2005 .15884513 .00973280053585768 .993543684482574 15.543407
612552 2.77083333345551 13.4791666663955 2010  .2160491 .00575962476432323         .9960363 15.739015
612552 2.77083333345551 13.4791666663955 2015  .1654788 .00376936281099916         .9968488 15.892147
612553 2.81250000012236 13.4791666663955 2000         .  .0111839063465595 .990696012973785 15.395984
612553 2.81250000012236 13.4791666663955 2005 .15884513 .00968935340642929 .994269490242004 15.543407
612553 2.81250000012236 13.4791666663955 2010  .2160491 .00581180397421122         .9959334 15.739015
612553 2.81250000012236 13.4791666663955 2015  .1654788 .00384826818481088         .9973352 15.892147
612554  2.8541666667892 13.4791666663955 2000         .  .0112042371183634 .991451442241669 14.863758
612554  2.8541666667892 13.4791666663955 2005 .20897754  .0095862802118063 .994564950466156 15.053534
612554  2.8541666667892 13.4791666663955 2010  .2674411 .00588257005438209         .9956774 15.290533
612554  2.8541666667892 13.4791666663955 2015 .17675897 .00391481164842844         .9969919 15.453298
612555 2.89583333345605 13.4791666663955 2000         .  .0114256078377366 .991257309913635 14.863758
612555 2.89583333345605 13.4791666663955 2005 .20897754 .00964953191578388 .994801461696625 15.053534
612555 2.89583333345605 13.4791666663955 2010  .2674411 .00580572756007314          .996235 15.290533
612555 2.89583333345605 13.4791666663955 2015 .17675897 .00385410059243441         .9970776 15.453298
612556  2.9375000001229 13.4791666663955 2000         .  .0112883280962706 .990532875061035 15.847743
612556  2.9375000001229 13.4791666663955 2005  .2267739 .00957189407199621 .994763970375061  16.05213
612556  2.9375000001229 13.4791666663955 2010 .28537965 .00596617860719562         .9965305 16.303185
612556  2.9375000001229 13.4791666663955 2015  .1805613 .00390501879155636         .9974594 16.469175
612557 2.97916666678975 13.4791666663955 2000         .  .0113649414852262 .991434395313263 15.847743
612557 2.97916666678975 13.4791666663955 2005  .2267739 .00968039501458406 .994557976722717  16.05213
612557 2.97916666678975 13.4791666663955 2010 .28537965 .00592478085309267          .995773 16.303185
612557 2.97916666678975 13.4791666663955 2015  .1805613 .00395362172275782         .9971524 16.469175
612558 3.02083333345659 13.4791666663955 2000         .  .0111322328448296 .991210341453552 15.373527
612558 3.02083333345659 13.4791666663955 2005 .21816477 .00974737480282784 .995118200778961 15.570872
612558 3.02083333345659 13.4791666663955 2010 .27670395 .00586639903485775         .9961905 15.815154
612558 3.02083333345659 13.4791666663955 2015 .17872886 .00384415732696652         .9976268  15.97959
612559 3.06250000012344 13.4791666663955 2000         .  .0109965046867728 .992109060287476 15.373527
612559 3.06250000012344 13.4791666663955 2005 .21816477 .00943726394325495  .99527645111084 15.570872
612559 3.06250000012344 13.4791666663955 2010 .27670395 .00583592336624861         .9966778 15.815154
612559 3.06250000012344 13.4791666663955 2015 .17872886 .00401432812213898         .9973729  15.97959
612560 3.10416666679029 13.4791666663955 2000         .  .0113323573023081 .992041945457458  14.89746
612560 3.10416666679029 13.4791666663955 2005  .2095828 .00946436263620853 .995438098907471 15.087736
612560 3.10416666679029 13.4791666663955 2010 .26805162 .00599796092137694         .9966905 15.325218
612560 3.10416666679029 13.4791666663955 2015 .17688897 .00396168138831854         .9971074 15.488092
612561 3.14583333345714 13.4791666663955 2000         .  .0110401883721352 .991976678371429  14.89746
612561 3.14583333345714 13.4791666663955 2005  .2095828 .00958679988980293 .995556831359863 15.087736
612561 3.14583333345714 13.4791666663955 2010 .26805162 .00583096733316779         .9964032 15.325218
612561 3.14583333345714 13.4791666663955 2015 .17688897 .00387062761001289         .9977303 15.488092
612562 3.18750000012398 13.4791666663955 2000         .  .0109043167904019 .991712152957916 14.707047
612562 3.18750000012398 13.4791666663955 2005  .2061671 .00945729669183493 .995767056941986 14.894495
612562 3.18750000012398 13.4791666663955 2010 .26460683 .00590855488553643         .9961565 15.129256
612562 3.18750000012398 13.4791666663955 2015 .17615303 .00406615389510989         .9978445 15.291505
612563 3.22916666679083 13.4791666663955 2000         .  .0111725609749556 .991860210895538 14.707047
612563 3.22916666679083 13.4791666663955 2005  .2061671  .0094987777993083 .995502114295959 14.894495
612563 3.22916666679083 13.4791666663955 2010 .26460683 .00594787485897541         .9967039 15.129256
612563 3.22916666679083 13.4791666663955 2015 .17615303 .00397224863991141         .9976168 15.291505
612564 3.27083333345768 13.4791666663955 2000         .  .0111121172085404 .991329789161682  14.57839
612564 3.27083333345768 13.4791666663955 2005  .2038649 .00957683473825455 .995276927947998 14.763928
612564 3.27083333345768 13.4791666663955 2010 .26228437 .00598770519718528         .9968782  14.99685
612564 3.27083333345768 13.4791666663955 2015  .1756559 .00399381387978792         .9975035 15.158677
612565 3.31250000012453 13.4791666663955 2000         .  .0114873237907887 .992097437381744  14.57839
612565 3.31250000012453 13.4791666663955 2005  .2038649 .00946289673447609 .995153963565826 14.763928
612565 3.31250000012453 13.4791666663955 2010 .26228437 .00585305970162153         .9963712  14.99685
612565 3.31250000012453 13.4791666663955 2015  .1756559 .00393765699118376         .9975535 15.158677
612566 3.35416666679138 13.4791666663955 2000         .  .0113830491900444 .991927325725555 14.350263
612566 3.35416666679138 13.4791666663955 2005  .1997931 .00933026615530252  .99556028842926 14.532412
612566 3.35416666679138 13.4791666663955 2010  .2581767 .00596584426239133         .9969015 14.762075
612566 3.35416666679138 13.4791666663955 2015  .1747739 .00400689383968711         .9977459  14.92315
612567 3.39583333345822 13.4791666663955 2000         .  .0112893683835864 .991742551326752 14.350263
612567 3.39583333345822 13.4791666663955 2005  .1997931 .00948534719645977 .995655298233032 14.532412
612567 3.39583333345822 13.4791666663955 2010  .2581767 .00597272533923388          .996778 14.762075
612567 3.39583333345822 13.4791666663955 2015  .1747739 .00397419044747949         .9978979  14.92315
612568 3.43750000012507 13.4791666663955 2000         .  .0112604601308703 .991497755050659 14.434425
612568 3.43750000012507 13.4791666663955 2005 .20129374 .00944491103291512 .995439946651459 14.617825
612568 3.43750000012507 13.4791666663955 2010 .25969052 .00585258239880204         .9967705  14.84869
612568 3.43750000012507 13.4791666663955 2015  .1750993 .00398071948438883         .9977684 15.010043
612569 3.47916666679192 13.4791666663955 2000         .  .0110805658623576  .99097603559494 14.434425
612569 3.47916666679192 13.4791666663955 2005 .20129374 .00929000787436962 .995123147964478 14.617825
612569 3.47916666679192 13.4791666663955 2010 .25969052 .00608902750536799          .997037  14.84869
612569 3.47916666679192 13.4791666663955 2015  .1750993 .00406918488442898         .9978561 15.010043
612570 3.52083333345876 13.4791666663955 2000         .  .0112151755020022 .991125106811523  14.24318
612570 3.52083333345876 13.4791666663955 2005  .1978866 .00939759518951178  .99509209394455 14.423738
612570 3.52083333345876 13.4791666663955 2010 .25625297 .00595856970176101         .9969543 14.651872
612570 3.52083333345876 13.4791666663955 2015 .17435986 .00402311561629176         .9977231 14.812595
612571 3.56250000012561 13.4791666663955 2000         .  .0111126946285367 .990645706653595  14.24318
612571 3.56250000012561 13.4791666663955 2005  .1978866 .00936393439769745 .995209515094757 14.423738
612571 3.56250000012561 13.4791666663955 2010 .25625297 .00603470485657454         .9964469 14.651872
612571 3.56250000012561 13.4791666663955 2015 .17435986 .00407639611512423         .9980413 14.812595
612572 3.60416666679246 13.4791666663955 2000         .  .0110590206459165 .990896940231323 14.360893
612572 3.60416666679246 13.4791666663955 2005  .1999826 .00932643376290798 .995149374008179   14.5432
612572 3.60416666679246 13.4791666663955 2010 .25836778 .00612880475819111         .9970773 14.773016
612572 3.60416666679246 13.4791666663955 2015 .17481497 .00410562753677368          .997489 14.934126
612573 3.64583333345931 13.4791666663955 2000         .    .01135338190943 .991044461727142 14.360893
612573 3.64583333345931 13.4791666663955 2005  .1999826 .00932671315968037 .994888842105865   14.5432
612573 3.64583333345931 13.4791666663955 2010 .25836778 .00605473481118679         .9963985 14.773016
612573 3.64583333345931 13.4791666663955 2015 .17481497 .00424656225368381         .9981191 14.934126
612574 3.68750000012615 13.4791666663955 2000         .  .0113125629723072 .990260899066925 14.683136
612574 3.68750000012615 13.4791666663955 2005 .20573898 .00950930174440145 .994950354099274  14.87023
612574 3.68750000012615 13.4791666663955 2010 .26417488 .00610860949382186         .9966736  15.10465
612574 3.68750000012615 13.4791666663955 2015 .17606068 .00418301625177264         .9977734  15.26682
612575   3.729166666793 13.4791666663955 2000         .  .0114521058276296 .991335511207581 14.683136
612575   3.729166666793 13.4791666663955 2005 .20573898 .00961873307824135 .994603216648102  14.87023
612575   3.729166666793 13.4791666663955 2010 .26417488 .00611228682100773         .9970282  15.10465
612575   3.729166666793 13.4791666663955 2015 .17606068 .00424239179119468         .9978874  15.26682
end

After that I regress as follows:

 acreg growth_GDP_PPP_total ln_gdp_ppp_total (HIV_mean_prev = MC_mean_prev), id(grid_id) time(year) lagcut(15)spatial latitude(latitude) longitude(latitude) dist(40)

[ericmelse]

Dear Michael,

I suggest you register at CODE OCEAN and write them for their suggestion how to proceed (including a link to this post).

Best,
Eric