Parallelising tasks

[Federico Nutarelli]

Hi all,

maybe this is a naive question. However, let's say I have either a for loop or a collapse like for instance the following:

collapse (first) hs4codeprod prod6_code exporter_iso_3digit dummymax* (sum) tradevalue tradequantity (max) _fillin, by(id_iso4)

since it takes a lot to run it all, I was thinking about parallelising it in the 8 cores of my PC. Is it possible to do so in stata?

Thank you a lot

[Felix Bittmann]

Yes, probably, if you can rewrite your code so it can be done my multiple instances. For example, split up by variables and merge afterwards.
https://journals.sagepub.com/doi/abs...36867X19874242
https://github.com/gvegayon/parallel
If you cannot reach behind the paywall, I think the github rep provides the working paper.

[Federico Nutarelli]

Thank you very much

[Joro Kolev]

You can also try the user written -gtools-,
-https://gtools.readthedocs.io/en/latest/usage/gcollapse/-

If you try it, do report whether it is doing it a lot faster.

[Oscar Ozfidan]

Federico Nutarelli I would recommend batcher package from ssc. There is also parallel package from ssc but batcher is much better for me as it does not require creation of multiple do files. You can run as many instances as you want given the core limits of your set up with batcher from a single file.

[Federico Nutarelli]

Thank you all for your kind replies. I tried only parallel for my task. But I will try all of them in a later project. I will let you know if the speed increased a lot