statsby vs rangerun

[River Huang]

Dear All, I heard that -rangerun- can do whatever -statsby- does, and more quickly. My question is how to write a -rangerun- code for estimating the following regressions?

* Example generated by -dataex-. To install: ssc install dataex
clear
input long id float yw double(ri rm) float year
 1 2861  -.04798   .01786 2015
 1 2862  .019231  .027828 2015
 1 2863  -.06311 -.001805 2015
 1 2864 -.032639 -.034765 2015
 1 2865 -.030151 -.037525 2015
 1 2866  .032568  .044766 2015
 1 2867  .002867  .016261 2015
 1 2868        0  .018274 2015
 1 2869 -.039314 -.013548 2015
 1 2870  .115327  .039701 2015
 1 2871  .022015   .07656 2015
 1 2872 -.015666  .027218 2015
 1 2873  .051061  .056553 2015
 1 2874  .249211  .041905 2015
 1 2875  .034848  .051511 2015
 1 2876 -.046663  .031184 2015
 1 2877  .034696  .008503 2015
 1 2878 -.050898 -.044991 2015
 1 2879 -.025237  .034662 2015
 1 2880  .053074  .093365 2015
 1 2881  -.05839 -.008247 2015
 1 2882  .063969   .10488 2015
 1 2883  .012883  .034483 2015
 1 2884  -.11387 -.136604 2015
 1 2885 -.058783 -.074094 2015
 1 2886 -.050835 -.140998 2015
 1 2887  .136955  .042479 2015
 1 2888 -.069987  .038188 2015
 1 2889 -.031838  .038027 2015
 1 2890 -.076233 -.105651 2015
 1 2891  .020227  .028003 2015
 1 2892  .002379  .067808 2015
 1 2893  -.09019 -.124472 2015
 1 2894 -.058261 -.093368 2015
 1 2895  .093259 -.040442 2015
 1 2896 -.074324  .022072 2015
 1 2897 -.013686 -.045713 2015
 1 2898 -.024052  .000718 2015
 1 2899 -.005687 -.009631 2015
 1 2900  .039085   .05271 2015
 1 2901  .030275  .077331 2015
 1 2902  .021371   .01406 2015
 1 2903  -.00959 -.007376 2015
 1 2904  .090669  .063103 2015
 1 2905 -.012107  .003349 2015
 1 2906  .025327  .022665 2015
 1 2907 -.065339  -.05515 2015
 1 2908  .033248  .029686 2015
 1 2909 -.023927 -.024785 2015
 1 2910  .033812  .049859 2015
 1 2911  .014718  .015578 2015
 1 2913  -.07256  -.11381 2016
 1 2914 -.059353 -.096372 2016
 1 2915 -.005736  .008176 2016
 1 2916 -.038462 -.067808 2016
 1 2917    -.008  .015627 2016
 1 2919  .012097  .043275 2016
 1 2920   -.0249 -.040242 2016
 1 2921  .062308  .032311 2016
 1 2922 -.023077 -.023095 2016
 1 2923  .037402  .061767 2016
 1 2924  .004744  .013456 2016
 1 2925   .00661  .010947 2016
 1 2926 -.008443 -.004801 2016
 1 2927  .029328  .033933 2016
 1 2928 -.030331 -.043164 2016
 1 2929  .001896 -.004306 2016
 1 2930  -.00473 -.007785 2016
 1 2931  -.01616 -.034009 2016
 1 2932 -.004831  .001119 2016
 1 2933 -.002913  .000574 2016
 1 2934  .022395  .047394 2016
 1 2936 -.004857 -.011127 2016
 1 2937 -.001166 -.007345 2016
 1 2938  .016336  .030832 2016
 1 2939  .003444  .024083 2016
 1 2940  .028604   .02332 2016
 1 2941 -.005562 -.011295 2016
 1 2942  .029083 -.016186 2016
 1 2943 -.017391  .000192 2016
 1 2944  .050885  .024757 2016
 1 2945  .001053  .024473 2016
 1 2946 -.006309 -.013214 2016
 1 2947        0  -.00175 2016
 1 2948 -.007407  .007795 2016
 1 2950  .009934   .01416 2016
 1 2951 -.008743 -.007065 2016
 1 2953  .002205  .021257 2016
 1 2954    .0044  .007937 2016
 1 2955  .004381  .002376 2016
 1 2956 -.006543  .005227 2016
 1 2957  .007684   .02185 2016
 1 2958        0  .000973 2016
 1 2959   .04793  .020443 2016
 1 2960 -.007277 -.007483 2016
 1 2961  .010471 -.002223 2016
 1 2962 -.041451 -.034551 2016
 1 2963 -.018378 -.004889 2016
18 2861  -.11284   .01786 2015
18 2862  .142544  .027828 2015
18 2863   .03263 -.001805 2015
18 2864  .019703 -.034765 2015
18 2865 -.088225 -.037525 2015
18 2866 -.032387  .044766 2015
18 2867  .029339  .016261 2015
18 2868 -.018065  .018274 2015
18 2869  .066639 -.013548 2015
18 2870  .038712  .039701 2015
18 2871 -.050554   .07656 2015
18 2872 -.038865  .027218 2015
18 2873   .11201  .056553 2015
18 2874  .002909  .041905 2015
18 2875  .171139  .051511 2015
18 2876  .018576  .031184 2015
18 2877 -.068085  .008503 2015
18 2878  .079256 -.044991 2015
18 2879  .085524  .034662 2015
18 2880  .238864  .093365 2015
18 2881  .123596 -.008247 2015
18 2882    .0658   .10488 2015
18 2883 -.058735  .034483 2015
18 2884 -.132775 -.136604 2015
18 2885 -.226667 -.074094 2015
18 2886 -.206302 -.140998 2015
18 2887 -.117603  .042479 2015
18 2888  .097199  .038188 2015
18 2889  .307544  .038027 2015
18 2890 -.227515 -.105651 2015
18 2891  .221754  .028003 2015
18 2892  .009404  .067808 2015
18 2893 -.147516 -.124472 2015
18 2894 -.136612 -.093368 2015
18 2895 -.198312 -.040442 2015
18 2896  .357895  .022072 2015
18 2897  -.10155 -.045713 2015
18 2898  .091458  .000718 2015
18 2899 -.011858 -.009631 2015
18 2900    .1004   .05271 2015
18 2901  .123228  .077331 2015
18 2902 -.017476   .01406 2015
18 2903   .08498 -.007376 2015
18 2904  -.03643  .063103 2015
18 2905  .045999  .003349 2015
18 2906  .144277  .022665 2015
18 2907 -.138194  -.05515 2015
18 2908   .00336  .029686 2015
18 2909  .029833 -.024785 2015
18 2910  .155779  .049859 2015
18 2911   .19821  .015578 2015
18 2913 -.126423  -.11381 2016
18 2914 -.184737 -.096372 2016
18 2915  .131564  .008176 2016
18 2916 -.054981 -.067808 2016
18 2917  .094209  .015627 2016
18 2919  .273171  .043275 2016
18 2920 -.213027 -.040242 2016
18 2921 -.123661  .032311 2016
18 2922  .058611 -.023095 2016
18 2923  .121228  .061767 2016
18 2924  .098526  .013456 2016
18 2925  .048147  .010947 2016
18 2926  .092073 -.004801 2016
18 2927 -.051182  .033933 2016
18 2928 -.098666 -.043164 2016
18 2929 -.028292 -.004306 2016
18 2930   .06047 -.007785 2016
18 2931 -.077508 -.034009 2016
18 2932 -.051282  .001119 2016
18 2933 -.004826  .000574 2016
18 2934  .132396  .047394 2016
18 2936 -.099297 -.011127 2016
18 2937  .187317 -.007345 2016
18 2940 -.032046   .02332 2016
18 2941 -.017827 -.011295 2016
18 2942 -.069144 -.016186 2016
18 2943  .012999  .000192 2016
18 2944 -.020165  .024757 2016
18 2945  .043031  .024473 2016
18 2946 -.000897 -.013214 2016
18 2947 -.030521  -.00175 2016
18 2948  .039815  .007795 2016
18 2950  -.01107   .01416 2016
18 2951 -.071828 -.007065 2016
18 2953  .003015  .021257 2016
18 2954 -.044088  .007937 2016
18 2955 -.012579  .002376 2016
18 2956 -.006369  .005227 2016
18 2957  .073718   .02185 2016
18 2958  .059701  .000973 2016
18 2959 -.032864  .020443 2016
18 2960  .007767 -.007483 2016
18 2961        0 -.002223 2016
18 2962  .040462 -.034551 2016
18 2963  .090741 -.004889 2016
end
format %tw yw

bys id (year): gen t = _n
xtset id t

gen F1rm = F1.rm
gen F2rm = F2.rm
gen L1rm = L1.rm
gen L2rm = L2.rm

statsby _b, by(id year) saving("b.dta", replace): reg ri F2rm F1rm rm L1rm L2rm
merge m:1 id year using "b.dta"
gen e = ri-(F2rm*_b_F2rm + F1rm*_b_F1rm + rm*_b_rm + L1rm*_b_L1rm + L2rm*_b_L2rm + _b_cons)

[Robert Picard]

No need to use rangerun (from SSC) in this case, rangestat (from SSC) can do the job:

rangestat (reg) ri F2rm F1rm rm L1rm L2rm, interval(year 0 0) by(id)
gen e2 = ri-(F2rm*b_F2rm + F1rm*b_F1rm + rm*b_rm + L1rm*b_L1rm + L2rm*b_L2rm + b_cons)

As specified above, rangestat will perform a single regression per id group and distinct value of year and carry over the results to the rest of the observations that have the same values for id and year.

[River Huang]

Thank you so much, Robert. [PS: I suppose that you meant by(id year).]
1. I can not believe -rangestat- is so fast. For more than thirty thousand regressions, my code above took more than 5 minutes, and your code took only 11 seconds!
2. A few months ago, a code took me more than one day to run (as I recalled). I wonder if your wonderful -rangestat-, and most likely -rangerun- command can be used to speed up the estimation.

clear
input long id float yw double(ri rm) float(year rind)
1 2808 -.026122  -.01162 2014   -.02187188
1 2809  -.00922 -.038157 2014   -.01669857
1 2810 -.028765 -.000472 2014  -.014618086
1 2811  .015679  .030798 2014   .007250646
1 2812 -.022298 -.010732 2014  -.005257044
1 2813 -.001754  .007665 2014  -.003436207
1 2814  .061511  .037516 2014   .036805287
1 2815 -.039735 -.001748 2014  -.011532088
1 2816 -.040517 -.028998 2014   -.02259154
1 2817  -.02965  .002966 2014   -.01255455
1 2818 -.044444  -.02618 2014  -.015652513
1 2819  .045543  .020002 2014    .02558933
1 2820 -.000927  -.00901 2014   .012808963
1 2821  .000928  .012285 2014   .004678128
1 2822  .056534  .031577 2014    .04573155
1 2823 -.052632 -.012104 2014  -.025931247
1 2824  .041667 -.034124 2014  -.008572405
1 2825 -.009778 -.005805 2014 -.0018938944
1 2826 -.008079 -.007324 2014  -.003502477
1 2827   .01991  .007051 2014   .007512956
1 2828  .027507  .007389 2014    .00437666
1 2829 -.006908  .005358 2014    .00332471
1 2830 -.010435 -.002495 2014   .002933668
1 2831  .081195  .021431 2014    .02904329
1 2832  -.02668 -.018155 2014   -.01518601
1 2833 -.003046  .015652 2014  -.005131681
1 2834  .009165  .018624 2014   .013686324
1 2835 -.024218 -.001491 2014  -.007349936
1 2836 -.007239  .008263 2014   .007533229
1 2837  .064583  .030152 2014    .04487861
1 2838  .046967  .031252 2014     .0416014
1 2839 -.014953  .008916 2014  -.010414993
1 2840  .003795  .017491 2014    .01047475
1 2841 -.021739  .009872 2014  -.010110507
1 2842 -.009662 -.013027 2014  -.013635258
1 2843   .03122  .051286 2014    .04187739
1 2844 -.018921   .00628 2014   -.01356984
1 2845  .000964 -.000651 2014 -.0006862036
1 2846 -.025048  .010357 2014 .00023193713
1 2847  .001976  .009685 2014  .0006743635
1 2848  .001972  .008467 2014    -.0011938
1 2849 -.003937 -.015561 2014  .0001800596
1 2850  .005929 -.019364 2014  -.006815978
1 2851  .083497  .052101 2014    .05594328
1 2852 -.010879  -.00246 2014  -.004929824
1 2853  .001833   .01543 2014    .06424792
1 2854 -.011894  .008564 2014  -.007100914
1 2855  .151852  .074414 2014     .1581282
1 2856  .168006  .085317 2014     .1850046
1 2857 -.040606   .00436 2014   -.05101458
1 2858   .07604   .04234 2014    .08529846
1 2859  .006667   .01011 2014     .0697964
1 2861  -.04798   .01786 2015  -.009245682
1 2862  .019231  .027828 2015    .05227714
1 2863  -.06311 -.001805 2015   -.02982882
1 2864 -.032639 -.034765 2015   -.07727697
1 2865 -.030151 -.037525 2015   -.03931737
1 2866  .032568  .044766 2015    .04443033
1 2867  .002867  .016261 2015   .004759528
1 2868        0  .018274 2015   .006919141
1 2869 -.039314 -.013548 2015   -.05308905
1 2870  .115327  .039701 2015   .062592335
1 2871  .022015   .07656 2015   .067883536
1 2872 -.015666  .027218 2015  -.011318897
1 2873  .051061  .056553 2015    .04385842
1 2874  .249211  .041905 2015    .06777364
1 2875  .034848  .051511 2015    .04459958
1 2876 -.046663  .031184 2015  -.032021668
1 2877  .034696  .008503 2015    .01202734
1 2878 -.050898 -.044991 2015  -.037017886
1 2879 -.025237  .034662 2015  -.018817553
1 2880  .053074  .093365 2015    .05103419
1 2881  -.05839 -.008247 2015   -.05323207
1 2882  .063969   .10488 2015    .07439438
1 2883  .012883  .034483 2015   .031557143
1 2884  -.11387 -.136604 2015   -.11872815
1 2885 -.058783 -.074094 2015  -.033933423
1 2886 -.050835 -.140998 2015  -.006434457
1 2887  .136955  .042479 2015    .14670287
1 2888 -.069987  .038188 2015   -.05056261
1 2889 -.031838  .038027 2015   -.03295565
1 2890 -.076233 -.105651 2015   -.08608025
1 2891  .020227  .028003 2015  .0045610815
1 2892  .002379  .067808 2015   .007599622
1 2893  -.09019 -.124472 2015   -.09144273
1 2894 -.058261 -.093368 2015   -.04754843
1 2895  .093259 -.040442 2015     .0813361
1 2896 -.074324  .022072 2015   -.04470138
1 2897 -.013686 -.045713 2015 -.0039050495
1 2898 -.024052  .000718 2015   -.00138637
1 2899 -.005687 -.009631 2015   -.01809099
1 2900  .039085   .05271 2015   .032081977
1 2901  .030275  .077331 2015    .03682121
1 2902  .021371   .01406 2015   .014840276
1 2903  -.00959 -.007376 2015  -.008879421
1 2904  .090669  .063103 2015    .09158126
1 2905 -.012107  .003349 2015  -.002904605
1 2906  .025327  .022665 2015   .005304998
1 2907 -.065339  -.05515 2015   -.04647819
1 2908  .033248  .029686 2015    .04298611
2 2808 -.023661  -.01162 2014  -.020101015
2 2809 -.058673 -.038157 2014  -.062758975
2 2810  -.04065 -.000472 2014   -.01064314
2 2811  .100282  .030798 2014    .08124962
2 2812 -.052632 -.010732 2014  -.026290694
2 2813  .004065  .007665 2014   .005708393
2 2814 -.005398  .037516 2014    .04012561
2 2815 -.028494 -.001748 2014  -.011388687
2 2816 -.061453 -.028998 2014   -.05995426
2 2817  .081845  .002966 2014    .02931733
2 2818  .035763  -.02618 2014 -.0011639306
2 2819  .063745  .020002 2014    .05722804
2 2820  .013733  -.00901 2014   .032468516
2 2821 -.001232  .012285 2014    .02583385
2 2822 -.019729  .031577 2014  -.009546324
2 2823 -.021384 -.012104 2014  -.006718597
2 2824  .020566 -.034124 2014  -.006429412
2 2825 -.005038 -.005805 2014  -.019597156
2 2826 -.029114 -.007324 2014   -.03143678
2 2827  .060606  .007051 2014   .018010318
2 2828  .058442  .007389 2014    .02861592
2 2829   .04908  .005358 2014 -.0024844094
2 2830 -.035088 -.002495 2014  -.012149008
2 2831  .021818  .021431 2014    .01079587
2 2832 -.035587 -.018155 2014  -.021365123
2 2833   .02337  .015652 2014   .025344463
2 2834  .018029  .018624 2014    .02538036
2 2835  .040142 -.001491 2014    .01597141
2 2836  .009081  .008263 2014   .031080503
2 2837  .103487  .030152 2014    .05458571
2 2838 -.007136  .031252 2014  .0016034786
2 2839 -.030801  .008916 2014  .0019395652
2 2840 -.004237  .017491 2014   .023115367
2 2841 -.005319  .009872 2014   .020118285
2 2842 -.024599 -.013027 2014   -.01969013
2 2843  .077851  .051286 2014    .05297193
2 2844 -.039674   .00628 2014    .01051143
2 2845 -.016949 -.000651 2014   .007931211
2 2846  .005388  .010357 2014   .010043426
2 2847 -.016077  .009685 2014   .009287447
2 2848  .042484  .008467 2014    .03806262
2 2849 -.022989 -.015561 2014  -.016197996
2 2850 -.027807 -.019364 2014   -.03049445
2 2851  .034103  .052101 2014    .06173365
2 2852        0  -.00246 2014   .006720366
2 2853  .028723   .01543 2014    .02134069
2 2854 -.031024  .008564 2014   .015910394
2 2855  .163287  .074414 2014    .12965658
2 2856  .051376  .085317 2014     .0424426
2 2857   .02007   .00436 2014    .04687195
2 2858 -.017964   .04234 2014  .0039552194
2 2859  .026132   .01011 2014    .00998237
2 2861 -.032374   .01786 2015  -.028004853
2 2862  .005204  .027828 2015   .024947865
2 2863  .029586 -.001805 2015    .03256724
2 2864 -.057471 -.034765 2015  -.011607916
2 2865 -.073171 -.037525 2015   -.04532814
2 2866  .049342  .044766 2015    .06251788
2 2867  .000784  .016261 2015   .021907527
2 2868 -.001566  .018274 2015  .0091216015
2 2869 -.063529 -.013548 2015  -.014745548
2 2870  .024288  .039701 2015    .03702578
2 2871  .074407   .07656 2015    .09807802
2 2872  .015982  .027218 2015    .06005246
2 2873  .007491  .056553 2015    .05859169
2 2874  .064684  .041905 2015     .0838385
2 2875  .024441  .051511 2015   .006177005
2 2876 -.017042  .031184 2015   .032025207
2 2877  .012483  .008503 2015    .03985193
2 2878 -.054795 -.044991 2015   -.06110151
2 2879  .026087  .034662 2015    .05035169
2 2880  .023305  .093365 2015    .11367215
2 2881 -.009662 -.008247 2015  .0019375472
2 2882  .085017   .10488 2015       .15519
2 2883  -.00578  .034483 2015    .05340284
2 2884 -.124677 -.136604 2015   -.12177646
2 2885 -.008856 -.074094 2015   -.07797858
2 2886  .020104 -.140998 2015   -.19296995
2 2887  .092701  .042479 2015   .017700717
2 2888   .00668  .038188 2015    .10407042
2 2889   .05574  .038027 2015    .08059753
2 2890 -.079818 -.105651 2015   -.07495554
2 2891  .029619  .028003 2015   .016581595
2 2892  .023973  .067808 2015    .12243433
2 2893 -.069565 -.124472 2015    -.1182234
2 2894 -.023724 -.093368 2015   -.09093007
2 2895  .047128 -.040442 2015   -.06174164
2 2896 -.052743  .022072 2015    .06654926
2 2897 -.025984 -.045713 2015  -.065524414
2 2898 -.016768  .000718 2015  .0042858915
2 2899 -.013178 -.009631 2015  -.010624265
2 2900  .039277   .05271 2015    .05561905
2 2901  .031746  .077331 2015    .10001537
2 2902  .006593   .01406 2015    .04423802
2 2903 -.005095 -.007376 2015   -.01986303
2 2904  .039503  .063103 2015    .05355026
2 2905 -.021816  .003349 2015  .0004598762
2 2906  .042446  .022665 2015    .06096356
2 2907 -.015873  -.05515 2015   -.03697226
2 2908  .330996  .029686 2015    .10013583
end
format %tw yw

bys id (yw): gen t  = _n
xtset id t

bys id: gen L1rm = L.rm
bys id: gen L1rind = L.rind

capture program drop myrego
program myrego, eclass
  quietly rego `0'
  tempname r2
  matrix `r2' = e(shapley)
  ereturn post `r2'
  ereturn local cmd = "myrego"
end

statsby _b, by(id year) saving("rego_mi.dta", replace): myrego ri rm L1rm rind L1rind

a. Firstly, the command -rego- is a user-written command, which is used to decompose R-square from a regression into contribution of every single explanatory variable, and can be installed here http://research.uni-leipzig.de/rego/?page_id=109 or by typing

net from http://research.uni-leipzig.de/rego/
net install rego

b. Could you kindly provide some instructions on how to use -rangestat- or -rangerun- to estimate the regression?

[Robert Picard]

I did not mean by(id year)because I defined the interval in terms of the variable year. Both rangestat and rangerun generate separate results for each observation in the dataset. When you specify

rangestat (reg) ri F2rm F1rm rm L1rm L2rm, interval(year 0 0) by(id)

all observations with the same value for year within each id group will have identical results because the subset of observations used is the same. rangestat is optimized to recognize this beforehand and will only compute the statistics once and carry over the results for all other observations where year and id are the same.

Here's how to perform the same task with rangerun. Because rangerun does not know what your program does, the optimization described above is not performed. You have to create a valid interval for a single observation within groups that have the same value for id and year. For the others, I use a value of -1, which creates an invalid interval. If the interval is invalid, the user's program is not run.

capture program drop myrego
program myrego
  rego ri rm L1rm rind L1rind
  matrix my_shapley = e(shapley)
  gen long nobs = e(N)
  gen b_SV1 = my_shapley[1,1]
  gen b_SV2 = my_shapley[1,2]
  gen b_SV3 = my_shapley[1,3]
  gen b_SV4 = my_shapley[1,4]
end

* use -1 for high interval bound for repeat observations within groups
bysort id year (yw): gen high = cond(_n==1, year, -1)

rangerun myrego, interval(year year high) by(id)
list id year b_* if !mi(b_SV1)

[River Huang]

1. My bad. Indeed, I have tried

   interval(year 0 0) by(id)

   and

   interval(year 0 0) by(id year)

   They provide identical results. So you are right.
2. I underestimated the speed of -rangestat-. I re-ran a code and found that it took 906.76 seconds using -statsby- but only 6.86 seconds using -rangestat. Amazing!!!
3. I will try to run your suggested code and see what happens. Many thanks again, Robert.

[River Huang]

Hi, Robert, Thanks again.

1. I tested the code, and found that your code by -rangerun- (325 seconds) is about 60 times faster than mine (18078 seconds).
2. I am (always) confused with the use of interval(keyvar x x). In my two examples above, I want to obtain firm-year-level statistics (whatever they are) using weekly data within a particular year. Why did you suggest interval(year 0 0) in the first example and interval(year year high) in the second one. Is the definition and use of `high' necessary? Can you explain in more details the differences between interval(year 0 0) and interval(year year high)?

[Robert Picard]

Both rangestat and rangerun include very detailed help files with lots of fully worked out examples. I suggest you read them carefully to understand how the interval() option works.

The interval(year 0 0)option specifies a degenerate interval where the lower bound is the same as the upper bound, each set to the value of year for the current observation. So if year is 2016 for the current observation, results will be generated using all observations that fall inrange(year, 2016, 2016). If you also specify by(id), results will be generated using only observations with the same id as well. So for the rangestat call in #2, results for an observation with id==18 and year==2016 are computed using the equivalent of

regress ri F2rm F1rm rm L1rm L2rm if id == 18 & inrange(year, 2016, 2016)

rangestat calculates results for each observation (think of problems with a rolling window of time where the observations to use change for each observation). In your data example in #1, there are 48 observations that meet the above condition. rangestat is smart enough to know that in this case, there's no need to repeat the regression 48 times as the results would be identical. Instead, it performs the task for one observation where the condition is true and carries over the results to the other observations where id == 18 & inrange(year, 2016, 2016).

Since rangerun knows nothing about the content of the user-written program it runs, it cannot be sure that the results will be the same in a situation like the one described in the previous paragraph so rangerun will run the program for each observation in the data with a valid interval. The solution to avoid running the same regression needlessly is to use a valid interval for only one observation per group. To repeat the example in #4:

clear
input long id float yw double(ri rm) float(year rind)
1 2808 -.026122  -.01162 2014   -.02187188
1 2809  -.00922 -.038157 2014   -.01669857
1 2810 -.028765 -.000472 2014  -.014618086
1 2811  .015679  .030798 2014   .007250646
1 2812 -.022298 -.010732 2014  -.005257044
1 2813 -.001754  .007665 2014  -.003436207
1 2814  .061511  .037516 2014   .036805287
1 2815 -.039735 -.001748 2014  -.011532088
1 2816 -.040517 -.028998 2014   -.02259154
1 2817  -.02965  .002966 2014   -.01255455
1 2818 -.044444  -.02618 2014  -.015652513
1 2819  .045543  .020002 2014    .02558933
1 2820 -.000927  -.00901 2014   .012808963
1 2821  .000928  .012285 2014   .004678128
1 2822  .056534  .031577 2014    .04573155
1 2823 -.052632 -.012104 2014  -.025931247
1 2824  .041667 -.034124 2014  -.008572405
1 2825 -.009778 -.005805 2014 -.0018938944
1 2826 -.008079 -.007324 2014  -.003502477
1 2827   .01991  .007051 2014   .007512956
1 2828  .027507  .007389 2014    .00437666
1 2829 -.006908  .005358 2014    .00332471
1 2830 -.010435 -.002495 2014   .002933668
1 2831  .081195  .021431 2014    .02904329
1 2832  -.02668 -.018155 2014   -.01518601
1 2833 -.003046  .015652 2014  -.005131681
1 2834  .009165  .018624 2014   .013686324
1 2835 -.024218 -.001491 2014  -.007349936
1 2836 -.007239  .008263 2014   .007533229
1 2837  .064583  .030152 2014    .04487861
1 2838  .046967  .031252 2014     .0416014
1 2839 -.014953  .008916 2014  -.010414993
1 2840  .003795  .017491 2014    .01047475
1 2841 -.021739  .009872 2014  -.010110507
1 2842 -.009662 -.013027 2014  -.013635258
1 2843   .03122  .051286 2014    .04187739
1 2844 -.018921   .00628 2014   -.01356984
1 2845  .000964 -.000651 2014 -.0006862036
1 2846 -.025048  .010357 2014 .00023193713
1 2847  .001976  .009685 2014  .0006743635
1 2848  .001972  .008467 2014    -.0011938
1 2849 -.003937 -.015561 2014  .0001800596
1 2850  .005929 -.019364 2014  -.006815978
1 2851  .083497  .052101 2014    .05594328
1 2852 -.010879  -.00246 2014  -.004929824
1 2853  .001833   .01543 2014    .06424792
1 2854 -.011894  .008564 2014  -.007100914
1 2855  .151852  .074414 2014     .1581282
1 2856  .168006  .085317 2014     .1850046
1 2857 -.040606   .00436 2014   -.05101458
1 2858   .07604   .04234 2014    .08529846
1 2859  .006667   .01011 2014     .0697964
1 2861  -.04798   .01786 2015  -.009245682
1 2862  .019231  .027828 2015    .05227714
1 2863  -.06311 -.001805 2015   -.02982882
1 2864 -.032639 -.034765 2015   -.07727697
1 2865 -.030151 -.037525 2015   -.03931737
1 2866  .032568  .044766 2015    .04443033
1 2867  .002867  .016261 2015   .004759528
1 2868        0  .018274 2015   .006919141
1 2869 -.039314 -.013548 2015   -.05308905
1 2870  .115327  .039701 2015   .062592335
1 2871  .022015   .07656 2015   .067883536
1 2872 -.015666  .027218 2015  -.011318897
1 2873  .051061  .056553 2015    .04385842
1 2874  .249211  .041905 2015    .06777364
1 2875  .034848  .051511 2015    .04459958
1 2876 -.046663  .031184 2015  -.032021668
1 2877  .034696  .008503 2015    .01202734
1 2878 -.050898 -.044991 2015  -.037017886
1 2879 -.025237  .034662 2015  -.018817553
1 2880  .053074  .093365 2015    .05103419
1 2881  -.05839 -.008247 2015   -.05323207
1 2882  .063969   .10488 2015    .07439438
1 2883  .012883  .034483 2015   .031557143
1 2884  -.11387 -.136604 2015   -.11872815
1 2885 -.058783 -.074094 2015  -.033933423
1 2886 -.050835 -.140998 2015  -.006434457
1 2887  .136955  .042479 2015    .14670287
1 2888 -.069987  .038188 2015   -.05056261
1 2889 -.031838  .038027 2015   -.03295565
1 2890 -.076233 -.105651 2015   -.08608025
1 2891  .020227  .028003 2015  .0045610815
1 2892  .002379  .067808 2015   .007599622
1 2893  -.09019 -.124472 2015   -.09144273
1 2894 -.058261 -.093368 2015   -.04754843
1 2895  .093259 -.040442 2015     .0813361
1 2896 -.074324  .022072 2015   -.04470138
1 2897 -.013686 -.045713 2015 -.0039050495
1 2898 -.024052  .000718 2015   -.00138637
1 2899 -.005687 -.009631 2015   -.01809099
1 2900  .039085   .05271 2015   .032081977
1 2901  .030275  .077331 2015    .03682121
1 2902  .021371   .01406 2015   .014840276
1 2903  -.00959 -.007376 2015  -.008879421
1 2904  .090669  .063103 2015    .09158126
1 2905 -.012107  .003349 2015  -.002904605
1 2906  .025327  .022665 2015   .005304998
1 2907 -.065339  -.05515 2015   -.04647819
1 2908  .033248  .029686 2015    .04298611
2 2808 -.023661  -.01162 2014  -.020101015
2 2809 -.058673 -.038157 2014  -.062758975
2 2810  -.04065 -.000472 2014   -.01064314
2 2811  .100282  .030798 2014    .08124962
2 2812 -.052632 -.010732 2014  -.026290694
2 2813  .004065  .007665 2014   .005708393
2 2814 -.005398  .037516 2014    .04012561
2 2815 -.028494 -.001748 2014  -.011388687
2 2816 -.061453 -.028998 2014   -.05995426
2 2817  .081845  .002966 2014    .02931733
2 2818  .035763  -.02618 2014 -.0011639306
2 2819  .063745  .020002 2014    .05722804
2 2820  .013733  -.00901 2014   .032468516
2 2821 -.001232  .012285 2014    .02583385
2 2822 -.019729  .031577 2014  -.009546324
2 2823 -.021384 -.012104 2014  -.006718597
2 2824  .020566 -.034124 2014  -.006429412
2 2825 -.005038 -.005805 2014  -.019597156
2 2826 -.029114 -.007324 2014   -.03143678
2 2827  .060606  .007051 2014   .018010318
2 2828  .058442  .007389 2014    .02861592
2 2829   .04908  .005358 2014 -.0024844094
2 2830 -.035088 -.002495 2014  -.012149008
2 2831  .021818  .021431 2014    .01079587
2 2832 -.035587 -.018155 2014  -.021365123
2 2833   .02337  .015652 2014   .025344463
2 2834  .018029  .018624 2014    .02538036
2 2835  .040142 -.001491 2014    .01597141
2 2836  .009081  .008263 2014   .031080503
2 2837  .103487  .030152 2014    .05458571
2 2838 -.007136  .031252 2014  .0016034786
2 2839 -.030801  .008916 2014  .0019395652
2 2840 -.004237  .017491 2014   .023115367
2 2841 -.005319  .009872 2014   .020118285
2 2842 -.024599 -.013027 2014   -.01969013
2 2843  .077851  .051286 2014    .05297193
2 2844 -.039674   .00628 2014    .01051143
2 2845 -.016949 -.000651 2014   .007931211
2 2846  .005388  .010357 2014   .010043426
2 2847 -.016077  .009685 2014   .009287447
2 2848  .042484  .008467 2014    .03806262
2 2849 -.022989 -.015561 2014  -.016197996
2 2850 -.027807 -.019364 2014   -.03049445
2 2851  .034103  .052101 2014    .06173365
2 2852        0  -.00246 2014   .006720366
2 2853  .028723   .01543 2014    .02134069
2 2854 -.031024  .008564 2014   .015910394
2 2855  .163287  .074414 2014    .12965658
2 2856  .051376  .085317 2014     .0424426
2 2857   .02007   .00436 2014    .04687195
2 2858 -.017964   .04234 2014  .0039552194
2 2859  .026132   .01011 2014    .00998237
2 2861 -.032374   .01786 2015  -.028004853
2 2862  .005204  .027828 2015   .024947865
2 2863  .029586 -.001805 2015    .03256724
2 2864 -.057471 -.034765 2015  -.011607916
2 2865 -.073171 -.037525 2015   -.04532814
2 2866  .049342  .044766 2015    .06251788
2 2867  .000784  .016261 2015   .021907527
2 2868 -.001566  .018274 2015  .0091216015
2 2869 -.063529 -.013548 2015  -.014745548
2 2870  .024288  .039701 2015    .03702578
2 2871  .074407   .07656 2015    .09807802
2 2872  .015982  .027218 2015    .06005246
2 2873  .007491  .056553 2015    .05859169
2 2874  .064684  .041905 2015     .0838385
2 2875  .024441  .051511 2015   .006177005
2 2876 -.017042  .031184 2015   .032025207
2 2877  .012483  .008503 2015    .03985193
2 2878 -.054795 -.044991 2015   -.06110151
2 2879  .026087  .034662 2015    .05035169
2 2880  .023305  .093365 2015    .11367215
2 2881 -.009662 -.008247 2015  .0019375472
2 2882  .085017   .10488 2015       .15519
2 2883  -.00578  .034483 2015    .05340284
2 2884 -.124677 -.136604 2015   -.12177646
2 2885 -.008856 -.074094 2015   -.07797858
2 2886  .020104 -.140998 2015   -.19296995
2 2887  .092701  .042479 2015   .017700717
2 2888   .00668  .038188 2015    .10407042
2 2889   .05574  .038027 2015    .08059753
2 2890 -.079818 -.105651 2015   -.07495554
2 2891  .029619  .028003 2015   .016581595
2 2892  .023973  .067808 2015    .12243433
2 2893 -.069565 -.124472 2015    -.1182234
2 2894 -.023724 -.093368 2015   -.09093007
2 2895  .047128 -.040442 2015   -.06174164
2 2896 -.052743  .022072 2015    .06654926
2 2897 -.025984 -.045713 2015  -.065524414
2 2898 -.016768  .000718 2015  .0042858915
2 2899 -.013178 -.009631 2015  -.010624265
2 2900  .039277   .05271 2015    .05561905
2 2901  .031746  .077331 2015    .10001537
2 2902  .006593   .01406 2015    .04423802
2 2903 -.005095 -.007376 2015   -.01986303
2 2904  .039503  .063103 2015    .05355026
2 2905 -.021816  .003349 2015  .0004598762
2 2906  .042446  .022665 2015    .06096356
2 2907 -.015873  -.05515 2015   -.03697226
2 2908  .330996  .029686 2015    .10013583
end
format %tw yw

bys id (yw): gen t  = _n
xtset id t

bys id: gen L1rm = L.rm
bys id: gen L1rind = L.rind

capture program drop myrego
program myrego
  rego ri rm L1rm rind L1rind
  matrix my_shapley = e(shapley)
  gen long nobs = e(N)
  gen b_SV1 = my_shapley[1,1]
  gen b_SV2 = my_shapley[1,2]
  gen b_SV3 = my_shapley[1,3]
  gen b_SV4 = my_shapley[1,4]
end

* use -1 for high interval bound for repeat observations within groups
bysort id year (yw): gen high = cond(_n==1, year, -1)

rangerun myrego, interval(year year high) by(id)

by id year: gen first5 = _n <= 5
list id year high b_* if first5

and the results:

. list id year high b_* if first5

     +--------------------------------------------------------------+
     | id   year   high      b_SV1      b_SV2      b_SV3      b_SV4 |
     |--------------------------------------------------------------|
  1. |  1   2014   2014   .3231212   .0037154   .4807238   .0028451 |
  2. |  1   2014     -1          .          .          .          . |
  3. |  1   2014     -1          .          .          .          . |
  4. |  1   2014     -1          .          .          .          . |
  5. |  1   2014     -1          .          .          .          . |
     |--------------------------------------------------------------|
 53. |  1   2015   2015   .1794958   .0149125   .5592141   .0176887 |
 54. |  1   2015     -1          .          .          .          . |
 55. |  1   2015     -1          .          .          .          . |
 56. |  1   2015     -1          .          .          .          . |
 57. |  1   2015     -1          .          .          .          . |
     |--------------------------------------------------------------|
101. |  2   2014   2014   .1681696    .015967   .5335397   .0063831 |
102. |  2   2014     -1          .          .          .          . |
103. |  2   2014     -1          .          .          .          . |
104. |  2   2014     -1          .          .          .          . |
105. |  2   2014     -1          .          .          .          . |
     |--------------------------------------------------------------|
153. |  2   2015   2015   .1414909   .0708542   .1937031   .0445665 |
154. |  2   2015     -1          .          .          .          . |
155. |  2   2015     -1          .          .          .          . |
156. |  2   2015     -1          .          .          .          . |
157. |  2   2015     -1          .          .          .          . |
     +--------------------------------------------------------------+

The upper bound is set to -1 for all observations except the first one in each group. With only 4 observations with a valid interval, the myrego program is called only 4 times. You can easily check what rangerun does by reloading the data and reducing to one group, say for id==2 & year==2015:

keep if id == 2 & year == 2015
myrego
keep in l  // keep the last observation
list id year b_*

and the results

. keep if id == 2 & year == 2015
(152 observations deleted)

. myrego
 
------------------------------------------------------------------------------
Gr        Regressor |       Coef.      Std.Err.   P>|t|  Std.Coef. Shapley %R2
--------------------+---------------------------------------------------------
 1               rm |    -.043807      .3583234   0.903    -0.0389     31.3995
 2             L1rm |   -.8751579 **   .3988247   0.034    -0.7755     15.7239
 3             rind |    .6679408 **   .2933587   0.028     0.7274     42.9864
 4           L1rind |    .3636653      .3234948   0.267     0.3899      9.8902
 -        Intercept |   -.0015966      .0082622   0.848     
--------------------+---------------------------------------------------------
       Observations |          48
         Overall R2 |     0.45061
           Root MSE |    .0505198
      F-stat. Model |    8.817324 ***             0.000
     Log Likelihood |    77.82967
------------------------------------------------------------------------------

. keep in l  // keep the last observation
(47 observations deleted)

. list id year b_*

     +-------------------------------------------------------+
     | id   year      b_SV1      b_SV2      b_SV3      b_SV4 |
     |-------------------------------------------------------|
  1. |  2   2015   .1414909   .0708542   .1937031   .0445665 |
     +-------------------------------------------------------+

.

[River Huang]

Dear Robert, many thanks for the detailed description. Indeed, I have gone through the help files of -rangestat- and -rangerun-. According to your explanation above, I am now understanding more of the usage of -interval-. However, could you explain the role of the second `year' in the command: rangerun myrego, interval(year year high) by(id).

[Robert Picard]

From the help file:

interval(keyvar low high) is required and defines the interval that selects the set of observations to use to calculate
results for the current observation. keyvar is a numeric variable. Observations whose values for keyvar fall within
the closed interval bounds are selected. low and high can each be specified using a numeric variable, a # (a number in
Stata parlance), or a system missing value. If a # is used, the bound for each observation is computed by adding # to
keyvar. If low is specified using a system missing value, low is set to missing for all observations. rangerun applies
the same rules as inrange() for missing bounds: if the lower bound is missing, observations will match up to and
including the value of high. If both low and high are missing, all observations will match.

So the value of the variable year for the current observation is the low bound.

[River Huang]

Thanks again for your answer.

[River Huang]

Hi, Robert, I have another question with data (two countries here, but more than 150 countries in the real data set) as:

* Example generated by -dataex-. To install: ssc install dataex
clear
input str3 countrycode int year float(rgdpna rconna)
"ABW" 1970  345.5351  254.4331
"ABW" 1971  376.9389  277.5572
"ABW" 1972   411.197 302.78293
"ABW" 1973  448.5685  330.3014
"ABW" 1974  489.3366  360.3205
"ABW" 1975  533.8098 393.06845
"ABW" 1976   582.325  428.7926
"ABW" 1977  635.2495 467.76175
"ABW" 1978   692.984 510.27695
"ABW" 1979  755.9658   556.653
"ABW" 1980  824.6715  607.2352
"ABW" 1981  899.6216  662.4445
"ABW" 1982  981.3835  722.6376
"ABW" 1983 1070.5764  788.2687
"ABW" 1984 1167.8755  860.0565
"ABW" 1985 1274.0176  938.0665
"ABW" 1986 1389.8064 1023.1405
"ABW" 1987 1613.2655  1188.661
"ABW" 1988 1914.1178 1408.6727
"ABW" 1989  2146.297 1579.2063
"ABW" 1990 2231.3206  1648.263
"ABW" 1991 2408.9978 1766.6312
"ABW" 1992 2550.7036 1875.5323
"ABW" 1993  2737.101 2045.8425
"ABW" 1994 2961.6504 2125.0317
"ABW" 1995 3035.8916 2250.0024
"ABW" 1996  3073.691 2409.3872
"ABW" 1997 3313.8826 2532.8706
"ABW" 1998  3534.807 2613.8496
"ABW" 1999  3575.615  2643.374
"ABW" 2000 3737.0205  2697.451
"ABW" 2001  3626.163 2821.9255
"ABW" 2002  3506.925  2879.548
"ABW" 2003 3577.1304  2929.657
"ABW" 2004  3860.181  2984.166
"ABW" 2005  3906.984  3045.632
"ABW" 2006  3948.216  3107.141
"ABW" 2007  4028.451  3236.897
"ABW" 2008 4034.0225 3237.6514
"ABW" 2009 3577.1304  3105.971
"ABW" 2010  3456.778  3058.898
"ABW" 2011  3579.359  3031.459
"ABW" 2012 3529.2124 3094.9575
"ABW" 2013 3694.1394  3221.636
"ABW" 2014    3737.6  3158.193
"AGO" 1970  49259.54  23016.14
"AGO" 1971  52215.11   24382.1
"AGO" 1972  52319.54  24440.83
"AGO" 1973  56557.42  26460.19
"AGO" 1974   58310.7 27304.645
"AGO" 1975  55395.16 25814.354
"AGO" 1976  52514.62 24446.496
"AGO" 1977  52934.73   24778.6
"AGO" 1978  50129.19   23628.5
"AGO" 1979  50279.58  23648.28
"AGO" 1980  51486.29  23410.99
"AGO" 1981  49220.89  23832.29
"AGO" 1982  49220.89 24121.344
"AGO" 1983  51288.17  25853.99
"AGO" 1984  54365.46  26479.09
"AGO" 1985  56268.25 28384.533
"AGO" 1986  58055.14  27515.77
"AGO" 1987  62119.52 27535.326
"AGO" 1988  65581.37  28156.63
"AGO" 1989 65835.195  28986.05
"AGO" 1990  65542.75 28555.596
"AGO" 1991  66192.51  23902.82
"AGO" 1992  62328.01  21396.77
"AGO" 1993  47379.62 13829.438
"AGO" 1994  48014.21 16557.697
"AGO" 1995  55216.34  19968.58
"AGO" 1996  62695.04 13061.095
"AGO" 1997  67255.65  26280.48
"AGO" 1998  70410.71 23454.707
"AGO" 1999  71946.72 30304.975
"AGO" 2000 74144.414 22939.273
"AGO" 2001 76472.555  29491.41
"AGO" 2002  87584.02  30811.45
"AGO" 2003  90474.29   42849.7
"AGO" 2004 100383.81  44309.47
"AGO" 2005 115470.43  46957.06
"AGO" 2006  128804.6  50507.54
"AGO" 2007 146840.63  53958.07
"AGO" 2008 163236.88  59145.45
"AGO" 2009 166732.53  62854.73
"AGO" 2010  172672.2 67384.266
"AGO" 2011    175880  73934.58
"AGO" 2012  189307.5 77311.805
"AGO" 2013 197281.34  84326.98
"AGO" 2014 206743.03  91807.53
end

I want to regress `rconna' on `rgdpna' (for each country and use observations for all years) with the following command:

rangestat (reg) rconna rgdpna, interval(year 0 0) by(countrycode)

but got error message (in red)

no result for all obs: reg rconna rgdpna

I also tried

rangestat (reg) rconna rgdpna, interval(year year .) by(countrycode)

but found the estimates of `rgdpna' varies over yea. which is supposed to be a constant for a particular country.

[Robert Picard]

You have only one observation per year per country. If you use interval(year 0 0), the lower and upper bounds of the interval are the value of year for the current observation. Within groups of observations with the same countrycode, there is only one observation that falls within the specified interval and you cannot run a regression with just one observation. The second version you tried puts the lower bound at the value of year for the current observation. So if the current observation is in 2006, you performed a regression using data from a given country from 2006 and higher. What you want is

rangestat (reg) rconna rgdpna, interval(year . .) by(countrycode)

[River Huang]

Got it, and thanks a lot. By the way, how can I obtain the t- as well as p-values for the coefficients in this case? I guess that we have to rely upon -rangerun- rather than -rangestat-.

[Robert Picard]

Both t-stats and p-values can be calculated from information returned by rangestat. Note that rangestat will not return results if a variable is omitted so the degrees of freedom in your example is the number of observations minus 2 for all regressions.

rangestat (reg) rconna rgdpna, interval(year . .) by(countrycode)

gen df = reg_nobs - 2
foreach v in rgdpna cons {
    gen double t_`v' = b_`v' / se_`v'
    gen double p_`v' = 2 * ttail(df, abs(t_`v'))
}

list if countrycode == "AGO" & year == 2014

reg rconna rgdpna if countrycode == "AGO"

[River Huang]

Dear Robert, thank you so much.