compare and test "r2_a" across two regressions

[Mo Gad]

Hi everyone!

I want to compare adjusted R² across two models. To be more specific, assume that using auto data, I want to test the differences in the ability of weight to explain the price for domestic cars compared with that of foreign cars. I want to follow the steps below:

(A) get the actual difference in R2. This should be straightforward as follows: (The observed difference is 33% )

sysuse auto, clear
regress price weight if foreign==1
regress price weight if foreign==0

is the observed difference higher than would be explained by a random variable? So I want to do an approximate randomization test as follows:

(B) shuffle the foreign dummy variable randomly across all cars.
(C) re-estimate the regressions from step A (both domestic and foreign cars) based on the shuffled variable constructed in step B and record the difference in R² values from these regressions,
(D) note if the difference in R² from the randomly generated samples is larger than the observed difference in R² of 33%.
Repeat the steps (B to C) 1000 times and the get p-value as the number of times that the randomly generated difference in R² is larger than the actual/observed difference in R² divided by the number of iterations.



Please see my try below:

sysuse auto, clear
regress price weight if foreign==1
regress price weight if foreign==0
local obs_r2=`e(r2)'
tempname compare
tempfile results
postfile `compare' r2_1 r2_2 using "`results'"
forvalues i=1/1000{
shufflevar foreign
quietly regress price weight if foreign_shuffled==1
local topost `topost' (`e(r2_a)')
quietly regress price weight if foreign_shuffled==0
local topost `topost' (`e(r2_a)')
}

postclose `compare'

use "`results'", clear
sum r2_1 r2_2
count if r2_2- r2_1>0.33

Will be great if anyone could help why this is not working?

Thanks!

[Carlo Lazzaro]

Mo:
I receive the following error message:

command shufflevar is unrecognized
r(199);

and

search shufflevar

gives back no entry.
Is it a built-in/community-contributed command for Stata 17?.

[Felix Bittmann]

Yes this should be on SSC, but maybe not for version 17?

sysuse auto, clear
regress price weight if foreign==1
regress price weight if foreign==0
local obs_r2=`e(r2)'
tempname compare
tempfile results
postfile `compare' r2_1 r2_2 using `results'
forvalues i=1/1000{
    shufflevar foreign
    quietly regress price weight if foreign_shuffled==1
    local a `e(r2_a)'
    quietly regress price weight if foreign_shuffled==0
    local b `e(r2_a)'
    drop foreign_shuffled
    post `compare' (`a') (`b')    
}

postclose `compare'

use `results', clear
sum r2_1 r2_2
count if r2_2- r2_1>0.33

[Mo Gad]

Thank you...Carlo Lazzaro and Felix Bittmann!

Felix is right shufflevar is on SSC (see below), but I am using Stata16.

ssc install shufflevar

Felix Bittmann the code worked nicely, many thanks!!