Svy bootstrap - last estimates not found

[Luísa de Azevedo]

Dear all,

I am attempting to write a program to get the bootstraped standard errors of a generated regressor (the predicted wages are estimated by a linear regression in a first stage and then used in a probit model in a second).

It works fine when I use the standard bootstrap command, however when I try to account for the fact that I am using survey data I get the following error message: "last estimates not found".

Why doesn't Stata recognize the results stored in matrices b and V?

I read a very similar post in the old list, but it never came to a conclusion.

Would anyone kindly help me, please?

Bellow is the program I am trying to run:

svyset metapsu [pw=peso], strata(metastrat) singleunit(centered)
bsweights bw, reps(500) n(0) seed(9999)

program define ajuste, eclass properties(svyb)
syntax [pw iw] [if] [, NULLOPT]
svy: reg lsal_hora brancos negros amarelos mulher EF_i EF_c EM_i EM_c ES_i ES_c idade idade2 filtro_conj urbana metropol
predict yhat
svy: probit trabnr_filtro yhat lrenda_nao_trab1 lrenda_dom_out mulher educ idade idade2 brancos filtro_conj n_pes_com_rend metropol
matrix b = e(b)
matrix V = e(V)
ereturn post b V
end

ajuste
drop yhat
svy bootstrap _b, brsweight(bw*): ajuste

Thank you,
Luísa

[Jeff Pitblado (StataCorp)]

I can see a few things wrong with your code.

1. You need to svyset the resampling weights.

2. The call to ajuste looks like replay syntax to svy.
This means that svy thinks that you want it to replay
estimation results instead of perform the estimation. To get around
this, just add a required argument to ajuste.

3. ajuste should not be using the svy prefix if you want
it to work properly with svy bootstrap. Thus you will need to
pass the specified weights and if condition directly to
regress and probit.

4. brsweight() is not an option of svy bootstrap.

Here is how I would change your code:

* ASSUMPTION: 2-stage sampling, no FPC in the first stage
svyset metapsu [pw=peso], strata(metastrat) singleunit(centered)

* ASSUMPTION: using Kolenikov's tool for generating bootstrap
* replicate weights from the current -svyset-tings
bsweights bw, reps(500) n(0) seed(9999)

* need the original sampling weights and replicate weights for the
* call to -svy bootstrap-
svyset [pw=peso], bsrweight(bw*) singleunit(centered)

* ASSUMPTION: user-written estimation command to be used with
* -svy bootstrap-; must support weights and -if- conditions; also
* needs to require an argument to prevent -svy bootstrap- from
* confusing the estimation syntax with a replay syntax
program define ajuste
        syntax anything [pw iw] [if] [, NULLOPT]
        * NOTE: "anything" is required, but we ignore its contents

        * NOTE: linear regression, using the specified weights and
        * -if- condition; you cannot use -svy- prefix in here if you
        * want this command to work properly with -svy bootstrap-
        regress lsal_hora       brancos             ///
                                negros              ///
                                amarelos            ///
                                mulher              ///
                                EF_i                ///
                                EF_c                ///
                                EM_i                ///
                                EM_c                ///
                                ES_i                ///
                                ES_c                ///
                                idade               ///
                                idade2              ///
                                filtro_conj         ///
                                urbana              ///
                                metropol            ///
                                [`weight'`exp'] `if'

        * NOTE: compute the linear prediction, but make sure our
        * variable doesn't already exist
        capture drop yhat
        predict yhat

        * NOTE: probit regression, using the specified weights and
        * -if- condition; you cannot use -svy- prefix in here if you
        * want this command to work properly with -svy bootstrap-
        probit trabnr_filtro    yhat                ///
                                lrenda_nao_trab1    ///
                                lrenda_dom_out      ///
                                mulher              ///
                                educ                ///
                                idade               ///
                                idade2              ///
                                brancos             ///
                                filtro_conj         ///
                                n_pes_com_rend      ///
                                metropol            ///
                                [`weight'`exp'] `if'
end

* use "now" for the -anything- argument
ajuste now [pw=peso]
svy bootstrap _b: ajuste now

[Luísa de Azevedo]

Dear Jeff,
Thank you so much for your response.
The program worked out fine with the adjustments you suggested.
Apart from the error using the weights with svy bootstrap instead of svyseting them, I guess my mistake was trying to write a program that would give me both the uncorrected (accounting for the fact that I am using survey data, though) and bootstraped standard errors. Maybe I can run the first and second stage with svy separately to get the uncorrected standard errors I would like to compare to the bootstraped ones.
Best,
Luísa

[Steve Samuels]

Jeff's NOTE about dropping yhat

* NOTE: compute the linear prediction, but make sure our
* variable doesn't already exist
capture drop yhat
predict yhat

corrected a crucial (and common) error in attempts to write bootstrapped programs: creation of a new variable with generate or predict. The code will work when the program is run alone and for the first bootstrapped sample, but will fail for later samples because the new variable already exists. Fixes: drop the variable, as here, or use temporary variables

[Luísa de Azevedo]

Dear Jeff,

With your help I was able to succesfully estimate the bootstraped standard errors of the regression coefficients.

However, to analyse the results I also need the bootstraped standard errors of the marginal effects, which I couldn't obtain.

Margins, dydx(*) - with the adjustment in the dummy and interaction variables - does not work at all. All attempts resulted in "estimates post: matrix has missing values". I've already used the subpopulation option to restrict the calculation of the marginal effects to the observations that do not have missing values on the variables included in the regression. I've even tried using the options force and noestimcheck.

Mfx, on the other hand, goes a little further. First, I've just added it after the probit regression, but got the following error: "insuficient observations to compute bootstrap standard errors, no results will be saved".

I then ran the program noisily and learned that Stata was having problems handling negative weights.
http://www.stata.com/support/faqs/st...-zero-weights/ recommends the use of iweghts instead of pweights to deal with this kind of situation, but it didn't work either.

Would you be willing to help me again, please?

Thank you,
Luísa

[Jeff Pitblado (StataCorp)]

Your going to have to be more specific.

With respect to which variables and for what prediction do you want marginal effects?

Perhaps you can show us the Stata code you used, because

margins, dydx(*)

might produce something after svy bootstrap, but I imagine you are trying
to use margins within your ajuste program.

[skolenik]

bsweights with n(0) will produce negative weights. If the only issue are negative weights, then you need to specify something like n(-1) to make sure all replicate weights are non-negative. This was documented in my Stata Journal paper (http://stata-journal.com/article.html?article=st0187), and I see no good reason to use n(0) number of bootstrap samples.

[Luísa de Azevedo]

Dear Jeff and Stas,

Your support is highly appreciated.

It took me a few days to answer because I was putting your advice into practice and bsweights takes a while to run.

Yes, I am using margins within ajuste. I just added it after the probit regression in the above code, suggested by you. I followed example 4 of svy postestimation manual, so also included the set buildfvinfo on command (before the probit regression) and the option post of margins.

After I corrected the issue of the negative weights (thank you very much, Stas), margins dydx(*) atmeans, post worked out fine.

However, I want the average marginal effects of all covariates rather than the marginal effects at the average of the covariates.

I guess the error "estimates post: matrix has missing values" does not have to do with program ajuste, because it appears even if I try to run margins dydx(*) after svy bootstrap.

There is something I've omitted before (I had thought it was irrelevant) that may be related to this error: I am using the option subpop after svy and margins.

Thank you,
Luísa

[Pedro Queiroz]

Dear Jeff,

I am trying to run the same code you made available here since I am estimating equations similar to Luísa's. However, when I run svy bootstrap, it says: unknown weight type. It may be a quite simple mistake I am making on the code, do you have any idea of what I may be doing wrong? I have no problems in running the ajuste, only svy bootstrap. I'd much appreciate any kind help with that.

Thank you,
Pedro

[Jeff Pitblado (StataCorp)]

It is hard to saying anything definitive here without seeing what you are doing.

[Pedro Queiroz]

I did exactly as in your code, adapting for my variables. When I run svy bootstrap it says "unknown weight type".

Here's my do file, I followed your code:

svyset psu [pw=FATOR_EXPANSAO2], strata(estrato) singleunit(centered)

bsweights bw, reps(500) n(0) seed(9999)

svyset [pw=FATOR_EXPANSAO2], bsrweight(bw*) singleunit(centered)

***FATOR_EXPANSAO2 is my weight variable

program define ajuste
syntax anything [pw iw] [if] [, NULLOPT]

* NOTE: linear regression, using the specified weights and
* -if- condition; you cannot use -svy- prefix in here if you
* want this command to work properly with -svy bootstrap-
regress rend_mulher_pc rdosmtpc ///
crianca ///
adolescente1 ///
idoso1 ///
total_pes ///
idademulher ///
escolaridade_mulher1 ///
norte ///
nordeste ///
sul ///
centro_oeste ///
idade2 ///
rural ///
metropolitano ///
fd1 ///
[`FATOR_EXPANSAO2'`exp'] `if'

capture drop yhat
predict yhat

* NOTE: probit regression, using the specified weights and
* -if- condition; you cannot use -svy- prefix in here if you
* want this command to work properly with -svy bootstrap-
probit d1 yhat ///
lnrenda ///
mulher_trab ///
domestica1 ///
escolaridade_mulher1 ///
anosestudo ///
norte ///
nordeste ///
sul ///
centro_oeste ///
idade ///
rural ///
metropolitano ///
multiplos_adultos ///
mae_solteira ///
sozinho ///
prep ///
[`FATOR_EXPANSAO2'`exp'] `if'
end

ajuste now [FATOR_EXPANSAO2]
svy bootstrap _b: ajuste now

[Steve Samuels]

Pedro,

It is not a good idea ordinarily to ask a new question in a thread about a different question. How is anyone to know that it is new question, and not just a reply to the old one. Here I will answer, as I think the initial problem is simple. If I am wrong, or if you have further questions, I suggest that you start a new thread and link to this one.

You don't show us the exact commands and results in sequence as requested in FAQ Section 12.

Say exactly what you typed and exactly what Stata typed (or did) in response. N.B. exactly!

I think you made a minor mistake in copying Jeff's code. It looks to me like the error message "unknown weight type" was generated by the statement

ajuste now [FATOR_EXPANSAO2]

and not by the svy bootstrap command. The correct version would be:

ajuste now [pw = FATOR_EXPANSAO2]

Stata code and output are much more readable if they are enclosed in CODE delimiters. Press the "A" (for "Advanced editor") in the upper right of the edit window, then click the "#" symbol. See: http://www.statalist.org/forums/help...ntent_advanced.

Steve

[Pedro Queiroz]

Thank You Steve!

Next time I will add a new post as a new question. I will try what you suggested and see what happens. I appreciate your help!

Pedro

[Lucy Bilaver]

Could Jeff have left the svy commands in the original "adjuste" program and then estimated the SEs with bootstrap instead of svy bootstrap? Would this have been incorrect?

[Lukas Lang]

Hi,

I am resuming this post because I cannot replicate successfully Jeff's code in #2.

These are the codes I am running:

svyset psu [pw=w], strata(strata)
bsweights bw, reps(100) n(-1) dots seed(0123456789)
svyset psu [pw=w], strata(strata) bsrweight(bw*)

capture program drop myboot
program define myboot
    syntax anything [pw iw] [if] [, NULLOPT]
    reg yvar xvar ${controls} [aw=w]
end

qui myboot now [pw=w]

svy bootstrap _b[xvar]: myboot now

The last command returns a table with observed coefficient but no standard errors:

------------------------------------------------------------------------------
             |   Observed   Bootstrap                         Normal-based
             | coefficient  std. err.      z    P>|z|     [95% conf. interval]
-------------+----------------------------------------------------------------
       _bs_1 |   1.094125          .        .       .            .           .
------------------------------------------------------------------------------

If I run exactly the same command but in the following form, then it works:

svy bootstrap _b[xvar]: reg yvar xvar ${controls}

I added the noise option to figure out what could be happening and it seems that the replications of the code

svy bootstrap _b[xvar]: myboot now

return always the same results, like the same bootstrap sample is used all the times.

If I then change the program as follows:

capture program drop myboot
program define myboot
    bsample
    syntax anything [pw iw] [if] [, NULLOPT]
    reg yvar xvar ${controls} [aw=w]
end

Then bootstrapped standard errors are estimated but the two commands

svy bootstrap _b[xvar]: myboot now
svy bootstrap _b[xvar]: reg yvar xvar ${controls}

return different results which is not reassuring.

How is that possible?