Saving intermediate matrices from a loop using post?

[Sam Bras]

Dear Stata community,

I want to save matrices from each iteration of an r-class program which I am looping to do Monte-Carlo simulations. I have already written a program in which I save the matrices in a cumulative matrix, which works fine if you are not building very large matrices. However, now I am doing 10.000 loops and my program has become very slow, too slow to be useful actually. To solve this issue I read about the -post- command and tried to implement it, however I cannot figure out how to store matrices with -post- is this even possible? It works fine for scalars and the program is very fast that way, but I need to store matrices as well.

Please find a piece of my code below:

forvalues i=1/`nreps' {
            preserve
                bsample `bootstrap_period'
                display _newline(2) `i'
                capture noisily ovport `developed_equities' `emerging_equities' `developed_gov_bond' `emerging_gov_bond' `developed_corp_bond' `real_estate' , casewise nport(`portfolios') noshort rfrate(`rf1')
                
                matrix this_run_weights = r(weights)                                    // Optimal weight for each asset class under tangency with the efficien frontier
                matrix colnames this_run_weights = run_`i'
                matrix cumu_weights_raw = nullmat(cumu_weights_raw),this_run_weights

[Mike Lacy]

At least for me, none of what you posted is visible after the line starting with "matrix cumu_weights_raw" I suspect it will look this way on others' screens, including yours. Presuming this is right, you'll want to repost your code, and make sure it is readable ("preview") before finishing your posting to the list. Also, using shorter lines, which do not require scrolling to the right to make them visible, would make your code easier for others to read and digest. Finally, knowing something of the dimensions of your matrices, and how you tried to -post- them would help, too.

While I'm not certain I would know the answer to your problem, these are things that would help me understand it well enough to know whether I could.

[Sam Bras]

Thank you for your response Mike. Sorry for not posting the full code, I presumed it wouldn't be necessary since I just want to figure out how to -post- a matrix.

To answer your question, the matrices are quite large as I'm doing 10.000 reps.

If I run my current code it would take several hours, I think because it is building the very large matrices. I know how to post individual scalars by writing

postfile run_rf1 expected_return using results, replace

before the loop and:

post run_rf1 (r(rop))

in the loop. However, how can I post a matrix in the postfile?

* Setting a large matrix size in order to accommodate large number of iterations
set matsize 10100

* Drawing samples and running the ovport program over and over and cumulate the results in a matrix:
* Set-up the model here:
*     Amount of simulations:
    local nreps 10000
*     Amount of portfolios on efficient frontier:
    local portfolios 100
*    Amount of months for bootstrap
    local bootstrap_period 60
*    Seed to replicate results:
    set seed 12345

* Set-up portfolio and economic rules
    * Indices that go into the portfolio, this allows you to change the indices that are used for each asset class.
    local developed_equities        r_his_eq_msci_world_1
    local emerging_equities            r_his_eq_msci_em_1
    local developed_gov_bond        r_his_gbond_ftse_world_1
    local emerging_gov_bond             r_his_embond_global_1
    local developed_corp_bond         r_his_cbond_bofa_global_1
    local real_estate                r_his_re_spprop_dev_1
    
    * Naming of the indices, make sure the bellow and the bottom match!
    local name_developed_equities    MSCI_World
    local name_emerging_equities    MSCI_Emerging_Markets
    local name_developed_gov_bond    FTSE_Dev_Gov_Bonds
    local name_emerging_gov_bond    EMBI_Emerging_Gov_Bonds
    local name_developed_corp_bond    BofA_Dev_Gov_Bonds
    local name_real_estate            SP_Developed_Property
    
    * Set up the various interest rates to for which to run the portfolios    
    local rf1 = (ln(1-0.01)/12)
    local rf2 = (ln(1-0.005)/12)
    local rf3 = (ln(1+0)/12)
    local rf4 = (ln(1+0.01)/12)
    local rf5 = (ln(1+0.02)/12)
    local rf6 = (ln(1+0.03)/12)
    local rf7 = (ln(1+0.04)/12)        
    local rf8 = (ln(1+0.05)/12)

timer on 1

{
        {
        forvalues i=1/`nreps' {
            preserve
                bsample `bootstrap_period'
                display _newline(2) `i'
                capture noisily ovport `developed_equities' `emerging_equities' `developed_gov_bond' ///
                `emerging_gov_bond' `developed_corp_bond' `real_estate' , casewise nport(`portfolios') noshort rfrate(`rf1')
                
                matrix this_run_weights = r(weights)                                    
                // Optimal weight for each asset class under tangency with the efficien frontier
                matrix colnames this_run_weights = run_`i'
                matrix cumu_weights_raw = nullmat(cumu_weights_raw),this_run_weights
                
                matrix this_run_return = r(rop)                                            
                // Expected return of the portfolio
                matrix rownames this_run_return = run_`i'
                matrix cumu_return = nullmat(cumu_return)\this_run_return
                
                matrix this_run_stdev = r(sdop)                                            
                // Expected standard deviation of the portfolio
                matrix rownames this_run_stdev = run_`i'
                matrix cumu_stdev = nullmat(cumu_stdev)\this_run_stdev
                
                matrix this_run_variance = r(varop)                                        
                // Expected variance of the portfolio
                matrix rownames this_run_variance = run_`i'
                matrix cumu_variance = nullmat(cumu_variance)\this_run_variance
                
                matrix this_run_sharper = r(sharper)                                    
                // Sharpe ratio of the portfolio
                matrix rownames this_run_sharper = run_`i'
                matrix cumu_sharper = nullmat(cumu_sharper)\this_run_sharper
                
                matrix this_run_exprets = r(exprets)                                    
                //Expected return for each asset class
                matrix colnames this_run_exprets = run_`i'
                matrix cumu_exprets_raw = nullmat(cumu_exprets_raw),this_run_exprets
                
                matrix this_run_mcr = r(mcr)                                            
                // Asset marginal contributions to portfolio risk.  Risk decomposition is performed using the Euler's theorem, 
                // so the marginal contributions to risk is a set of partial derivatives of portfolio risk with respect 
                // to each asset weight.
                matrix colnames this_run_mcr = run_`i'                                    
                
                matrix cumu_mcr_raw = nullmat(cumu_mcr_raw),this_run_mcr                
                
                matrix this_run_cr = r(cr)                                                
                // Asset contributions to portfolio risk. This is equal to the marginal contributions multiplied by its 
                // respective weights. The sum of the elements of this vector is equal to the
                // portfolio risk (portfolio standard deviation).
                matrix colnames this_run_cr = run_`i'                                    
                matrix cumu_cr_raw = nullmat(cumu_cr_raw),this_run_cr                    
                
                matrix this_run_pcr = r(pcr)                                            
                // Decomposition of risk: vector of asset percent contributions to portfolio risk.
                // This is equal to the contributions divided by portfolio risk.  The sum of
                // the elements of this vector is equal to one.
                matrix colnames this_run_pcr = run_`i'                                    
                matrix cumu_pcr_raw = nullmat(cumu_pcr_raw),this_run_pcr                
                
                matrix this_run_betas = r(betas)                                        
                // Asset betas with respect to the portfolio. An asset beta is defined as the
                // covariance between the asset returns and the portfolio returns divided by
                // the portfolio variance.
                matrix colnames this_run_betas = run_`i'                                
                matrix cumu_betas_raw = nullmat(cumu_betas_raw),this_run_betas            
                            
            restore
            }
        matrix rownames cumu_weights_raw = Weight_`name_developed_equities' Weight_`name_emerging_equities' ///
        Weight_`name_developed_gov_bond' Weight_`name_emerging_gov_bond' Weight_`name_developed_corp_bond' Weight_`name_real_estate'
        matrix cumu_weights = cumu_weights_raw'
        
        matrix rownames cumu_mcr_raw = mcr_`name_developed_equities' mcr_`name_emerging_equities' ///
        mcr_`name_developed_gov_bond' mcr_`name_emerging_gov_bond' mcr_`name_developed_corp_bond' mcr_`name_real_estate'
        matrix cumu_mcr = cumu_mcr_raw'
        
        matrix rownames cumu_cr_raw = cr_`name_developed_equities' cr_`name_emerging_equities' ///
        cr_`name_developed_gov_bond' cr_`name_emerging_gov_bond' cr_`name_developed_corp_bond' cr_`name_real_estate'
        matrix cumu_cr = cumu_cr_raw'
        
        matrix rownames cumu_pcr_raw = pcr_`name_developed_equities' pcr_`name_emerging_equities' ///
        pcr_`name_developed_gov_bond' pcr_`name_emerging_gov_bond' pcr_`name_developed_corp_bond' pcr_`name_real_estate'
        matrix cumu_pcr = cumu_pcr_raw'
            
        matrix rownames cumu_betas_raw = beta_`name_developed_equities' beta_`name_emerging_equities' ///
        beta_`name_developed_gov_bond' beta_`name_emerging_gov_bond' beta_`name_developed_corp_bond' beta_`name_real_estate'
        matrix cumu_betas = cumu_betas_raw'
        
        matrix colnames cumu_return = Expected_return_portfolio
        matrix colnames cumu_stdev = Expected_stdev_portfolio
        matrix colnames cumu_variance = Expected_variance_portfolio
        matrix colnames cumu_sharper = Sharpe_ratio
        }
    matrix portfolio_matrix_rf1 = cumu_return,cumu_stdev,cumu_variance, ///
    cumu_sharper,cumu_weights,cumu_mcr,cumu_cr,cumu_pcr,cumu_betas
    matsave portfolio_matrix_rf1 , saving path(C:\Users\****\OneDrive ///
    - ******* \Documents\Asset Class Research\Results\Portfolios) replace
}

[Mike Lacy]

I don't know anything about your particular problem, but I have some more general thoughts. My simplest idea would be that you sprinkle a bunch of -timer- calls throughout the code to verify where the slow spots are. Perhaps you have already done this. Another observation would be that having a large matrix size set in Stata *can* slow down code execution substantially. (See https://www.statalist.org/forums/for...mp2-vs-v-12-ic) This issue was in an earlier Stata version and may no longer apply. If this problem still prevails and perhaps in any event, using Mata to store any large matrices would be a good idea.