[Clyde Schechter]

OK, while trying to write code, I realized that I do not fully understand this. When you say "calculated on previous 36 months of squared demeaned rt", does this mean that squared demeaned rt replace sigma² in the equation of #13 or that it replaces u².

Also, the equation shown in #13 is recursive: the value in time period n is calculated from the results for time period n-1. But to make this work, there has to be some starting value to use in the first time period. Your screenshot table shows blanks (missing values) in the first row--but that will just lead to missing values for all observations when the recursion begins. There must be some values that are stipulated for the first observation.

Also, I was assuming that the 0.06 shown in row 3 in the screenshot's w column is the 1- lambda (because lambda = 94%). But then what are the values of w in the remaining rows? I was expecting to see powers of 0.06, but that is not the case.

[Sartaj Hussain]

OK, while trying to write code, I realized that I do not fully understand this. When you say "calculated on previous 36 months of squared demeaned rt", does this mean that squared demeaned rt replace sigma² in the equation of #13 or that it replaces u².

it replaces u².

Also, the equation shown in #13 is recursive: the value in time period n is calculated from the results for time period n-1. But to make this work, there has to be some starting value to use in the first time period. Your screenshot table shows blanks (missing values) in the first row--but that will just lead to missing values for all observations when the recursion begins. There must be some values that are stipulated for the first observation.

The starting value is just the u².

Also, I was assuming that the 0.06 shown in row 3 in the screenshot's w column is the 1- lambda (because lambda = 94%). But then what are the values of w in the remaining rows? I was expecting to see powers of 0.06, but that is not the case.

The values of w in the remaining rows are current w multiplied by the previous month w.

I am writing the procedure step wise. Please take a look. Hope it makes things more explicit.

Volatility can be estimated using the EWMA by following the process:
Step 1: Demean the rt's
Step 2: Sort the demeaned rt’s in descending order of dates, i.e., from the current to the oldest price.
Step 3: Calculate squared rt’s by squaring the rt’s computed in the previous step.
Step 4: Select the EWMA parameter alpha. For volatility modelling, the value of alpha is 0.8 or greater. The weights are given by a simple procedure. The first weight (1 – a); is the weights that follow are given by a * Previous Weight.
Step 5: Multiply the squared rt’s in step 3 to the corresponding weights computed in step 4. Sum the above product to get the EWMA variance.
Step 6: Finally, the volatility can be computed as the square root of the variance calculated in step 5.

[Clyde Schechter]

Thank you for the instructions in #17. They are very clear and helpful. And I believe I have correctly implemented them below. I should point out, however, that I do not think they implement the formula shown in #13. I have run it on the example data from #1 in this thread.

local lambda 0.97
local window 36

capture program drop one_window
program define one_window
    if _N == window[1] {
        local lambda = lambda[1]
        tsset mdate
        summ rt, meanonly
        gen rt_demeaned_sq = (rt-`r(mean)')^2
        gen wt = 1-`lambda' in 1
        replace wt = `lambda'*L1.wt in 2/L
        gen weighted_rt_demeaned_sq = wt*rt_demeaned_sq
        gen ewma_rt = sum(weighted_rt_demeaned_sq)
        regress rt mkt
        predict resid, resid
        gen weighted_resid_sq = wt*resid^2
        gen ewma_idio = sum(weighted_resid_sq)
        keep in L
        keep stock_id stock date mdate rt mkt ewma_*
        replace date = current_date
    }
    exit
end

gen lambda = `lambda'
gen window = `window'
gen mdate = mofd(date)
format mdate %tm

rangerun one_window, by(stock_id) interval(mdate -`window' -1) sprefix(current_) verbose

Notes:
1. The window size and lambda are specified in the to -local- commands that begin the code. You can change those numbers in those two commands to whatever you need them to be. Do not change any other references to lambda or window, however.
2. The code will allow only for one window size and one lambda to be applied to all of the data. That is, you cannot, in this code, use different window sizes or different lambdas for different stocks. If you need to do that, you have to create separate data sets, and change the values in the -local- commands, and run the code separately on each data set.
3. After you have tested this code on a subset of your real data and are satisfied it works, you can remove the -verbose- option at the end of the -rangerun- command. That will keep your screen from being overrun with endless intermediate output.
4. -rangerun- is written by Robert Picard and is available from SSC. To use it, you must also have -rangestat-, by Robert Picard, Nick Cox, and Roberto Ferrer, also available from SSC.
5. The code does not demean the residuals from the regression because the mean of the residuals is automatically 0, so demeaning would just be a waste of time.

[Sartaj Hussain]

I should point out, however, that I do not think they implement the formula shown in #13. I have run it on the example data from #1 in this thread.

Please have a look at this. I drew inspiration from it. It explains procedure we used and the functional model gives same results.
https://youtu.be/ffDLG7Vt6JE

Hoping that #17 makes it very clear, I want to add one more thing. Since we are estimating volatilities of different stocks whose data might have different empirical properties. Therefore, to choose an ideal lamada value for each stock month combination, is it possible to introduce optimisation. In that case, different lamda values from 0.78 to 0.97 could be iterated and one with lowest RMSE, root mean square error is used for final estimation for a given stock month combination.

[Clyde Schechter]

Yes, it is possible--though it entails changing the code. So before I work on that please check that, for a single value of lambda, the code in #18 performs as you expect it to. In addition, clarify your wishes in regard to these questions:

1. Lowest RMSE of what? The only statistic calculation in this problem that generates an RMSE is the regression of rt on mkt, and that will be the same regardless of the value of lambda. So what is it you actually want to minimize?
2. Once question 1 is answered, what do you want to do if different lambdas minimize for ewma_rt and ewma_idio?
3. When you refer to different lambda values from 0.78 to 0.97, how fine grained do you want to make that? Should I do 0.78 0.79 0.80...0.96 0.97? Or 0.78 0.785 0.79 0.795 0.80...0.96 0.96 0.965 0.97? Or 0.78 0.88 0.97? Or what? The more fine-grained you go the closer to exact the optimization will be, but also the longer the calculations will take.

[Sartaj Hussain]

Please read #19 to this thread again. I have updated it. About your queries in #20, i shall reply seperately.

[Clyde Schechter]

I just realized that the code in #18 is incorrect: it calculates the weights in the wrong order. This is corrected below:

capture program drop one_window
program define one_window
    if _N == window[1] {
        local lambda = lambda[1]
        gsort -mdate
        summ rt, meanonly
        gen rt_demeaned_sq = (rt-`r(mean)')^2
        gen wt = 1-`lambda' in 1
        replace wt = `lambda'*wt[_n-1] in 2/L
        gen weighted_rt_demeaned_sq = wt*rt_demeaned_sq
        gen ewma_rt = sum(weighted_rt_demeaned_sq)
        regress rt mkt
        predict resid, resid
        gen weighted_resid_sq = wt*resid^2
        gen ewma_idio = sum(weighted_resid_sq)
        keep in L
        keep stock_id stock date mdate rt mkt ewma_*
        replace date = current_date
        replace rt = current_rt
        replace mkt = current_mkt
    }
    exit
end

This also raises a new question. The instructions say:

Step 2: Sort the demeaned rt’s in descending order of dates, i.e., from the current to the oldest price. [emphasis added]

But the window doesn't actually contain the current observation, right? It starts one month before the current observation and goes back 36 (or 60, or whatever) months from there, right?

[Sartaj Hussain]

The data in the file is having old to recent order. The window for each stock for instance starts from mdate 2001m12 to 2005m1. And are estimating volatility for mdate 2005m1 using previous 36 months of data.

[Clyde Schechter]

The data in the file is having old to recent order

Right, and that is why what I did in #18 was wrong, because the data need to be in order of most recent first in order for the weights to be done correctly. The code in #22 fixes that (and also fixes another error where the EMWAs were associated with the incorrect values of rt and mkt for the date.)

That video was helpful. And now I understand how this method implements the recursive formula as well. But it leaves me confused about one new issue. In the code, I compute the weighted mean square of rt, and of the regression residuals. But after watching the video, it isn't clear to me whether you want that or whether you want the square roots of those. In the video, he computes it both ways.

Added: I retract what I just said about this method implementing the recursive formula. It would if we didn't demean the returns. But because each month demeans the returns over a different set of values, the recursive formula would fail to reflect that change in the set of values used to demean for that month's calculations. Similarly, each month has a different estimation sample for the rt:mkt regression used in calculating the residuals. So, again, the recursive formula doesn't reflect that. The recursive formula would apply if we were using an expanding window, but not a fixed-width moving window.

[Sartaj Hussain]

Yes, it is possible--though it entails changing the code. So before I work on that please check that, for a single value of lambda, the code in #18 performs as you expect it to. In addition, clarify your wishes in regard to these questions:

1. Lowest RMSE of what? The only statistic calculation in this problem that generates an RMSE is the regression of rt on mkt, and that will be the same regardless of the value of lambda. So what is it you actually want to minimize?
2. Once question 1 is answered, what do you want to do if different lambdas minimize for ewma_rt and ewma_idio?
3. When you refer to different lambda values from 0.78 to 0.97, how fine grained do you want to make that? Should I do 0.78 0.79 0.80...0.96 0.97? Or 0.78 0.785 0.79 0.795 0.80...0.96 0.96 0.965 0.97? Or 0.78 0.88 0.97? Or what? The more fine-grained you go the closer to exact the optimization will be, but also the longer the calculations will take.

It should be heteroscedasticity adjusted RMSE instead of a simple RMSE and HRMSE is given by:

Where the numerator, i.e. sigma squared t is the realised volatility, which is standard deviation of the rt and the denominator, i.e. sigma hat squared t is the ewma volatility of rt. For a given month, we minimise HRMSE subject to:

The idea is to estimate ewma volatility for a stock in a given month using optimum lemda value.

[Clyde Schechter]

This is progress, but we're not there yet. What do i and t stand for in that formula? Is i the stock_id and t the month? (n, presumably, is the maximum value of whatever the variable i represents.) If my guess here about i and t is correct, then you will have a separate value of HRMSE for each month, but the same HRMSE will apply to all stocks in that month. Is that what you expect? That seems to make sense to me, but before I write code I want to confirm that I correctly understand the problem to be solved.

Also, you did not answer my question about how fine-grained you want to be about lambda.

[Sartaj Hussain]

It is slightly different. In our case, i should mean the month and t be the time period. The errors for each month in a given estimation window are summed and then averaged by number of months. Given this, we will have independent lemdas for each stock in a month to estimate ewma. So will be the hrmse values. In brief, lemdas will vary both over stocks as well as months.

Lets fine grain lemda value upto two decimal places.

The data on lemdas used should also generate.

[Sartaj Hussain]

Is it explained well in #27.

[Clyde Schechter]

OK. As you still did not indicate the level of fine-grainedness you preferred for estimating the optimal lambda, I have set the "mesh" at 0.01, so we are estimating for 0.78, 0.79, 0.80,...,0.95, 0.96, 0.97. I believe the following code accomplishes the purpose:

* Example generated by -dataex-. For more info, type help dataex
clear
input float stock_id str54 stock int date float(rt mkt)
1 "3M India Ltd."         15310    -.0887372  -.03955433
1 "3M India Ltd."         15341  -.026404494  .021463245
1 "3M India Ltd."         15372    .04250817   .07242663
1 "3M India Ltd."         15400      .149262  -.02732929
1 "3M India Ltd."         15431   .008990207  -.04528218
1 "3M India Ltd."         15461   -.10039777   -.0853004
1 "3M India Ltd."         15492   .008666431  .002052819
1 "3M India Ltd."         15522    .05207785    -.118447
1 "3M India Ltd."         15553    .05333333    .0436935
1 "3M India Ltd."         15584  -.003164557  -.04591998
1 "3M India Ltd."         15614   .003174603 -.012602217
1 "3M India Ltd."         15645  -.008227848   .04842421
1 "3M India Ltd."         15675  -.034620292   .04941667
1 "3M India Ltd."         15706  -.022806147 -.033169586
1 "3M India Ltd."         15737   -.06308135  .009157823
1 "3M India Ltd."         15765   -.13267148   -.0782898
1 "3M India Ltd."         15796     .1862643 -.010596635
1 "3M India Ltd."         15826    .10526316   .09825777
1 "3M India Ltd."         15857    .06095238   .06033916
1 "3M India Ltd."         15887  .0041891085   .04996381
1 "3M India Ltd."         15918    .10250298     .136254
1 "3M India Ltd."         15949    .11783784  .003297209
1 "3M India Ltd."         15979   .003384913   .06014735
1 "3M India Ltd."         16010     .2060241    .0712712
1 "3M India Ltd."         16040    .03886114   .18632157
1 "3M India Ltd."         16071    -.2037696  -.07444008
1 "3M India Ltd."         16102   -.02415459  .006037154
1 "3M India Ltd."         16131    .02722772  .007745117
1 "3M India Ltd."         16162     .1827711  .008075608
1 "3M India Ltd."         16192   -.17327085   -.1837298
1 "3M India Ltd."         16223    .06653524   .05848542
1 "3M India Ltd."         16253   .016751386   .05154314
1 "3M India Ltd."         16284    .03397341 -.002812333
1 "3M India Ltd."         16315     .1354945   .09279332
1 "3M India Ltd."         16345   -.04432401  .005587839
1 "3M India Ltd."         16376   .067341775   .07218837
1 "3M India Ltd."         16406     .0975332    .0309204
1 "3M India Ltd."         16437  -.006310512 -.030511327
2 "3P Land Holdings Ltd." 15310   -.04444445  -.03955433
2 "3P Land Holdings Ltd." 15341  -.011627907  .021463245
2 "3P Land Holdings Ltd." 15372     .2235294   .07242663
2 "3P Land Holdings Ltd." 15400   -.09615385  -.02732929
2 "3P Land Holdings Ltd." 15431   -.28723404  -.04528218
2 "3P Land Holdings Ltd." 15461     .9701493   -.0853004
2 "3P Land Holdings Ltd." 15492    .25757575  .002052819
2 "3P Land Holdings Ltd." 15522    -.1566265    -.118447
2 "3P Land Holdings Ltd." 15553    -.3214286    .0436935
2 "3P Land Holdings Ltd." 15584   -.15789473  -.04591998
2 "3P Land Holdings Ltd." 15614            0 -.012602217
2 "3P Land Holdings Ltd." 15645            0   .04842421
2 "3P Land Holdings Ltd." 15675         .125   .04941667
2 "3P Land Holdings Ltd." 15706   -.12777779 -.033169586
2 "3P Land Holdings Ltd." 15737     .2101911  .009157823
2 "3P Land Holdings Ltd." 15765    -.2105263   -.0782898
2 "3P Land Holdings Ltd." 15796    .13333334 -.010596635
2 "3P Land Holdings Ltd." 15826    -.1117647   .09825777
2 "3P Land Holdings Ltd." 15857     .3642384   .06033916
2 "3P Land Holdings Ltd." 15887   -.05339806   .04996381
2 "3P Land Holdings Ltd." 15918    .11794872     .136254
2 "3P Land Holdings Ltd." 15949    -.0825688  .003297209
2 "3P Land Holdings Ltd." 15979         -.19   .06014735
2 "3P Land Holdings Ltd." 16010     .4691358    .0712712
2 "3P Land Holdings Ltd." 16040    .25210086   .18632157
2 "3P Land Holdings Ltd." 16071    -.3389262  -.07444008
2 "3P Land Holdings Ltd." 16102    .11675127  .006037154
2 "3P Land Holdings Ltd." 16131     .0909091  .007745117
2 "3P Land Holdings Ltd." 16162   -.06666667  .008075608
2 "3P Land Holdings Ltd." 16192          .25   -.1837298
2 "3P Land Holdings Ltd." 16223   -.14285715   .05848542
2 "3P Land Holdings Ltd." 16253    .15833333   .05154314
2 "3P Land Holdings Ltd." 16284    .26258993 -.002812333
2 "3P Land Holdings Ltd." 16315    .36182335   .09279332
2 "3P Land Holdings Ltd." 16345    .10878661  .005587839
2 "3P Land Holdings Ltd." 16376    .10377359   .07218837
2 "3P Land Holdings Ltd." 16406    .14529915    .0309204
2 "3P Land Holdings Ltd." 16437   -.19701493 -.030511327
3 "A B B India Ltd."      15310  -.004149378  -.03955433
3 "A B B India Ltd."      15341    .09068628  .021463245
3 "A B B India Ltd."      15372      .154382   .07242663
3 "A B B India Ltd."      15400    .03387191  -.02732929
3 "A B B India Ltd."      15431   .002824327  -.04528218
3 "A B B India Ltd."      15461   -.07641757   -.0853004
3 "A B B India Ltd."      15492    .09270177  .002052819
3 "A B B India Ltd."      15522   .031069767    -.118447
3 "A B B India Ltd."      15553    .01930711    .0436935
3 "A B B India Ltd."      15584   -.00495663  -.04591998
3 "A B B India Ltd."      15614   -.14659314 -.012602217
3 "A B B India Ltd."      15645 -.0002084636   .04842421
3 "A B B India Ltd."      15675   .036905754   .04941667
3 "A B B India Ltd."      15706    .12306455 -.033169586
3 "A B B India Ltd."      15737    .14753805  .009157823
3 "A B B India Ltd."      15765    -.1015759   -.0782898
3 "A B B India Ltd."      15796    .07033692 -.010596635
3 "A B B India Ltd."      15826    .11487912   .09825777
3 "A B B India Ltd."      15857    .07335177   .06033916
3 "A B B India Ltd."      15887   .028338984   .04996381
3 "A B B India Ltd."      15918     .2542194     .136254
3 "A B B India Ltd."      15949   .003153911  .003297209
3 "A B B India Ltd."      15979    .09955984   .06014735
3 "A B B India Ltd."      16010   .010293557    .0712712
end
format %td date

capture program drop one_window
program define one_window
    local best_lambda .
    local best_hrmse_sq .
    local best_ewma_idio .
    if _N == window[1] {
        gsort -mdate
        //  CALCULATE RUNNING VARIANCE BY WELFORD'S ALGORITHM
        gen double M1 = rt in 1
        replace M1 = M1[_n-1] + (rt - M1[_n-1])/_n in 2/L
        gen double M2 = 0 in 1
        replace M2 = M2[_n-1] + (rt - M1[_n-1])*(rt - M1) in 2/L
        gen double sigma_sq = cond(_n == 1, 0, M2/(_n-1))
        drop M1 M2
        //  LOOP OVER VALUES OF LAMBDA CALCULATING EWMA IDIO
        //  AND SAVING ONE WITH SMALLEST "RMSE"
        gen double rt_demeaned_sq = .
        gen double wt = .
        gen double weighted_rt_demeaned_sq = .
        gen double ewma_rt = .
        gen double weighted_resid_sq = .
        gen double ewma_idio = .
        gen double hrmse_sq = .
        forvalues ilambda = 78/97 {
            local lambda = `ilambda'/100
            summ rt, meanonly
            replace rt_demeaned_sq = (rt-`r(mean)')^2
            replace wt = 1-`lambda' in 1
            replace wt = `lambda'*wt[_n-1] in 2/L
            replace weighted_rt_demeaned_sq = wt*rt_demeaned_sq
            replace ewma_rt = sum(weighted_rt_demeaned_sq)
            quietly regress rt mkt
            predict resid, resid
            replace weighted_resid_sq = wt*resid^2
            replace ewma_idio = sum(weighted_resid_sq)
            replace hrmse_sq = sum((1-sigma_sq/ewma_idio)^2)/_n
            drop resid
            if hrmse_sq[_N] < `best_hrmse_sq' {
                local best_lambda = `lambda'
                local best_hrmse_sq = hrmse_sq[_N]
                local best_ewma_idio = ewma_idio[_N]
            }
        }
        keep in L
        keep stock_id stock date mdate rt mkt ewma_* sigma_sq
        replace date = current_date
        replace rt = current_rt
        replace mkt = current_mkt
        gen lambda = `best_lambda'
        replace ewma_idio = `best_ewma_idio'
        gen hrmse = sqrt(`best_hrmse_sq')
    }
    exit
end

gen mdate = mofd(date)
format mdate %tm

local window 36
gen window = `window'

rangerun one_window, by(stock_id) interval(mdate -`window' -1) sprefix(current_)

[Sartaj Hussain]

Can you please explain what: "level of fine-grainedness" means? I could not get it well. Also, do through some light on the "WELFORD'S ALGORITHM".