Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #76
    Here is the code that we used in past for getting monthly idiovol from daily return data. Instead of simple standard deviation, i want it to generate ewma standard deviation like we did in #29 of this thread. The difference is now use for each month daily data within that month instead of recursive function in #29. The lambda should optimise in a range of 0.78 and 0.94.

    Code:
    gen mdate = mofd(date)
    format mdate %tm
    gen month = month(date)
    gen year = year(date)
    
    capture program drop one_stock_month
    program define one_stock_month
        capture regress rt mkt smb hml
        if c(rc) == 0 {
            predict resid, resid
            summ resid
            if `r(N)' >= 17 {
                gen idio_vol = r(sd)
                gen n_obs = e(N)
            }
        }
        else if !inlist(c(rc), 2000, 2001) { // ERROR OTHER THAN INSUFFICIENT OBSERVATIONS
            gen unexpected_error_code = c(rc)
            assert 0
        }
        exit
    end
    
    //gen mdate = ym(year, month)
    assert missing(mdate) == missing(month, year)
    format mdate %tm
    
    
    runby one_stock_month, by(stock mdate)
    gen idiovol_m = idio_vol * sqrt(n_obs)
    
    collapse(mean) idio_vol idiovol_m n_obs, by (stock_id stock mdate year month)


    Comment


    • #77
      As I do not have example data with daily observations, the following is untested.

      Code:
      format mdate %tm
      gen month = month(date)
      gen year = year(date)
      
      capture program drop ewma_sd
      program define ewma_sd, rclass
          syntax varname, lambda(real)
          capture assert `lambda' > 0
          if c(rc) {
              display as error "lambda must be > 0"
              exit c(rc)
          }
          tempvar demeaned_sq wt ewama hrmse_sq sigma_sq
          summ `varname'
          scalar `sigma_sq' = r(Var)
          gen `demeaned_sq' = (`varname'-`r(mean)')^2
          replace `wt' = 1-`lambda' in 1
          replace `wt' = `lambda'*`wt'[_n-1] in 2/L
          gen `ewma' = sum(`wt'*`demeaned_sq')
          gen `hrmse_sq' = sum((1-`sigma_sq'/`ewma')^2)/_n
          return scalar ewma_sd = `ewma'[_N]
          return scalar hrmse_sq = `hrmse_sq'[_N]
          exit
      end
      
      capture program drop one_stock_month
      program define one_stock_month
          tempname best_lambda best_ewma_sd best_hrmse_sq
          capture regress rt mkt smb hml
          if c(rc) == 0 {
              predict resid, resid
              count if !missing(resid)
              if `r(N)' >= 17 {
                  gen n_obs = e(N)
                  scalar `best_lambda' = .
                  scalar `best_ewma_sd' = .
                  scalar `best_hrmse_sq' = .
                  forvalues ilambda = 78/97 {
                      local lambda = `ilambda'/100
                      ewma_sd resid, lambda(lambda')
                      if `r(hrmse_sq)' < `best_hrmse_sq' {
                          scalar `best_hrmse_sq' = r(hrmse_sq)
                          scalar `best_ewma_sd' = r(ewma)
                          scalar `best_lambda' = `lambda'
                      }
                  }
                  gen lambda = `best_lambda'
                  gen idio_vol = `best_ewma_sd'
              }
          }
          else if !inlist(c(rc), 2000, 2001) { // ERROR OTHER THAN INSUFFICIENT OBSERVATIONS
              gen unexpected_error_code = c(rc)
              assert 0
          }
          exit
      end
      
      assert missing(mdate) == missing(month, year)
      format mdate %tm
      
      runby one_stock_month, by(stock mdate)
      gen idiovol_m = idio_vol * sqrt(n_obs)
      
      collapse(mean) idio_vol idiovol_m n_obs, by (stock_id stock mdate year month)

      Comment


      • #78
        I tried #77 but there was the following error. Here is the example data for testing. Please name idiovol in #77 as idiovol_ewma.

        Click image for larger version

Name:	error.PNG
Views:	1
Size:	8.2 KB
ID:	1700858



        Code:
        * Example generated by -dataex-. For more info, type help dataex
        clear
        input float stock_id str52 stock int date float(mdate rt mkt smb hml)
        1 "20 Microns Ltd."  20270 666            .             .   .0042193085    .008847037
        1 "20 Microns Ltd."  20271 666 -.0017674112    .003729447    .009692091  -.0045263628
        1 "20 Microns Ltd."  20272 666   .007634525     .00601779    .006099631   -.005851874
        1 "20 Microns Ltd."  20275 666  -.009533726    .004697514    .012641537    .012195588
        1 "20 Microns Ltd."  20276 666  .0045071105  -.0011293275    .008867059    .007311211
        1 "20 Microns Ltd."  20277 666  -.011138892   -.012970348    .007245593   -.003485014
        1 "20 Microns Ltd."  20278 666   -.00336095    .001071178    .006848036     .01298435
        1 "20 Microns Ltd."  20279 666   -.02872754   .0027842545   .0019145465 -.00020462903
        1 "20 Microns Ltd."  20282 666  .0014299294    .011726256    .004453171   .0010474827
        1 "20 Microns Ltd."  20283 666  .0014272725    .001615675    .010216958  -.0034305414
        1 "20 Microns Ltd."  20284 666  -.000201392    .004787312    .007412346   -.006688587
        1 "20 Microns Ltd."  20285 666  -.005079441      .0097098 -.00053673517  -.0021607243
        1 "20 Microns Ltd."  20286 666  -.005103353   .0025296155   .0006656644  .00056716404
        1 "20 Microns Ltd."  20289 666  -.001843428    .003286005    .008675159  -.0034965165
        1 "20 Microns Ltd."  20290 666   -.01171455   -.012316382  -.0042970013   -.006017856
        1 "20 Microns Ltd."  20291 666 -.0018642074    .005958966    .003638487    .004659578
        1 "20 Microns Ltd."  20292 666    .01313302   -.005283029    .008817341    .005334578
        1 "20 Microns Ltd."  20293 666  -.016647683   -.003317458     .00556506   .0030184104
        1 "20 Microns Ltd."  20296 666  -.008561518   -.007408879    .010240247   -.006843528
        1 "20 Microns Ltd."  20297 666  .0014860267    .002675868    .008019182   -.001620295
        1 "20 Microns Ltd."  20298 666  -.010301325    .012807379     .01165691  -.0006447816
        1 "20 Microns Ltd."  20299 666  -.010404396    .013389136  -.0007830746    .013293313
        1 "20 Microns Ltd."  20300 666  -.010509592    .016246747  -.0012361696   -.005835332
        1 "20 Microns Ltd."  20303 667  -.000200314     .01434937    .006502342     .01764884
        1 "20 Microns Ltd."  20304 667   .013688575     .01974439    .004777255     .01270075
        1 "20 Microns Ltd."  20305 667   .013499392    .023205124    .007132859    .002565903
        1 "20 Microns Ltd."  20306 667   -.05087491   .0042659035  -.0045091743   .0037530605
        1 "20 Microns Ltd."  20307 667    .13325271   -.005775879   .0031053196     .00409895
        1 "20 Microns Ltd."  20310 667   -.09596062   -.004357216     .00239611  -.0004651402
        1 "20 Microns Ltd."  20311 667  -.014088127  -.0009637162   -.005391168  -.0089498665
        1 "20 Microns Ltd."  20312 667  -.007237185    -.01256446   -.011633735   -.009613244
        1 "20 Microns Ltd."  20313 667  -.021471526   -.005021412   -.011873954   -.015346112
        1 "20 Microns Ltd."  20314 667   .005239851    .014991548   .0005984493    .006531065
        1 "20 Microns Ltd."  20317 667    .00160687   -.006232704  -.0026301546    .013647935
        1 "20 Microns Ltd."  20318 667  -.014583422   .0043687667    .014249478   .0003608065
        1 "20 Microns Ltd."  20319 667  -.003848873     .01161438    .011497116   -.004577872
        1 "20 Microns Ltd."  20320 667   -.01118825    -.00953579    -.00570118   -.012007738
        1 "20 Microns Ltd."  20321 667    -.0131622   -.007698826   -.003260958   -.005347867
        1 "20 Microns Ltd."  20324 667   -.04335121    -.06570665   -.016852617    -.03561482
        1 "20 Microns Ltd."  20325 667 -.0041208067    .015027613   -.014917485  .00013792608
        1 "20 Microns Ltd."  20326 667   .015548794   -.011725404    .008573346   -.002429816
        1 "20 Microns Ltd."  20327 667  -.007951176    .016824683    .013171615     .00589592
        1 "20 Microns Ltd."  20328 667    .01933201    .005336863   -.005003365   -.001343249
        1 "20 Microns Ltd."  20331 667   .005547889   -.004455866   .0012938606   .0008467354
        1 "20 Microns Ltd."  20332 668  -.036389716   -.019836847  -.0014446843   -.012877185
        1 "20 Microns Ltd."  20333 668    .01956361   .0018049945    .010106497    .001513611
        1 "20 Microns Ltd."  20334 668 -.0021372226    .009638972  -.0016217455     .01102497
        1 "20 Microns Ltd."  20335 668  -.007966229   -.025296975     -.0073821    -.01267442
        1 "20 Microns Ltd."  20338 668  -.017811762   -.016425258   -.010768353   .0010926612
        1 "20 Microns Ltd."  20339 668   .013744986    .007605531   -.013756854    .011574262
        1 "20 Microns Ltd."  20340 668   .013553218    .012292445  .00003501773     .01330324
        1 "20 Microns Ltd."  20341 668  -.011827145   .0021531584   .0003329711   -.005993267
        1 "20 Microns Ltd."  20342 668  -.006081591    .010905297    .007114007    .001377667
        1 "20 Microns Ltd."  20345 668   .015579855      .0082512 -.00053840317    .003542666
        1 "20 Microns Ltd."  20346 668  -.017674966   -.010863299    .003942807  -.0042494573
        1 "20 Microns Ltd."  20347 668   .003752255  -.0022234062   -.005304738   .0012659256
        1 "20 Microns Ltd."  20349 668   .003736694    .007227478   .0012199385    .004150007
        1 "20 Microns Ltd."  20352 668  -.004121883    .001751294    .009153182    .013417855
        1 "20 Microns Ltd."  20353 668   -.00807433    -.01477855    .009592444   -.008050804
        1 "20 Microns Ltd."  20354 668  -.006150543   -.002860879   .0021164156    .008230699
        1 "20 Microns Ltd."  20355 668   -.00219417    -.00333375    .004805115   -.004511183
        1 "20 Microns Ltd."  20359 668   .001801838  -.0015770913   .0018894458    .002699061
        1 "20 Microns Ltd."  20360 668   .001797846     .00175911  -.0003078228  -.0012893027
        1 "20 Microns Ltd."  20361 668  -.000189547    .008072901  -.0006321548   -.004117488
        1 "20 Microns Ltd."  20362 669  -.004173611  .00012563528    .007277767    -.00516295
        1 "20 Microns Ltd."  20366 669   .017810453     .02117105   -.002657907    .010039192
        1 "20 Microns Ltd."  20367 669   .003739726   .0038144654   .0004126388    .006856047
        1 "20 Microns Ltd."  20368 669   .007638242     .00239674   .0019675056    .015504044
        1 "20 Microns Ltd."  20369 669   -.00601479  -.0022589627    .005405358    .006004614
        1 "20 Microns Ltd."  20370 669   .001763578   .0008774677   .0006707767  -.0007982799
        1 "20 Microns Ltd."  20373 669   .007607724    -.01118331    .002226815    .003908804
        1 "20 Microns Ltd."  20374 669    .13907544   -.004622195      .0111378  -.0007405677
        1 "20 Microns Ltd."  20375 669   .016788382  -.0044044913    .010539166  -.0019355296
        1 "20 Microns Ltd."  20376 669   .014835495    .001978061   -.002443706    .005348848
        1 "20 Microns Ltd."  20377 669  -.003479021    .002638345   -.003994431   -.000457317
        1 "20 Microns Ltd."  20380 669    .05591606    .006615637    .004771267    .000418101
        1 "20 Microns Ltd."  20381 669  -.012689548    .007023136      .0062746    .002407652
        1 "20 Microns Ltd."  20382 669 -.0081020165    .005878343    .004166266    .001697975
        1 "20 Microns Ltd."  20384 669 -.0017855203   .0002192413  -.0018092375  -.0020113438
        1 "20 Microns Ltd."  20387 669  -.012970177  -.0042733015   -.005438377   -.005406835
        1 "20 Microns Ltd."  20388 669   .014372483  -.0022402762    .002410448  -.0020542557
        1 "20 Microns Ltd."  20389 669  -.025708964   -.009087654     .01018942   -.005797414
        1 "20 Microns Ltd."  20390 669   -.07547704   -.006532498    .005146565  .00035221525
        1 "20 Microns Ltd."  20391 669    .06883593   -.006518561    -.00700781  -.0013592686
        1 "20 Microns Ltd."  20394 670   .003120634   -.005812875   .0018372035   -.009535984
        1 "20 Microns Ltd."  20395 670  -.010091614    .005468268    .003365594    .004606066
        1 "20 Microns Ltd."  20396 670   .006476043   -.004039801 -.00003928848   .0042278683
        1 "20 Microns Ltd."  20397 670  -.011780028    -.01912563  -.0017207973   -.005808974
        1 "20 Microns Ltd."  20398 670  -.010240875   -.011294059   -.005532554   -.007643224
        1 "20 Microns Ltd."  20401 670   .015037802   -.009990253    .007383344    .008065933
        1 "20 Microns Ltd."  20402 670 -.0035250364   -.013837024    .010513665    .005461536
        1 "20 Microns Ltd."  20403 670 -.0035361846     .01104036      .0096573    .002777593
        1 "20 Microns Ltd."  20405 670  -.016970227    -.01645684  -.0010797991    .007136637
        1 "20 Microns Ltd."  20408 670   -.02578897   -.005650816  -.0020707469    .011336782
        1 "20 Microns Ltd."  20409 670  -.021207465    .003919278   .0020711026   -.003075988
        1 "20 Microns Ltd."  20410 670  -.007347338     -.0074408    .005637501   -.005492143
        1 "20 Microns Ltd."  20411 670   .008817306    .011444896   .0005765436   -.000876015
        1 "20 Microns Ltd."  20412 670  .0015940113   .0026333136    .006309196  -.0010258779
        1 "20 Microns Ltd."  20415 670   -.02158208  -.0021586178    .009734134   -.002787372
        1 "20 Microns Ltd."  20416 670  -.005656184   -.007174118     .01423654    .005323611
        1 "20 Microns Ltd."  20418 670   .007134304   -.005924334    .003094188   .0019620266
        1 "20 Microns Ltd."  20419 670  -.002009885 -.00005020163   -.007216605    .004809673
        1 "20 Microns Ltd."  20422 670   -.00383469  -.0027996136    .007633659    .010483582
        1 "20 Microns Ltd."  20423 671   .005292758   .0004139603   .0017446168    .008513784
        1 "20 Microns Ltd."  20424 671    .05799012  -.0019100425     .00675412    .002775951
        1 "20 Microns Ltd."  20425 671  -.003628129   -.005194707    .007575184   -.004923135
        1 "20 Microns Ltd."  20426 671  -.008812393  -.0025902116    .005208387   -.007012753
        1 "20 Microns Ltd."  20429 671  -.015843878    .003712974     .00122669    .004733671
        1 "20 Microns Ltd."  20430 671  -.003725272   -.005246898   -.004062135   -.010782647
        1 "20 Microns Ltd."  20431 671  -.026788523   -.009558908     -.0189911   -.008243792
        1 "20 Microns Ltd."  20432 671  -.003835766    .008346443    .011534528  -.0003332458
        1 "20 Microns Ltd."  20433 671 -.0038490864   -.006607338   .0020398851   -.005564062
        1 "20 Microns Ltd."  20436 671  -.000192779    .005149879   .0014610402    .007202588
        1 "20 Microns Ltd."  20437 671    .02916502    .007934771    .005703515   -.001740543
        1 "20 Microns Ltd."  20438 671  -.009106512     .00543624  -.0031767895 -.00019418634
        1 "20 Microns Ltd."  20439 671   .014194633    .005135213   .0026000105    .007614901
        1 "20 Microns Ltd."  20440 671  -.009059105   -.013275035    .006089601   .0015232882
        1 "20 Microns Ltd."  20443 671 -.0019827657   .0008475546    .001466765     .01064071
        1 "20 Microns Ltd."  20444 671   .005182487   -.009449074      .0106189  .00014222681
        1 "20 Microns Ltd."  20445 671   .035456765    .006357748  -.0019108928   .0007947236
        1 "20 Microns Ltd."  20446 671    .09619168  -.0018639097    .008789377  -.0002281014
        1 "20 Microns Ltd."  20450 671    .03905261     .00502922   .0004554503  -.0021606535
        1 "20 Microns Ltd."  20451 671  -.003215005    .003617077   -.003019128  -.0025403544
        1 "20 Microns Ltd."  20452 671     .1073819   .0008594976    .003175102   .0014514746
        1 "20 Microns Ltd."  20453 671  -.001561846    .006168416  -.0001561125    .003490314
        2 "360 One Wam Ltd." 20270 666            .             .   .0042193085    .008847037
        2 "360 One Wam Ltd." 20271 666            .    .003729447    .009692091  -.0045263628
        2 "360 One Wam Ltd." 20272 666            .     .00601779    .006099631   -.005851874
        2 "360 One Wam Ltd." 20275 666            .    .004697514    .012641537    .012195588
        2 "360 One Wam Ltd." 20276 666            .  -.0011293275    .008867059    .007311211
        2 "360 One Wam Ltd." 20277 666            .   -.012970348    .007245593   -.003485014
        2 "360 One Wam Ltd." 20278 666            .    .001071178    .006848036     .01298435
        2 "360 One Wam Ltd." 20279 666            .   .0027842545   .0019145465 -.00020462903
        2 "360 One Wam Ltd." 20282 666            .    .011726256    .004453171   .0010474827
        2 "360 One Wam Ltd." 20283 666            .    .001615675    .010216958  -.0034305414
        2 "360 One Wam Ltd." 20284 666            .    .004787312    .007412346   -.006688587
        2 "360 One Wam Ltd." 20285 666            .      .0097098 -.00053673517  -.0021607243
        2 "360 One Wam Ltd." 20286 666            .   .0025296155   .0006656644  .00056716404
        2 "360 One Wam Ltd." 20289 666            .    .003286005    .008675159  -.0034965165
        2 "360 One Wam Ltd." 20290 666            .   -.012316382  -.0042970013   -.006017856
        2 "360 One Wam Ltd." 20291 666            .    .005958966    .003638487    .004659578
        2 "360 One Wam Ltd." 20292 666            .   -.005283029    .008817341    .005334578
        2 "360 One Wam Ltd." 20293 666            .   -.003317458     .00556506   .0030184104
        2 "360 One Wam Ltd." 20296 666            .   -.007408879    .010240247   -.006843528
        2 "360 One Wam Ltd." 20297 666            .    .002675868    .008019182   -.001620295
        2 "360 One Wam Ltd." 20298 666            .    .012807379     .01165691  -.0006447816
        2 "360 One Wam Ltd." 20299 666            .    .013389136  -.0007830746    .013293313
        2 "360 One Wam Ltd." 20300 666            .    .016246747  -.0012361696   -.005835332
        2 "360 One Wam Ltd." 20303 667            .     .01434937    .006502342     .01764884
        2 "360 One Wam Ltd." 20304 667            .     .01974439    .004777255     .01270075
        2 "360 One Wam Ltd." 20305 667            .    .023205124    .007132859    .002565903
        2 "360 One Wam Ltd." 20306 667            .   .0042659035  -.0045091743   .0037530605
        2 "360 One Wam Ltd." 20307 667            .   -.005775879   .0031053196     .00409895
        2 "360 One Wam Ltd." 20310 667            .   -.004357216     .00239611  -.0004651402
        2 "360 One Wam Ltd." 20311 667            .  -.0009637162   -.005391168  -.0089498665
        2 "360 One Wam Ltd." 20312 667            .    -.01256446   -.011633735   -.009613244
        2 "360 One Wam Ltd." 20313 667            .   -.005021412   -.011873954   -.015346112
        2 "360 One Wam Ltd." 20314 667            .    .014991548   .0005984493    .006531065
        2 "360 One Wam Ltd." 20317 667            .   -.006232704  -.0026301546    .013647935
        2 "360 One Wam Ltd." 20318 667            .   .0043687667    .014249478   .0003608065
        2 "360 One Wam Ltd." 20319 667            .     .01161438    .011497116   -.004577872
        2 "360 One Wam Ltd." 20320 667            .    -.00953579    -.00570118   -.012007738
        2 "360 One Wam Ltd." 20321 667            .   -.007698826   -.003260958   -.005347867
        2 "360 One Wam Ltd." 20324 667            .    -.06570665   -.016852617    -.03561482
        2 "360 One Wam Ltd." 20325 667            .    .015027613   -.014917485  .00013792608
        2 "360 One Wam Ltd." 20326 667            .   -.011725404    .008573346   -.002429816
        2 "360 One Wam Ltd." 20327 667            .    .016824683    .013171615     .00589592
        2 "360 One Wam Ltd." 20328 667            .    .005336863   -.005003365   -.001343249
        2 "360 One Wam Ltd." 20331 667            .   -.004455866   .0012938606   .0008467354
        2 "360 One Wam Ltd." 20332 668            .   -.019836847  -.0014446843   -.012877185
        2 "360 One Wam Ltd." 20333 668            .   .0018049945    .010106497    .001513611
        2 "360 One Wam Ltd." 20334 668            .    .009638972  -.0016217455     .01102497
        2 "360 One Wam Ltd." 20335 668            .   -.025296975     -.0073821    -.01267442
        2 "360 One Wam Ltd." 20338 668            .   -.016425258   -.010768353   .0010926612
        2 "360 One Wam Ltd." 20339 668            .    .007605531   -.013756854    .011574262
        2 "360 One Wam Ltd." 20340 668            .    .012292445  .00003501773     .01330324
        2 "360 One Wam Ltd." 20341 668            .   .0021531584   .0003329711   -.005993267
        2 "360 One Wam Ltd." 20342 668            .    .010905297    .007114007    .001377667
        2 "360 One Wam Ltd." 20345 668            .      .0082512 -.00053840317    .003542666
        2 "360 One Wam Ltd." 20346 668            .   -.010863299    .003942807  -.0042494573
        2 "360 One Wam Ltd." 20347 668            .  -.0022234062   -.005304738   .0012659256
        2 "360 One Wam Ltd." 20349 668            .    .007227478   .0012199385    .004150007
        2 "360 One Wam Ltd." 20352 668            .    .001751294    .009153182    .013417855
        2 "360 One Wam Ltd." 20353 668            .    -.01477855    .009592444   -.008050804
        2 "360 One Wam Ltd." 20354 668            .   -.002860879   .0021164156    .008230699
        2 "360 One Wam Ltd." 20355 668            .    -.00333375    .004805115   -.004511183
        2 "360 One Wam Ltd." 20359 668            .  -.0015770913   .0018894458    .002699061
        2 "360 One Wam Ltd." 20360 668            .     .00175911  -.0003078228  -.0012893027
        2 "360 One Wam Ltd." 20361 668            .    .008072901  -.0006321548   -.004117488
        2 "360 One Wam Ltd." 20362 669            .  .00012563528    .007277767    -.00516295
        2 "360 One Wam Ltd." 20366 669            .     .02117105   -.002657907    .010039192
        2 "360 One Wam Ltd." 20367 669            .   .0038144654   .0004126388    .006856047
        2 "360 One Wam Ltd." 20368 669            .     .00239674   .0019675056    .015504044
        2 "360 One Wam Ltd." 20369 669            .  -.0022589627    .005405358    .006004614
        2 "360 One Wam Ltd." 20370 669            .   .0008774677   .0006707767  -.0007982799
        2 "360 One Wam Ltd." 20373 669            .    -.01118331    .002226815    .003908804
        2 "360 One Wam Ltd." 20374 669            .   -.004622195      .0111378  -.0007405677
        2 "360 One Wam Ltd." 20375 669            .  -.0044044913    .010539166  -.0019355296
        2 "360 One Wam Ltd." 20376 669            .    .001978061   -.002443706    .005348848
        2 "360 One Wam Ltd." 20377 669            .    .002638345   -.003994431   -.000457317
        end
        format %td date
        format %tm mdate
        Last edited by Sartaj Hussain; 08 Feb 2023, 20:54.

        Comment


        • #79
          OK, turns out there were a lot of typos in the code. I have now fixed those, and have renamed the variable we are creating idiovol_ewma, as you requested. Because the typos were scattered in many places, and were mostly single characters, I have not highlighted the changes. So just assume that everything has changed, but nothing has changed much.

          Code:
          gen month = month(date)
          gen year = year(date)
          
          capture program drop ewma_sd
          program define ewma_sd, rclass
              syntax varname, lambda(real)
              capture assert `lambda' > 0
              if c(rc) {
                  display as error "lambda must be > 0"
                  exit c(rc)
              }
              tempvar demeaned_sq wt ewma hrmse_sq sigma_sq
              summ `varlist'
              set tracedepth 1
              set trace on
              scalar `sigma_sq' = r(Var)
              gen `demeaned_sq' = (`varlist'-`r(mean)')^2
              gen `wt' = 1-`lambda' in 1
              replace `wt' = `lambda'*`wt'[_n-1] in 2/L
              gen `ewma' = sum(`wt'*`demeaned_sq')
              gen `hrmse_sq' = sum((1-`sigma_sq'/`ewma')^2)/_n
              return scalar ewma_sd = `ewma'[_N]
              return scalar hrmse_sq = `hrmse_sq'[_N]
              set trace off
              exit
          end
          
          capture program drop one_stock_month
          program define one_stock_month
              tempname best_lambda best_ewma_sd best_hrmse_sq
              capture regress rt mkt smb hml
              if c(rc) == 0 {
                  predict resid, resid
                  count if !missing(resid)
                  if `r(N)' >= 17 {
                      gen n_obs = e(N)
                      scalar `best_lambda' = .
                      scalar `best_ewma_sd' = .
                      scalar `best_hrmse_sq' = .
                      forvalues ilambda = 78/97 {
                          local lambda = `ilambda'/100
                          ewma_sd resid, lambda(`lambda')
                          if `r(hrmse_sq)' < `best_hrmse_sq' {
                              scalar `best_hrmse_sq' = r(hrmse_sq)
                              scalar `best_ewma_sd' = r(ewma)
                              scalar `best_lambda' = `lambda'
                          }
                      }
                      gen lambda = `best_lambda'
                      gen idiovol_ewma = `best_ewma_sd'
                  }
              }
              else if !inlist(c(rc), 2000, 2001) { // ERROR OTHER THAN INSUFFICIENT OBSERVATIONS
                  gen unexpected_error_code = c(rc)
                  assert 0
              }
              exit
          end
          
          assert missing(mdate) == missing(month, year)
          format mdate %tm
          
          runby one_stock_month, by(stock mdate) verbose
          gen idiovol_m = idiovol_ewma * sqrt(n_obs)
          
          collapse(mean) idiovol* n_obs, by (stock_id stock mdate year month)

          Comment


          • #80
            I tried #79. It does not generate the desired variables. Please run it on example data and debug.

            Comment


            • #81
              Sorry, I missed another typo. Now fixed. And I added lambda to the end results.
              Code:
              gen month = month(date)
              gen year = year(date)
              
              capture program drop ewma_sd
              program define ewma_sd, rclass
                  syntax varname, lambda(real)
                  capture assert `lambda' > 0
                  if c(rc) {
                      display as error "lambda must be > 0"
                      exit c(rc)
                  }
                  tempvar demeaned_sq wt ewma hrmse_sq sigma_sq
                  summ `varlist'
                  scalar `sigma_sq' = r(Var)
                  gen `demeaned_sq' = (`varlist'-`r(mean)')^2
                  gen `wt' = 1-`lambda' in 1
                  replace `wt' = `lambda'*`wt'[_n-1] in 2/L
                  gen `ewma' = sum(`wt'*`demeaned_sq')
                  gen `hrmse_sq' = sum((1-`sigma_sq'/`ewma')^2)/_n
                  return scalar ewma_sd = `ewma'[_N]
                  return scalar hrmse_sq = `hrmse_sq'[_N]
                  exit
              end
              
              capture program drop one_stock_month
              program define one_stock_month
                  tempname best_lambda best_ewma_sd best_hrmse_sq
                  capture regress rt mkt smb hml
                  if c(rc) == 0 {
                      predict resid, resid
                      count if !missing(resid)
                      if `r(N)' >= 17 {
                          gen n_obs = e(N)
                          scalar `best_lambda' = .
                          scalar `best_ewma_sd' = .
                          scalar `best_hrmse_sq' = .
                          forvalues ilambda = 78/97 {
                              local lambda = `ilambda'/100
                              ewma_sd resid, lambda(`lambda')
                              if `r(hrmse_sq)' < `best_hrmse_sq' {
                                  scalar `best_hrmse_sq' = r(hrmse_sq)
                                  scalar `best_ewma_sd' = r(ewma_sd)
                                  scalar `best_lambda' = `lambda'
                              }
                          }
                          gen lambda = `best_lambda'
                          gen idiovol_ewma = `best_ewma_sd'
                      }
                  }
                  else if !inlist(c(rc), 2000, 2001) { // ERROR OTHER THAN INSUFFICIENT OBSERVATIONS
                      gen unexpected_error_code = c(rc)
                      assert 0
                  }
                  exit
              end
              
              assert missing(mdate) == missing(month, year)
              format mdate %tm
              
              runby one_stock_month, by(stock mdate)
              gen idiovol_m = idiovol_ewma * sqrt(n_obs)
              
              collapse(mean) lambda idiovol* n_obs, by (stock_id stock mdate year month)

              Comment


              • #82
                Thanks. Kindly merge the #76 and #81 so that i can run them together in one go. Further, i tried #81, the results showed that in most cases lambda is 0.78. So, i prefer to reduce the floor value of lambda in the actual run of this code. If i set it at the minimum ebb, 0.01. I think that makes sense.

                Comment


                • #83
                  Code:
                  gen month = month(date)
                  gen year = year(date)
                  
                  capture program drop ewma_sd
                  program define ewma_sd, rclass
                      syntax varname, lambda(real)
                      capture assert `lambda' > 0
                      if c(rc) {
                          display as error "lambda must be > 0"
                          exit c(rc)
                      }
                      tempvar demeaned_sq wt ewma hrmse_sq sigma_sq
                      summ `varlist'
                      scalar `sigma_sq' = r(Var)
                      gen `demeaned_sq' = (`varlist'-`r(mean)')^2
                      gen `wt' = 1-`lambda' in 1
                      replace `wt' = `lambda'*`wt'[_n-1] in 2/L
                      gen `ewma' = sum(`wt'*`demeaned_sq')
                      gen `hrmse_sq' = sum((1-`sigma_sq'/`ewma')^2)/_n
                      return scalar ewma_sd = `ewma'[_N]
                      return scalar hrmse_sq = `hrmse_sq'[_N]
                      exit
                  end
                  
                  capture program drop one_stock_month
                  program define one_stock_month
                      tempname best_lambda best_ewma_sd best_hrmse_sq
                      capture regress rt mkt smb hml
                      if c(rc) == 0 {
                          predict resid, resid
                          count if !missing(resid)
                          if `r(N)' >= 17 {
                              gen n_obs = e(N)
                              summ resid
                              gen idiovol = r(sd)
                              scalar `best_lambda' = .
                              scalar `best_ewma_sd' = .
                              scalar `best_hrmse_sq' = .
                              forvalues ilambda = 1/97 {
                                  local lambda = `ilambda'/100
                                  ewma_sd resid, lambda(`lambda')
                                  if `r(hrmse_sq)' < `best_hrmse_sq' {
                                      scalar `best_hrmse_sq' = r(hrmse_sq)
                                      scalar `best_ewma_sd' = r(ewma_sd)
                                      scalar `best_lambda' = `lambda'
                                  }
                              }
                              gen lambda = `best_lambda'
                              gen idiovol_ewma = `best_ewma_sd'
                          }
                      }
                      else if !inlist(c(rc), 2000, 2001) { // ERROR OTHER THAN INSUFFICIENT OBSERVATIONS
                          gen unexpected_error_code = c(rc)
                          assert 0
                      }
                      exit
                  end
                  
                  assert missing(mdate) == missing(month, year)
                  format mdate %tm
                  
                  runby one_stock_month, by(stock mdate)
                  gen idiovol_m = idiovol_ewma * sqrt(n_obs)
                  
                  collapse(mean) lambda idiovol* n_obs, by (stock_id stock mdate year month)
                  Be forewarned: by starting lambda down at .01, this is going to take almost 5 times as long to run as before. We are trying 97 different values of lambda instead of 20.

                  Comment


                  • #84
                    Can you check how things generate in #83. I must make it clear again that idiovol_m and idiovol_ewma should be calculated on each month's daily residuals from the regression using rt as RHS and mkt, smb and hml as LHS variables. idiovol_m is the standard deviation of residuals within a month multiplied by the sqrt of number of rt observations in that month and idiovol_ewma is the ewma standard deviation of residuals multiplied by sqrt of no. of rt observations in that month. Please cross check the things. I say this because #76 and #83 yield different estimates of idiovol_m. Furthermore, idiovol_m and idiovol_ewma estimates are far apart.
                    Last edited by Sartaj Hussain; 10 Feb 2023, 19:06.

                    Comment


                    • #85
                      idiovol_m is the standard deviation of residuals within a month multiplied by the sqrt of number of rt observations in that month and idiovol_ewma is the ewma standard deviation of residuals multiplied by sqrt of no. of rt observations in that month.
                      That is not what I had understood. The value of idiovol_ema I was giving you was the square of that ewma standard deviation.

                      I have also corrected an error in the calculation of the HRMSE that is used to decide on the optimal value of lambda.

                      The code below changes the calculations for these. You will now find that idiovol_m and idiovol_ewma approximately each other fairly well. They are both rather different from just idiovol, due to the absence of a sqrt(N) multiplier in idiovol.

                      In short, these results will not agree with any of the previous versions. But they reflect the understanding I have gained from your remarks in #84.

                      Code:
                      gen month = month(date)
                      gen year = year(date)
                      
                      capture program drop ewma_sd
                      program define ewma_sd, rclass
                          syntax varname, lambda(real)
                          capture assert `lambda' > 0
                          if c(rc) {
                              display as error "lambda must be > 0"
                              exit c(rc)
                          }
                          tempvar wt ewma hrmse_sq sigma_sq demeaned
                          summ `varlist', meanonly
                          gen `demeaned' = `varlist'-r(mean)
                          gen `sigma_sq' = sum(`demeaned'^2)/r(N)
                          gen `wt' = 1-`lambda' in 1
                          replace `wt' = `lambda'*`wt'[_n-1] in 2/L
                          gen `ewma' = sum(`wt'*`demeaned'^2)
                          gen `hrmse_sq' = sum((1-`sigma_sq'/`ewma')^2)/_n
                          return scalar ewma_sd = sqrt(`ewma'[_N])
                          return scalar hrmse_sq = `hrmse_sq'[_N]
                          summ `wt', meanonly
                          exit
                      end
                      
                      
                      capture program drop one_stock_month
                      program define one_stock_month
                          tempname best_lambda best_ewma_sd best_hrmse_sq
                          capture regress rt mkt smb hml
                          if c(rc) == 0 {
                              predict resid, resid
                              count if !missing(resid)
                              if `r(N)' >= 17 {
                                  gen n_obs = e(N)
                                  summ resid
                                  gen idiovol = r(sd)
                                  gen idiovol_m = idiovol*sqrt(n_obs)
                                  scalar `best_lambda' = .
                                  scalar `best_ewma_sd' = .
                                  scalar `best_hrmse_sq' = .
                                  forvalues ilambda = 1/97 {
                                      local lambda = `ilambda'/100
                                      ewma_sd resid, lambda(`lambda')
                                      if `r(hrmse_sq)' < `best_hrmse_sq' {
                                          scalar `best_hrmse_sq' = r(hrmse_sq)
                                          scalar `best_ewma_sd' = r(ewma_sd)
                                          scalar `best_lambda' = `lambda'
                                      }
                                  }
                                  gen lambda = `best_lambda'
                                  gen idiovol_ewma = `best_ewma_sd'*sqrt(n_obs)
                              }
                          }
                          else if !inlist(c(rc), 2000, 2001) { // ERROR OTHER THAN INSUFFICIENT OBSERVATIONS
                              gen unexpected_error_code = c(rc)
                              assert 0
                          }
                          exit
                      end
                      
                      assert missing(mdate) == missing(month, year)
                      format mdate %tm
                      
                      runby one_stock_month, by(stock mdate)
                      
                      collapse(mean) lambda idiovol* n_obs, by (stock_id stock mdate year month
                      Added: Incidentally, at least in the example data, the optimal values of lambda are now coming out in the .92-.94 range. I think the high frequency of low lambda values you were observing before were the results of HRMSE being incorrectly calculated--because a low value of lambda would, in a sense, compensate for the error in the HRMSE calculation. Now that HRMSE is fixed, I think the lambdas will generally be higher than before, with few if any below .78. So before you run this on the entire data set, try it on a moderately sized subset and see. If I'm right about this, then you can change that -for ilambda = 1/97 {- line back to -for ilambda = 78/97-, and speed things up considerably.
                      Last edited by Clyde Schechter; 10 Feb 2023, 21:36.

                      Comment


                      • #86
                        Here is the output of our mean_idovol. From this i want construct a time series graph splitting February 2020 onwards as COVID-19 pandemic phase and before Feb 2020 as Before COVID-19 pandemic phase. Let the graph in COVID-19 pandemic phase be shaded. While you write the code, create the left, right legends and the title blank to be filled by me.

                        Code:
                        * Example generated by -dataex-. For more info, type help dataex
                        clear
                        input float(date ew_idiovol mw_idiovol ew_idiovol_ewma mw_idiovol_ewma)
                        671 .023417836          .  .09220687          .
                        672 .023206336  .01506742  .08711063  .05587107
                        673 .023858806  .01749573  .09204188  .06743933
                        674  .02350621  .01529275  .08909378  .05881445
                        675 .021021735   .0140253  .07534265   .0496587
                        676  .02251958 .014656768  .08701436  .05624857
                        677 .022495864 .012462307  .08931388   .0498612
                        678  .02204422 .013982908  .08293222  .05239205
                        679  .02228907 .014047056  .08801431  .05610633
                        680 .021726424  .01274506   .0824842  .04924339
                        681  .02344835 .013407458  .08527165  .04867215
                        682 .024432156 .017345645    .092978  .06682283
                        683 .019684607 .012478412  .07863837  .04992095
                        684  .01978501  .01339596  .07691032  .05174883
                        685  .02014276  .01408556  .07427165  .05137018
                        686 .019525994  .01245617  .07563717  .04875918
                        687  .02154324 .013479087  .07744316  .04795314
                        688  .02149327 .015359488  .08428822  .06036295
                        689  .02047993  .01238516  .07974316  .04807289
                        690  .02048278 .013368348  .07950368  .05074795
                        691  .02104581  .01367465  .08008914  .05328514
                        692  .02211592 .012793762  .08673228  .04973821
                        693  .02299095 .014420872  .08628366  .05245057
                        694 .023733985 .013670662  .09336489  .05507426
                        695  .02216521 .012159842  .08280101  .04621994
                        696 .022230243 .014376632  .08836822  .05501666
                        697   .0207579 .014437082  .07714292   .0533568
                        698 .018408773  .01268078 .067184865  .04675818
                        699 .020456556  .01325542  .07793207  .05020475
                        700 .022228796  .01472314  .08617847  .05749685
                        701  .02068876 .013254798   .0805367  .05111255
                        702 .022025427 .015420626  .08664648   .0606119
                        703   .0230039  .01382685    .089277  .05333565
                        704  .02231953 .015341328  .07839195  .05403251
                        705  .02584856  .02100759  .09949976   .0813115
                        706  .02218399 .015071272  .08451202  .05729685
                        707  .02012594 .013343028  .07543953  .05130188
                        708   .0195938  .01329471  .07777625  .05175913
                        709  .02384397 .014389946   .0899311  .05437915
                        710   .0215047  .01300814  .07908673  .04704358
                        711 .018514233  .01251668  .06778153  .04619909
                        712  .02347532 .014647798   .0913079  .05807031
                        713 .021079697  .01120274  .07598822  .04143645
                        714 .022706894 .014306654  .08986729  .05694377
                        715 .027740473  .01595076  .10521495  .06040176
                        716   .0254924 .015416665  .09420017  .05461931
                        717 .024948327 .016010037  .09266273  .05964346
                        718 .024596855 .013802493  .09344223  .05269103
                        719  .02182805 .011609272  .08364799  .04616707
                        720  .02411366  .01283362   .0976353  .05026184
                        721  .02540845  .01541484   .0954599  .05895171
                        722 .036775474  .03058305  .13816863  .11202332
                        723 .033955038 .027434485  .12352158  .09969267
                        724  .02587849 .021062745   .0940013  .07644185
                        725 .032392908  .01965377  .12862898  .07743605
                        726 .024768664  .01795377  .10117964  .07258658
                        727  .02849189  .01576133   .1108059   .0619081
                        728 .022607805 .016109452  .08948836  .06295903
                        729 .022240354 .016480017  .08696478      .0641
                        730   .0240185 .017315987  .08872441 .066085294
                        731  .02734557 .014366627   .1106225  .05820997
                        732 .023123946 .015995814  .08823797  .05936247
                        733  .02525088 .017119179  .09562696  .06504971
                        734  .02382095 .014731973  .09350375  .05730236
                        735 .024142226  .01583082  .08749191  .05775988
                        736 .027827427 .014967525  .10436919  .05673853
                        737 .026372453 .013161543  .10490674  .05247869
                        738  .02467473 .012883958  .09515885  .04854123
                        739  .02334748  .01390694  .09058758  .05378415
                        740 .021177307 .014274257   .0824979  .05461653
                        741  .02326395 .016826179  .08811764  .06268219
                        742 .021569913 .014130401  .08179868  .05289773
                        743 .023075437 .012143246   .0930823  .04966624
                        744 .024557184 .014362818  .09425472  .05423184
                        745  .02248281 .013313255  .08550703  .04970826
                        746  .02253647 .015321594  .08595773   .0601228
                        747   .0240195 .014472432    .089048  .05325527
                        748 .023990326 .016608886  .09238937  .06352431
                        749  .02061941 .013540097   .0809321  .05397228
                        750 .020271786  .01385697  .07761818  .05323698
                        751  .02270717 .013028264  .08650455  .04996042
                        752  .02184862  .01325078  .08754092  .05222613
                        753 .018624483  .01258016  .06743392  .04530554
                        754   .0211467 .012357004   .0816681  .04810936
                        755 .020575434 .011113273  .08225431  .04483847
                        end
                        format %tm date
                        .

                        Comment


                        • #87
                          From this i want construct a time series graph splitting February 2020 onwards as COVID-19 pandemic phase and before Feb 2020 as Before COVID-19 pandemic phase. Let the graph in COVID-19 pandemic phase be shaded
                          I don't understand what you mean by "splitting" here, nor do I understand what you want shaded. Also unclear whether you want a separate graph for each of the *idiovol* variables, or if you want them all in one graph overlayed.

                          The code for a basic graph of these time series, all overlaid in one graph, with a demarcation line at February 2020, would look like this:
                          Code:
                          format date %tmMon_CCYY
                          tsset date
                          tsline *idiovol*, tline(2020m2) lpatt(solid longdash shortdash dash_dot)
                          You would be able to control the axis titles with the -ttitle()- and -ytitle()- options, and you can control the contents of the legend with the -legend()- options. Here I have just left the defaults.

                          Comment


                          • #88
                            Click image for larger version

Name:	shade.png
Views:	1
Size:	22.9 KB
ID:	1701244


                            This is the example of shaded graph. In our case, COVID19 period is to be shaded. Yes i need both versions: a separate graph for each of the idiovol variable in black and white and a single graph overlayed.

                            Comment


                            • #89
                              Try this:
                              Code:
                              format date %tmMon_CCYY
                              gen covid = (date >= tm(2020m2))*0.15
                              tsset date
                              tsline *idiovol*, lpatt(solid longdash shortdash dash_dot) ///
                                  || bar covid date, lwidth(none) color(gs20%10) name(all, replace)
                                  
                              foreach v of varlist *idiovol* {
                                  summ `v', meanonly
                                  local bar_height = .05*ceil(r(max)/.05)
                                  replace covid = (date >= tm(2020m2))*`bar_height'
                                  tsline `v', lpatt(solid) || bar covid date, ///
                                      lwidth(none) color(gs20%10) name(`v', replace)
                              }

                              Comment


                              • #90
                                Brilliant. Let me have all individual graphs in one panel.

                                Comment

                                Working...
                                X