[Richard Williams]

I am pleasantly surprised that Nick's code had such a dramatic impact, and I wonder why it did. I had thought the number of calculations would be about the same either way. Does putting things in loops dramatically improve execution speed? Or is it some other change Nick made? I like loops because they are tidier and if you have to make a change, you only have to do it once, not 20 times. Loops are easier to debug. But if they are also speed demons the documentation could mention that. Maybe have a FAQ on tips for easy ways to speed up Stata.

[Raoul Eireiner]

Dear Richard,

I think Nick's code was just computationally less intense than my original one. Especially the following lines helped to reduce the runtime immensely.

quietly bysort Tradenumb (Date) : gen trade_start_date = Date[1]
quietly bysort Tradenumb (Date) : gen trade_start_date = Date[_N]

I think generally it doesn't make much difference if you generate variables line by line by eachother of if you solve that by using a loop. As you correctly noted, the use of loops definitely keeps the code way tidier and makes debugging a whole lot easier.

Hope that answers your question a bit.

[Raoul Eireiner]

But I still dont manage to avoid the r(103) error! Does anyone have a clue what goes wrong? Do you think do, nostop might help in this case?

[Nick Cox]

Where does the code stop?

I am surprised that people are surprised at the speed up. A loop that went round the data 3000 times for 3000 distinct groups was replaced by single statements. With an if condition Stata tests all the observations to see which satisfy the condition.

[Clyde Schechter]

When I run the code in #14 with the example data you gave in #4, I do not encounter that error. (I do get an error DT__1 not found when it gets to the command - replace Treasury_disc_`k' = DT__`j' if Treasury_coupon_`k'_date - Date == `j'-. But that is understandable because your example data does not contain any DT__1 variable, only DT__0.

Please post an example of the data you are using that produces your error with this code.

[Raoul Eireiner]

Dear Nick,

the code stops always in the last loop shown below.

 forvalues j= 0/3600 {
    forval k = 1/20 {
        replace Treasury_disc_`k' = DT__`j' if Treasury_coupon_`k'_date - Date == `j'
        replace Swap_disc_`k' = DS__`j' if Swap_coupon_`k'_date - Date == `j'
        local todrop `todrop' DT__`j' DS__`j'
    }
}

Obviously, it is actually able to replace some of the Swap_disc_`k' & Treasury_disc_`k' variables but when it stops, oddly enough, all DT__`j' & DS__`j' variables are still in the file, non is dropped. At the moment I'm trying if I can force STATA with capture to go through the whole DO-file without stopping. I will let you know how the outcome is. But I think by using capture only the file is continued but the outcome is still faulty.

[Raoul Eireiner]

Dear Clyde,

when I want to post a more exact example of my used data, STATA is still crashing when trying to use dataex. I'm therefore unfortunately not able to give you a better example than the one already included in #4. I'm terribly sorry about that, and understand that this way it's very hard to help me with my issue.

[Clyde Schechter]

Well, if the program is breaking inside the loop you show in #21, there is no surprise that the DT__* and DS__* variables have not been dropped at that point. Placing their names in a local macro named -todrop- doesn't cause them to be dropped. All it does is put their names in a list. Nothing is done with those variables until the command -drop `todrop'- is reached. Since that command comes after the loop, it is never reached.

The question remains, where is the r(103) "too many variables specified" command arising. As I can't reproduce the error with anything available to me, I can't find it for you. But here's something you can do yourself. Just before the loop that you show in #21, insert the following two lines of code:

set tracedepth 1
set trace on

Now when you run the code, as Stata executes the loop, it will show each line of code, both as written, and with all of its macros expanded. When the code finally reaches the point where the error occurs, you will be able to see the actual exact line of code that Stata is trying to execute. When you see that it may be obvious to you what is going wrong and how to fix it. If not, post that output and perhaps somebody can figure it out.

[Robert Picard]

A big mystery to me is what these DT_* and DS_* variables contain. In the #4 example, we only see DT_0 and DS_0 and both are constant with a value of 1. Do these variables change across observations within the same Tradenumb by-group?

To make sense of the problem, I created a demonstration dataset with 2 positions, each starting on a different date with weekends removed. I'm not sure of the length of each position so I generated 3600 days. I then created variables that follow the example in #4. Since I don't know what to put in the DT_* and DS_* variables, I make sure that each contains a different and decreasing number to make it easier to track how the values are copied:

clear all
set seed 431
set maxvar 10000
set obs 2
gen Tradenumb = _n
expand 3600
bysort Tradenumb: gen Date = Tradenumb * 7 + mdy(1,1,2008) + _n
format Date %tdDayDDmonCCYY
drop if inlist(dow(Date), 0, 6)  // Sunday and Saturday    

isid Tradenumb Date, sort

foreach v in Treasury Swap LIBOR LIBOR_discount Repo {
    gen `v' = runiform()
}
forvalues i = 0/3600 {
    gen DT__`i' = 1 - `i' / 3600
    gen DS__`i' = 1 - `i' / 5000
}
save "test_data.dta", replace

It takes a while for the code in #1 to process the demonstration dataset above.

Seems to me that most of the initialization code in #1 can be skipped. A lot of what follows appears to be date gymnastics with the goal of extracting the correct value from these DT_* and DS_* variables. There seems to be the notion of quarters (90 days) and half-years (180 days). I think that most of this can be simplified by defining a tday variable that tracks the number of days from the initial day for each position. You can then create bins for quarter and half-year. There's only one LIBOR date variable and that's easily derived using the same technique used for binning quarters. I don't quite understand why you need 20 variables for the treasury and swap discounts. As coded, the discount copied from the DT_* and DS_* variables changes for a given observation across each half-year variable (Treasury_disc_* and Swap_disc_*). Instead of copying the values, I store the index that would be used to determine which DT_* and DS_* would be used. Since I don't quite see the logic here, that's as far as I will go. My intuition is that the quarter 20 variables contain the index to use across all dates and that all the others are wrong. For all the other variables, the index is negative for all dates past the date of the quarter.

use "test_data.dta", clear

* verify assumptions about the data
isid Tradenumb Date, sort

* the day of each observation relative to the first trade date
by Tradenumb: gen tday = Date - Date[1] + 1

* create bins of 180 days
gen halfyear = ceil(tday/180)
tab halfyear Tradenumb

* create bins of 90 days
gen quarter = ceil(tday/90)
tab quarter Tradenumb

by Tradenumb: gen LIBOR_coupon_date = Date[1] + 90 * (ceil(tday/90))
by Tradenumb: gen REPO_reset_date = Date[_n-1]
format %td LIBOR_coupon_date REPO_reset_date

* copy relevant DT_* DS_* values at half-year coupon dates
qui forvalues i = 1/20 {
    by Tradenumb: gen days2coupon = `i' * 180
    gen Treasury_disc_`i' = days2coupon - tday
    gen Swap_disc_`i'     = days2coupon - tday
    drop days2coupon
}

drop DT_* DS_*

If the DT_* and DS_* variables do not change values across observations within each Tradenumb by-group, this problem can most likely be reduced to a simple merge.

[Raoul Eireiner]

Dear Robert!

As you correctly assume do the DT__* & DS__* variables in fact change (get smaller) across observations (=Dates) within the same Tradenumb by-group. Additionally, I need those 20 (Treasury_disc_* & Swap_disc_*) variables to finally multiply the respective coupon variable with the discount factor (i.e. Treasury_coupon_1*Treasury_disc_1, Treasury_coupon_2*Treasury_disc_2, etc). This will get me to the present values I am so desperately looking for.

[Raoul Eireiner]

Dear Clyde,

I do apologize for not replying the whole day yesterday, however, since the code is still running I wouldn't have been able to report much anyways. Nevertheless by now I think I know what caused the r(103) – "too many variables specified" error. By tracing what the code is exactly doing, like you proposed Clyde , I was able to find out that by running the code as in #14 the DT__`j' & DS__`j' variables (which are to be dropped) get recorded 20 times each (for each of the k values in the nested loop). Accordingly, when the whole outer loop is finished and subsequently the list of variables included in the local macro `todrop' are about to the dropped, STATA gives the error because, as mentioned before, the variable names are recorded more than once.

I would be happy if maybe one of you also sees this problem and could just give me a short confirmation.

If it is in fact this issue that causes the general problem, I think that changing the code from #14 to the code below, should offer a solution to the problem.

forvalues j= 0/3600 {
    forval k = 1/20 {
        replace Treasury_disc_`k' = DT__`j' if Treasury_coupon_`k'_date - Date == `j'
        replace Swap_disc_`k' = DS__`j' if Swap_coupon_`k'_date - Date == `j'
       
    }
    local todrop `todrop' DT__`j' DS__`j'
}
drop `todrop'

Then in my opinion the error should not be caused... What do you think about that?

[Daniel Bela]

Hi,

you could also try to use

local dropvars DT__`j' DS__`j'
local todrop : list todrop | dropvars

instead of the original

local todrop `todrop' DT__`j' DS__`j'

in order to prevent duplicate entries in your varlist that will be passed to drop. See -help macrolists-.

Regards
Bela

[Robert Picard]

I get the feeling that you do not understand much of what I did in #24. If you want more advice from me, please explain in words what these DT_* and DS_* variables are supposed to contain and use listsome (from SSC) to show a better data sample using the following command:

listsome Date DT__1 DS__1 DT__3599 DS__3599 Treasury Swap LIBOR LIBOR_discount Repo if Tradenumb == 1, max(20) random

Please pick another Tradenumb if the first contains less than a year's worth of data.

[Raoul Eireiner]

Hi,

I think I found out now what went wrong. It seems that the duplicates contained in the local macro did not cause any problems when the first block of code (as in #14 – US 10Y 5BP SHORT) was executed. However, the problems showed up at the second block of code (US 10Y 5BP LONG). As it seems, although I use clear all at the start of the new block of code, the local macro `todrop' still exists in the memory and is just being continued with the new DT__`j' & DS__`j' variables from the second block. I thus guess the solution lies in clearing the local macro `todrop' variable after it has been dropped successfully, so it can be newly filled in the following block of code... Does anyone have an idea how to achieve that?

[Raoul Eireiner]

Dear Clyde!

I really appreciate all of your help! I must have misunderstood your last post indeed a bit and couldn't make some sense of it partly. Thanks for taking all of this time to help me with my problems, you guys are literally saving my life by helping me out here. I now tried to include the additional data you asked for. Specifically, this Tradenumb (187) has a duration of 1621 days so should exceed the required year for sure
What is more, I try to explain the DT__* & DS__* variables another time. So the DT__* & DS__* variables include the daily discount factors for each day (respective to Date variable) for a specific maturity (like one year with DT__360 & DS__360, since 1 yr is approximated by 360 days). In one specific discount factor variable, say DT__100 or DS__100, the values are not the same from day to day (i.e. within different values of the Dates variable). The only exception to this represent the discount factors DT__0 & DS__0 because they are always 1.

I hope this clarifies things a bit and if something is still unclear, I am happy to try and explain it some more. It would just be helpful if you would let me know exactly, where you don't understand it anymore, so I can try to answer it more precisely.

 13002. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 14apr2000 |     1 |     1 | .9998397 |   .99982 | .5663999 | .4875446 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |     1.5725    |   .9998255    |   6.01    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13012. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 28apr2000 |     1 |     1 | .9998374 | .9998134 | .5458424 | .4807472 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |     1.5725    |   .9998194    |   5.74    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13014. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 02may2000 |     1 |     1 | .9998349 | .9998118 | .5409105 | .4782673 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |     1.5725    |   .9998176    |   5.71    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13081. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 03aug2000 |     1 |     1 | .9998319 | .9998128 | .5585617 | .4926076 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |     1.6925    |   .9998136    |    6.4    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13118. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 25sep2000 |     1 |     1 | .9998341 | .9998171 | .5647312 |  .500344 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |     1.6925    |    .999815    |   6.46    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13381. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 27sep2001 |     1 |     1 | .9999369 | .9999374 | .6283894 |  .586543 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |        .65    |   .9999278    |   3.01    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13421. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 22nov2001 |     1 |     1 | .9999447 | .9999406 | .6022442 | .5613635 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |        .65    |   .9999399    |   2.04    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13434. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 11dec2001 |     1 |     1 | .9999526 | .9999548 |  .593878 | .5531368 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |        .65    |   .9999472    |   1.74    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13485. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 20feb2002 |     1 |     1 | .9999488 | .9999485 | .6106085 | .5674087 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |   .4759375    |   .9999471    |   1.75    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13549. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 21may2002 |     1 |     1 | .9999474 | .9999471 | .5912688 | .5624459 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |    .509375    |   .9999472    |   1.64    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13570. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 19jun2002 |     1 |     1 | .9999515 |  .999952 | .6159678 | .5810646 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |    .509375    |    .999948    |   1.73    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13605. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 07aug2002 |     1 |     1 | .9999574 | .9999595 | .6392134 | .6056591 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |    .468595    |   .9999511    |   1.77    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13707. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 27dec2002 |     1 |     1 | .9999661 | .9999655 | .6744305 | .6429815 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |      .3525    |   .9999611    |   1.27    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13747. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 21feb2003 |     1 |     1 |  .999967 | .9999655 | .6694944 | .6442544 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |      .3525    |   .9999628    |   1.21    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13757. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 07mar2003 |     1 |     1 | .9999694 | .9999698 | .6881629 | .6588058 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |      .3525    |   .9999635    |   1.23    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13801. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 08may2003 |     1 |     1 | .9999686 | .9999665 | .6829022 | .6649994 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |      .3175    |   .9999644    |   1.24    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 13843. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 07jul2003 |     1 |     1 | .9999732 | .9999709 | .6798394 | .6594242 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |       .265    |   .9999692    |   1.03    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 14008. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 23feb2004 |     1 |     1 | .9999719 | .9999705 | .6591188 | .6320422 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |      .2925    |   .9999689    |      1    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 14054. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 27apr2004 |     1 |     1 | .9999678 | .9999651 | .6352125 | .6064061 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |      .2775    |   .9999675    |    .99    |
        +-----------------------------------------------------------------------+

        +-----------------------------------------------------------------------+
 14074. |      Date | DT__0 | DS__0 |    DT__1 |    DS__1 | DT__3599 | DS__3599 |
        | 25may2004 |     1 |     1 | .9999613 | .9999572 |  .616834 | .5864874 |
        |---------------------------+-------------------------------------------|
        |   Treasury   |    Swap    |      LIBOR    |   LIBOR_~t    |   Repo    |
        |     3.2925   |   3.605    |      .2775    |   .9999642    |    .99    |