Issues constructing spells using tsspell

[Will Fannin]

Hi, I have monthly data on participation in a program from 2017-2019, and I would like to construct spell measures. I am using tsspell, but I am having trouble correctly defining conditions that yield what I want. The example data contains 4 individuals. I have id 1 and 4 who enter the program and never exit over the sample period; id 2 who enters and exits; and id 3 who only exits. My goal is to specify a tsspell condition that only measures spells for people like id 2 who both enter and exit over the period.

* Example generated by -dataex-. For more info, type help dataex
clear
input long id int year byte month float modate double active
1 2019  1 708 0
1 2019  2 709 0
1 2019  3 710 0
1 2019  4 711 0
1 2019  5 712 0
1 2019  6 713 0
1 2019  7 714 0
1 2019  8 715 1
1 2019  9 716 1
1 2019 10 717 1
1 2019 11 718 1
1 2019 12 719 1
2 2018  1 696 0
2 2018  2 697 0
2 2018  3 698 0
2 2018  4 699 0
2 2018  5 700 0
2 2018  6 701 0
2 2018  7 702 0
2 2018  8 703 0
2 2018  9 704 0
2 2018 10 705 0
2 2018 11 706 0
2 2018 12 707 1
2 2019  1 708 1
2 2019  2 709 1
2 2019  3 710 1
2 2019  4 711 1
2 2019  5 712 1
2 2019  6 713 1
2 2019  7 714 0
2 2019  8 715 0
2 2019  9 716 0
2 2019 10 717 0
2 2019 11 718 0
2 2019 12 719 0
3 2017  1 684 1
3 2017  2 685 1
3 2017  3 686 1
3 2017  4 687 1
3 2017  5 688 1
3 2017  6 689 1
3 2017  7 690 1
3 2017  8 691 1
3 2017  9 692 1
3 2017 10 693 1
3 2017 11 694 1
3 2017 12 695 1
3 2018  1 696 1
3 2018  2 697 1
3 2018  3 698 1
3 2018  4 699 1
3 2018  5 700 1
3 2018  6 701 1
3 2018  7 702 1
3 2018  8 703 1
3 2018  9 704 1
3 2018 10 705 1
3 2018 11 706 1
3 2018 12 707 1
3 2019  1 708 1
3 2019  2 709 1
3 2019  3 710 0
3 2019  4 711 0
3 2019  5 712 0
3 2019  6 713 0
3 2019  7 714 0
3 2019  8 715 0
3 2019  9 716 0
3 2019 10 717 0
3 2019 11 718 0
3 2019 12 719 0
4 2018  1 696 0
4 2018  2 697 0
4 2018  3 698 0
4 2018  4 699 0
4 2018  5 700 0
4 2018  6 701 0
4 2018  7 702 1
4 2018  8 703 1
4 2018  9 704 1
4 2018 10 705 1
4 2018 11 706 1
4 2018 12 707 1
4 2019  1 708 1
4 2019  2 709 1
4 2019  3 710 1
4 2019  4 711 1
4 2019  5 712 1
4 2019  6 713 1
4 2019  7 714 1
4 2019  8 715 1
4 2019  9 716 1
4 2019 10 717 1
4 2019 11 718 1
4 2019 12 719 1
end
format %tm modate

First, I declare panel data and specify the condition that defines the beginning of a spell:

xtset id modate
ssc install tsspell
tsspell active, fcond((active==1 & active[_n-1]==0))

where a spell begins when the individual enters the program following inactivity in the previous month. However, this is not sufficient because some observations, such as id 1 and 4, enter the program and never come off over the sample period. I do not want to count these as spells! I only want to define a spell for observations that, like id 2, enter and exit the program over the sample. I believe I need to use the cond command to define the end condition (i.e., active==1 & active[_n+1] = 0) as well, but I cannot figure out how to specify both conditions with the tsspell function.

[Nick Cox]

I'd divide your problem into two simpler parts. The first is just to identify spells in which active is constant The constant could be 0 or 1. The second is to reduce the spells to a history. If you want panels only with a history of enter -- active -- exit you are looking for a string as calculated below containing 010

* Example generated by -dataex-. For more info, type help dataex
clear
input long id int year byte month float modate double active
1 2019  1 708 0
1 2019  2 709 0
1 2019  3 710 0
1 2019  4 711 0
1 2019  5 712 0
1 2019  6 713 0
1 2019  7 714 0
1 2019  8 715 1
1 2019  9 716 1
1 2019 10 717 1
1 2019 11 718 1
1 2019 12 719 1
2 2018  1 696 0
2 2018  2 697 0
2 2018  3 698 0
2 2018  4 699 0
2 2018  5 700 0
2 2018  6 701 0
2 2018  7 702 0
2 2018  8 703 0
2 2018  9 704 0
2 2018 10 705 0
2 2018 11 706 0
2 2018 12 707 1
2 2019  1 708 1
2 2019  2 709 1
2 2019  3 710 1
2 2019  4 711 1
2 2019  5 712 1
2 2019  6 713 1
2 2019  7 714 0
2 2019  8 715 0
2 2019  9 716 0
2 2019 10 717 0
2 2019 11 718 0
2 2019 12 719 0
3 2017  1 684 1
3 2017  2 685 1
3 2017  3 686 1
3 2017  4 687 1
3 2017  5 688 1
3 2017  6 689 1
3 2017  7 690 1
3 2017  8 691 1
3 2017  9 692 1
3 2017 10 693 1
3 2017 11 694 1
3 2017 12 695 1
3 2018  1 696 1
3 2018  2 697 1
3 2018  3 698 1
3 2018  4 699 1
3 2018  5 700 1
3 2018  6 701 1
3 2018  7 702 1
3 2018  8 703 1
3 2018  9 704 1
3 2018 10 705 1
3 2018 11 706 1
3 2018 12 707 1
3 2019  1 708 1
3 2019  2 709 1
3 2019  3 710 0
3 2019  4 711 0
3 2019  5 712 0
3 2019  6 713 0
3 2019  7 714 0
3 2019  8 715 0
3 2019  9 716 0
3 2019 10 717 0
3 2019 11 718 0
3 2019 12 719 0
4 2018  1 696 0
4 2018  2 697 0
4 2018  3 698 0
4 2018  4 699 0
4 2018  5 700 0
4 2018  6 701 0
4 2018  7 702 1
4 2018  8 703 1
4 2018  9 704 1
4 2018 10 705 1
4 2018 11 706 1
4 2018 12 707 1
4 2019  1 708 1
4 2019  2 709 1
4 2019  3 710 1
4 2019  4 711 1
4 2019  5 712 1
4 2019  6 713 1
4 2019  7 714 1
4 2019  8 715 1
4 2019  9 716 1
4 2019 10 717 1
4 2019 11 718 1
4 2019 12 719 1
end
format %tm modate

ssc install tsspell

xtset id modate 

tsspell active 

bysort id (modate) : gen pattern = strofreal(active) if _n == 1 

by id: replace pattern = pattern[_n-1] + cond(_seq == 1, strofreal(active), "") if _n > 1 

by id: replace pattern = pattern[_N]

tabdisp id, c(pattern)

----------------------
       id |    pattern
----------+-----------
        1 |         01
        2 |        010
        3 |         10
        4 |         01
----------------------

If you do something like

keep if strpos(pattern, "010")

only individual 2 will be kept.

For the string theory of history, see e.g. https://www.stata-journal.com/articl...article=pr0071

[Will Fannin]

Thanks for your comments, Nick Cox. The difficulty with this approach is that in my full dataset I have more complex patterns that are not conducive to dropping:

. tab pattern

            pattern |      Freq.     Percent        Cum.
--------------------+-----------------------------------
                 01 |    236,556       22.02       22.02
                010 |    200,304       18.65       40.67
              01001 |      2,592        0.24       40.91
             010010 |      1,536        0.14       41.06
            0100101 |        192        0.02       41.07
...

It is not that I want to keep only "010" and other similar patterns, but rather I only want to calculate spell length for periods of activity that have a beginning and end. For example, with a "01001" pattern, I want to keep the observation but I only want to calculate spell length for the first period of activity. For a "0100101" pattern, I want to calculate spell length for the first two periods of activity, and so on.

My goal is to calculate average spell length for the full sample, where spell is a period of activity that is preceded and succeeded by at least one period of inactivity.

[Nick Cox]

You seem to be implying that what I suggest is limited for not meeting criteria you didn't originally explain. I have no doubt that is true.

The occurrence of 00 suggests to me that you have gaps in your data too.

There are any number of approaches here, such as defining spells only for active == 1 and then checking what happened before and after each such spell.

[Will Fannin]

Nick Cox: I am having trouble checking what happened before and after each spell.

tsspell active if active==1
bysort id (modate) : gen end_actual = _end==1 & active[_n+1]==0

Above I have identified the periods of activity that are followed by a period of inactivity, but I want to add the condition that the period of activity is preceded by a period of inactivity. My thought here is something like:

bysort id (modate) : gen begin_actual = _seq==1 & active[_n-1]==0

But I also need to specify this only apply to periods of activity where end_actual==1. I don't know how to specify both conditions at the same time. I'm not sure how to translate this into code, I'm looking for something like: gen seq_actual = _seq if when _seq==1 for that spell active[_n-1]==0 and when _end==1 for that spell active[_n+1]==0.

[Nick Cox]

I can't follow #5.easily without seeing worked examples and indeed without playing with data. Trying to wrap your head around the subtleties of someone else's problem from a word description can be hard. You gave a data example in #1 and I needed some messing around with that to see what (I think) you want.

tsspell supports if as a matter of general form but its use doesn't always match the real problem whenever -- as is clearly true here -- the characteristics of what you are ignoring through the if are of importance.

I can make most progress by using two definitions of spells, a strict definition in which a spell has active 1 throughout and a wide definition of spell in which someone flips back and forth between active 1 and 0.

Your need is to copy information on previous and following states and the use of subscripts lets you copy values from the observations adjacent to each spell strict sense. The trick you need then is to use egen to spread values to all observations in the spell (although the calculation can be done in other ways).

* Example generated by -dataex-. For more info, type help dataex
clear
input long id int year byte month float modate double active
1 2019  1 708 0
1 2019  2 709 0
1 2019  3 710 0
1 2019  4 711 0
1 2019  5 712 0
1 2019  6 713 0
1 2019  7 714 0
1 2019  8 715 1
1 2019  9 716 1
1 2019 10 717 1
1 2019 11 718 1
1 2019 12 719 1
2 2018  1 696 0
2 2018  2 697 0
2 2018  3 698 0
2 2018  4 699 0
2 2018  5 700 0
2 2018  6 701 0
2 2018  7 702 0
2 2018  8 703 0
2 2018  9 704 0
2 2018 10 705 0
2 2018 11 706 0
2 2018 12 707 1
2 2019  1 708 1
2 2019  2 709 1
2 2019  3 710 1
2 2019  4 711 1
2 2019  5 712 1
2 2019  6 713 1
2 2019  7 714 0
2 2019  8 715 0
2 2019  9 716 0
2 2019 10 717 0
2 2019 11 718 0
2 2019 12 719 0
3 2017  1 684 1
3 2017  2 685 1
3 2017  3 686 1
3 2017  4 687 1
3 2017  5 688 1
3 2017  6 689 1
3 2017  7 690 1
3 2017  8 691 1
3 2017  9 692 1
3 2017 10 693 1
3 2017 11 694 1
3 2017 12 695 1
3 2018  1 696 1
3 2018  2 697 1
3 2018  3 698 1
3 2018  4 699 1
3 2018  5 700 1
3 2018  6 701 1
3 2018  7 702 1
3 2018  8 703 1
3 2018  9 704 1
3 2018 10 705 1
3 2018 11 706 1
3 2018 12 707 1
3 2019  1 708 1
3 2019  2 709 1
3 2019  3 710 0
3 2019  4 711 0
3 2019  5 712 0
3 2019  6 713 0
3 2019  7 714 0
3 2019  8 715 0
3 2019  9 716 0
3 2019 10 717 0
3 2019 11 718 0
3 2019 12 719 0
4 2018  1 696 0
4 2018  2 697 0
4 2018  3 698 0
4 2018  4 699 0
4 2018  5 700 0
4 2018  6 701 0
4 2018  7 702 1
4 2018  8 703 1
4 2018  9 704 1
4 2018 10 705 1
4 2018 11 706 1
4 2018 12 707 1
4 2019  1 708 1
4 2019  2 709 1
4 2019  3 710 1
4 2019  4 711 1
4 2019  5 712 1
4 2019  6 713 1
4 2019  7 714 1
4 2019  8 715 1
4 2019  9 716 1
4 2019 10 717 1
4 2019 11 718 1
4 2019 12 719 1
end
format %tm modate

ssc install tsspell

xtset id modate 

tsspell, cond(active==1) seq(actseq) spell(actspell) end(actend)

tsspell active, seq(anyseq) spell(anyspell) end(anyend)

bysort id (modate) : gen work = active[_n-1] if actseq == 1 

egen previous = max(work), by(id anyspell)

drop work 

bysort id (modate) : gen work = active[_n+1] if actend == 1 

egen next = max(work), by(id anyspell)

drop work 

list id modate-actend previous next, sepby(id actspell)

     +----------------------------------------------------------------------+
     | id    modate   active   actseq   actspell   actend   previous   next |
     |----------------------------------------------------------------------|
  1. |  1    2019m1        0        0          0        0          .      . |
  2. |  1    2019m2        0        0          0        0          .      . |
  3. |  1    2019m3        0        0          0        0          .      . |
  4. |  1    2019m4        0        0          0        0          .      . |
  5. |  1    2019m5        0        0          0        0          .      . |
  6. |  1    2019m6        0        0          0        0          .      . |
  7. |  1    2019m7        0        0          0        0          .      . |
     |----------------------------------------------------------------------|
  8. |  1    2019m8        1        1          1        0          0      . |
  9. |  1    2019m9        1        2          1        0          0      . |
 10. |  1   2019m10        1        3          1        0          0      . |
 11. |  1   2019m11        1        4          1        0          0      . |
 12. |  1   2019m12        1        5          1        1          0      . |
     |----------------------------------------------------------------------|
 13. |  2    2018m1        0        0          0        0          .      . |
 14. |  2    2018m2        0        0          0        0          .      . |
 15. |  2    2018m3        0        0          0        0          .      . |
 16. |  2    2018m4        0        0          0        0          .      . |
 17. |  2    2018m5        0        0          0        0          .      . |
 18. |  2    2018m6        0        0          0        0          .      . |
 19. |  2    2018m7        0        0          0        0          .      . |
 20. |  2    2018m8        0        0          0        0          .      . |
 21. |  2    2018m9        0        0          0        0          .      . |
 22. |  2   2018m10        0        0          0        0          .      . |
 23. |  2   2018m11        0        0          0        0          .      . |
     |----------------------------------------------------------------------|
 24. |  2   2018m12        1        1          1        0          0      0 |
 25. |  2    2019m1        1        2          1        0          0      0 |
 26. |  2    2019m2        1        3          1        0          0      0 |
 27. |  2    2019m3        1        4          1        0          0      0 |
 28. |  2    2019m4        1        5          1        0          0      0 |
 29. |  2    2019m5        1        6          1        0          0      0 |
 30. |  2    2019m6        1        7          1        1          0      0 |
     |----------------------------------------------------------------------|
 31. |  2    2019m7        0        0          0        0          .      . |
 32. |  2    2019m8        0        0          0        0          .      . |
 33. |  2    2019m9        0        0          0        0          .      . |
 34. |  2   2019m10        0        0          0        0          .      . |
 35. |  2   2019m11        0        0          0        0          .      . |
 36. |  2   2019m12        0        0          0        0          .      . |
     |----------------------------------------------------------------------|
 37. |  3    2017m1        1        1          1        0          .      0 |
 38. |  3    2017m2        1        2          1        0          .      0 |
 39. |  3    2017m3        1        3          1        0          .      0 |
 40. |  3    2017m4        1        4          1        0          .      0 |
 41. |  3    2017m5        1        5          1        0          .      0 |
 42. |  3    2017m6        1        6          1        0          .      0 |
 43. |  3    2017m7        1        7          1        0          .      0 |
 44. |  3    2017m8        1        8          1        0          .      0 |
 45. |  3    2017m9        1        9          1        0          .      0 |
 46. |  3   2017m10        1       10          1        0          .      0 |
 47. |  3   2017m11        1       11          1        0          .      0 |
 48. |  3   2017m12        1       12          1        0          .      0 |
 49. |  3    2018m1        1       13          1        0          .      0 |
 50. |  3    2018m2        1       14          1        0          .      0 |
 51. |  3    2018m3        1       15          1        0          .      0 |
 52. |  3    2018m4        1       16          1        0          .      0 |
 53. |  3    2018m5        1       17          1        0          .      0 |
 54. |  3    2018m6        1       18          1        0          .      0 |
 55. |  3    2018m7        1       19          1        0          .      0 |
 56. |  3    2018m8        1       20          1        0          .      0 |
 57. |  3    2018m9        1       21          1        0          .      0 |
 58. |  3   2018m10        1       22          1        0          .      0 |
 59. |  3   2018m11        1       23          1        0          .      0 |
 60. |  3   2018m12        1       24          1        0          .      0 |
 61. |  3    2019m1        1       25          1        0          .      0 |
 62. |  3    2019m2        1       26          1        1          .      0 |
     |----------------------------------------------------------------------|
 63. |  3    2019m3        0        0          0        0          .      . |
 64. |  3    2019m4        0        0          0        0          .      . |
 65. |  3    2019m5        0        0          0        0          .      . |
 66. |  3    2019m6        0        0          0        0          .      . |
 67. |  3    2019m7        0        0          0        0          .      . |
 68. |  3    2019m8        0        0          0        0          .      . |
 69. |  3    2019m9        0        0          0        0          .      . |
 70. |  3   2019m10        0        0          0        0          .      . |
 71. |  3   2019m11        0        0          0        0          .      . |
 72. |  3   2019m12        0        0          0        0          .      . |
     |----------------------------------------------------------------------|
 73. |  4    2018m1        0        0          0        0          .      . |
 74. |  4    2018m2        0        0          0        0          .      . |
 75. |  4    2018m3        0        0          0        0          .      . |
 76. |  4    2018m4        0        0          0        0          .      . |
 77. |  4    2018m5        0        0          0        0          .      . |
 78. |  4    2018m6        0        0          0        0          .      . |
     |----------------------------------------------------------------------|
 79. |  4    2018m7        1        1          1        0          0      . |
 80. |  4    2018m8        1        2          1        0          0      . |
 81. |  4    2018m9        1        3          1        0          0      . |
 82. |  4   2018m10        1        4          1        0          0      . |
 83. |  4   2018m11        1        5          1        0          0      . |
 84. |  4   2018m12        1        6          1        0          0      . |
 85. |  4    2019m1        1        7          1        0          0      . |
 86. |  4    2019m2        1        8          1        0          0      . |
 87. |  4    2019m3        1        9          1        0          0      . |
 88. |  4    2019m4        1       10          1        0          0      . |
 89. |  4    2019m5        1       11          1        0          0      . |
 90. |  4    2019m6        1       12          1        0          0      . |
 91. |  4    2019m7        1       13          1        0          0      . |
 92. |  4    2019m8        1       14          1        0          0      . |
 93. |  4    2019m9        1       15          1        0          0      . |
 94. |  4   2019m10        1       16          1        0          0      . |
 95. |  4   2019m11        1       17          1        0          0      . |
 96. |  4   2019m12        1       18          1        1          0      . |
     +----------------------------------------------------------------------+

If I understand correctly you now want to focus on active spells for which previous == 0 & next == 0.

You haven't picked up on my suggestion that some of your histories may include gaps.

[Will Fannin]

Nick Cox That is brilliant and exactly what I was looking for. Regarding gaps, I've read from help tsspell that the fcond (varname != varname[_n-1]) | (_n == 1) can be used to span gaps. I do not want to span gaps in this case because I want a "11" pattern to constitute two separate spells.