problems using tsfilter

[LPP85]

Dear Statalisters,

I'm having trouble using tsfilter.

I have a dataset such as the one attached. Year and fid are the time and ID variable, respectively, and there are a few numerical variables (3 in the small dataset here below, called var1-var3)

Var1-var3 have missing values due to entry and exit. So for example in var1 the first panel might span years 1990-1999, the second panel 1998-2005, the third panel 1992-2002 and so on.

I'm trying to run a simple code such as

tsfilter hp var1_hp var2_hp var3_hp = var1 var2 var3, smooth(100) trend(var1_tr var2_tr var3_tr)

and I get error messages, different ones depending on the options I specify.

What should I do?

Thanks!

[LPP85]

now with spreadsheet that makes sense

[Nick Cox]

Many people here can't or won't open spreadsheets. On a Stata forum, it is best to show an example of data as the results of a list within CODE delimiters. Alternatively, link to a text file that is readable directly into Stata.

You omit the most important information, precisely what you typed and precisely what error messages you received. Your command at first sight is legal, so the problems presumably lie elsewhere.

Please (re-read) the FAQ Advice (link at top left here), especially Sections 6 (real names strongly preferred) and 12 (how to phrase questions).

[LPP85]

Thanks for the pointers.

This is the code I'm trying to run

id year
tsset  fid year, yearly
tsfilter hp var1_hp var2_hp var3_hp = var1 var2 var3, smooth(100) trend(var1_tr var2_tr var3_tr)
tsfilter hp var1_hp var2_hp var3_hp = var1 var2 var3, smooth(100)

These are the errors I get

. tsfilter hp var1_hp var2_hp var3_hp = var1 var2 var3, smooth(100) trend(var1_tr var2_tr var3_tr)
note: no observations for panel 1
variable var1_tr not found
r(111);


. tsfilter hp var1_hp var2_hp var3_hp = var1 var2 var3, smooth(100)
note: no observations for panel 1
variable var1_hp not found
           st_varindex():  3500  invalid Stata variable name
    _TSFILTER_hpFilter():     -  function returned error
                 <istmt>:     -  function returned error
r(3500);

The data look like this

year,fid,var1,var2,var3
1990,186,5380.46,5273.3599,
1991,186,9582.5303,10368.38,
1992,186,17334.699,17622.801,
1993,186,34890.602,40294.5,
1994,186,73686.102,77342.898,
1995,186,137848.5,148581.8,
1996,186,266783,345723.69,
1997,186,623787.5,743274.31,
1998,186,1224103.1,1485118.8,
1999,186,1953772.3,2621088,
2000,186,3461360,4391060,
2001,186,5132468,8002538,
2002,186,,,
2003,186,,,
2004,186,,,
2005,186,,,
1990,199,6730200,5724200,-4.4245381
1991,199,7462300,6397400,-4.2713447
1992,199,7905000,8263000,-8.5878248
1993,199,9557700,9899900,-9.1797628
1994,199,11006300,12516300,-5.3864183
1995,199,12882200,14684000,-5.5218024
1996,199,14926455,17844250,-5.174973
1997,199,16669250,18903270,-3.7508495
1998,199,18836400,20590290,-2.9179957
1999,199,21044410,22226270,-1.8504258
2000,199,22445760,24025190,-2.0834074
2001,199,25840100,26492400,-1.083522
2002,199,28833600,29931300,-1.3835897
2003,199,,,-2.716855
2004,199,,,-2.3341496
2005,199,,,-.43715096
1990,213,279860,281410,-.97646224
1991,213,816130,872190,-1.372265
1992,213,1144800,1157610,-.97726661
1993,213,1832530,1452170,-.2499736
1994,213,1976650,1626120,-.50800192
1995,213,1929590,1738060,-1.4501308
1996,213,1948110,1871820,-2.2547512
1997,213,2142720,2066090,-1.974159
1998,213,2281710,2195860,-1.8239939
1999,213,2251990,2410180,-3.1917691
2000,213,2273300,2478240,-3.0210369
2001,213,2155080,2393620,-3.4089737
2002,213,,,-1.1497531
2003,213,,,-.15299854
2004,213,,,2.1067286
2005,213,,,
1990,218,187750,159450,-.24412143
1991,218,263820,199840,1.0281136
1992,218,320370,278420,-.55237561
1993,218,449100,357580,-.16231266
1994,218,445880,386920,-1.2650011
1995,218,492190,419060,-.52303284
1996,218,616040,500450,-.7832346
1997,218,667130,661350,-3.0928926
1998,218,746260,742510,-2.8482139
1999,218,800460,803310,-3.460408
2000,218,831290,864390,-4.3581085
2001,218,851846,1059646,-6.6976357
2002,218,,,-7.1716709
2003,218,,,-7.1523457
2004,218,,,-4.7286859
2005,218,,,-2.9388947
1990,223,130.90546,312.53818,
1991,223,594.84003,799.58185,
1992,223,6002.0474,11281.367,
1993,223,176105.45,342258.19,
1994,223,5359890,7441320,
1995,223,7145653.5,7923653.5,
1996,223,8931417,8405987,
1997,223,10717180,8888320,2.644784
1998,223,,10428090,
1999,223,,,
2000,223,,,
2001,223,,,
2002,223,,,
2003,223,,,
2004,223,,,
2005,223,,,
1990,228,,,
1991,228,,,
1992,228,,,
1993,228,,,
1994,228,,,
1995,228,,,
1996,228,,,
1997,228,,,
1998,228,,,
1999,228,,,
2000,228,,,.24993026
2001,228,,,.24907692
2002,228,,,-.46122479
2003,228,,,-.012119692
2004,228,,,2.4145153
2005,228,,,4.6984105
1990,238,6896000,6855000,-.67897362
1991,238,9383000,8958000,-.020526608
1992,238,13939000,10844000,2.1652706
1993,238,16688000,14801000,.85967058
1994,238,19382000,23852000,-3.3421047
1995,238,26017000,27086000,-1.1991336
1996,238,31152000,33118000,-1.3581538
1997,238,37616000,36696000,-.37365705
1998,238,45868000,42812000,.27792865
1999,238,54049000,53136000,-.33948159
2000,238,60370000,59530000,-.30521712
2001,238,71990000,69390000,.18351506
2002,238,80898000,102901104,-3.9898238
2003,238,98160000,114950000,-2.6576555
2004,238,111193128,131199712,-2.6754401
2005,238,132694488,150507856,-2.099288
1990,243,686300,351090,.61321408
1991,243,1019250,591430,.35260612
1992,243,1758220,896490,2.5820396
1993,243,1996290,1241790,.15736832
1994,243,2131620,1295960,-.40239471
1995,243,2487420,1405580,.88716537
1996,243,2639030,1612040,.046258844
1997,243,3470790,2441960,.80363762
1998,243,3881230,2753820,.46979788
1999,243,4368130,3349910,-.36547408
2000,243,5158970,3815760,-.0063850507
2001,243,6041592.5,4609502,.35270399
2002,243,6677937,5320783.5,.71179301
2003,243,8057211.5,6542520.5,1.0708821
2004,243,,,
2005,243,,,
1990,248,149000000,111100000,
1991,248,190700000,167700000,
1992,248,309600000,301300000,
1993,248,437100000,395400000,
1994,248,573400000,488500000,
1995,248,,,
1996,248,,,
1997,248,,,
1998,248,,,
1999,248,,,
2000,248,,,
2001,248,,,
2002,248,,,
2003,248,,,
2004,248,,,
2005,248,,,
1990,253,395190,380770,-29.581394
1991,253,478320,462770,-161.9258
1992,253,552820,568790,-580.005
1993,253,681810,721560,-1024.3307
1994,253,882200,976390,-1234.5045
1995,253,1051010,1044190,-678.58331
1996,253,1062170,1110470,-1242.0476
1997,253,1123740.1,1039300,-520.88629
1998,253,1200890,739940,1599.3103
1999,253,1202810,973810,648.51306
2000,253,1401900,999270,1501.1949
2001,253,149920,178010,-260.07233
2002,253,,,-308.19342
2003,253,,,-268.81647
2004,253,,,-279.28958
2005,253,,,-215.0351
1990,258,278878,313357,-1.9100211
1991,258,427421,385958,.069679484
1992,258,572145,519467,.23530436
1993,258,575395,554204,-.6979416
1994,258,575538,593097,-1.1747178
1995,258,716318,643959.19,-.46826684
1996,258,849880,763480,-.11929325
1997,258,968299,891827,-.68204278
1998,258,1191439,1167456,-1.2392075
1999,258,1415498,1422976,-1.6397732
2000,258,1555412,1638188,-1.7806439
2001,258,1819129,1938043,-1.9331825
2002,258,2071559,2075869,-.95514774
2003,258,2169435,2496856,-2.5947673
2004,258,2338913,2412893,-1.0462242
2005,258,2476459,2666070,-1.7186484
1990,268,,,
1991,268,,,
1992,268,,,
1993,268,,,
1994,268,,,
1995,268,,,
1996,268,,,
1997,268,,,
1998,268,,,
1999,268,,,
2000,268,,,
2001,268,,,
2002,268,,,
2003,268,,,
2004,268,,,
2005,268,,,
1990,273,9726500,10904000,-2.7125547
1991,273,11949800,10695100,-.44808626
1992,273,14633900,12076700,1.2311897
1993,273,15375900,13438300,-.022448726
1994,273,17260000,15338200,-.84420097
1995,273,23436300,22525800,-.63113117
1996,273,32642800,30089700,-.39527631
1997,273,39959712,40222400,-1.8329854
1998,273,42841600,42968800,-2.1440067
1999,273,54499200,54740000,-1.8315789
2000,273,70301904,68730200,-1.6022701
2001,273,,,
2002,273,,,
2003,273,,,
2004,273,,,
2005,273,,,
1990,278,5158,10205.8,
1991,278,229287,171353,5.1534414
1992,278,228421,218725,-3.1933289
1993,278,297772,270603,-.49618605
1994,278,307800,293637,-2.5167961
1995,278,425905,340525,-3.072073
1996,278,476124,392604,-3.6273496
1997,278,545676,468489,-4.1826262
1998,278,656124,561543,-4.7379031
1999,278,866520,727318,-5.2931795
2000,278,941131,822674,-5.8484564
2001,278,935700,1031600,-6.4037333
2002,278,1008561.3,908313,-2.4641345
2003,278,1223009.3,1057349.6,-2.8464758
2004,278,1460441.9,1193157,-2.2201355
2005,278,1743100,1451000,-1.8325808
1990,283,100330,71880,1.4574454
1991,283,129840,74910,5.2080402
1992,283,124490,89820,-.66410923
1993,283,137270,85920,1.1142868
1994,283,134260,103360,-.31053701
1995,283,147190,92440,.78657395
1996,283,151490,111720,-1.1392283
1997,283,163190,125330,-.79928601
1998,283,172810,136530,-3.1612165
1999,283,190770,148660,-.70354301
2000,283,193560,152900,-1.9784002
2001,283,196396,162030,2.1753123
2002,283,,,
2003,283,,,
2004,283,,,
2005,283,,,
1990,288,79693800,54456000,2.9121311
1991,288,98063296,87309104,-.13235193
1992,288,139395808,118697104,.96127731
1993,288,169201296,139143504,1.2099391
1994,288,221424576,169716832,2.0336342
1995,288,274420768,230009600,.16407052
1996,288,295379040,270788128,-.89870441
1997,288,325883360,282488800,-1.0784035
1998,288,375777664,322693600,-.23780034
1999,288,405030944,397592480,-2.7825799
2000,288,424672480,431862944,-3.9154069
2001,288,497203616,453010368,-.85714
2002,288,507811968,491874784,-3.2687595
2003,288,606557632,521506432,-.65683556
2004,288,764850624,570260096,1.6818104
2005,288,843000000,715740032,.7156269
1990,293,61386,99588,-8.1055641
1991,293,317268,340400,-2.4059033
1992,293,578342,679700,-4.0171547
1993,293,886063,938100,-3.8222384
1994,293,1418900,1358200,-2.8688004
1995,293,1872020,1928160,-1.5796199
1996,293,2240820,2087320,2.4276118
1997,293,2557850,2283440,.21561034
1998,293,2697560,2419950,-.57585371
1999,293,2628450,2646590,-2.3770242
2000,293,2843560,2927960,-2.068105
2001,293,2756410,2950180,-2.2071223
2002,293,2917480,3035440,-1.3203553
2003,293,3224630,3275949.8,-1.6226636
2004,293,3589950,3578928.8,-1.0799787
2005,293,4177580,4035080.5,-.72259963
1990,298,171900,115100,
1991,298,366600,235100,
1992,298,668700,426600,
1993,298,1002400,812900,
1994,298,1508700,1203300,
1995,298,2114100,1350600,
1996,298,2879800,1723000,
1997,298,3812800,2566400,
1998,298,4522100,3035700,
1999,298,4519600,3304800,-3.7454188
2000,298,4684200,3192200,-4.0653734
2001,298,4886772,5563960,-4.3853283
2002,298,5278800.5,6175674.5,-4.6938987
2003,298,6452050,7486328.5,-4.4245634
2004,298,7651944.5,8134594.5,-2.6589308
2005,298,8234334.5,8481945,-1.7889329
1990,299,51494.5,41229.102,2.8708441
1991,299,69137,54386.801,3.704901
1992,299,71044,72162.703,-1.6921055
1993,299,89730.898,87777.5,-1.0567117
1994,299,148153.3,143964.09,-1.1198051
1995,299,214421.41,231888.41,-2.1385331
1996,299,566125.19,460432.59,2.6885509
1997,299,998550.63,811580.19,3.1371439
1998,299,872460,1008788.8,-3.7578619
1999,299,1070381.5,1133614.3,-1.5998695
2000,299,1607274.8,1640117.3,-1.1913238
2001,299,1848729.4,2050992,-3.9192181
2002,299,2388920.8,2440166.3,-3.226886
2003,299,3138493.3,3295066,-4.1619239
2004,299,5107631,5142049,-1.4062643
2005,299,8373159.5,7418440,2.1225345
1990,469,1840300,1982500,-1.9510926
1991,469,2962800,2694200,2.4453332
1992,469,4077500,4387400,-1.1746944
1993,469,4695300,4298700,4.1760964
1994,469,5221300,4670400,3.0039999
1995,469,5498100,4859000,3.4416666
1996,469,5966200,5377500,-1.4006103
1997,469,6214600,5603700,-1.6890363
1998,469,6557800,5641300,-.44119695
1999,469,6885600,6118300,-2.4876463
2000,469,7427700,6975800,-3.2534549
2001,469,7434700,8084300,-5.5207696
2002,469,7832000,7784200,-4.7220902
2003,469,8914500,8637100,-3.9291017
2004,469,10188000,10025300,-3.9252009
2005,469,11086460,11431700,-4.3810773
1990,518,1603800,,-5.3711638
1991,518,1764300,,-4.9688973
1992,518,1977900,,-2.9110448
1993,518,2658700,,-2.1955423
1994,518,3185900,,-3.2918477
1995,518,3932200,,-4.0862932
1996,518,5465600,,-3.1532362
1997,518,8743500,,-.91004181
1998,518,11651100,,-.4642882
1999,518,12290500,,-1.3870575
2000,518,,,
2001,518,,,
2002,518,,,
2003,518,,,
2004,518,,,
2005,518,,,
1990,524,7433200,7927500,-5.1708598
1991,524,8429700,9498800,-6.6416898
1992,524,9405000,10198200,-4.8264332
1993,524,10644800,11409500,-5.6465125
1994,524,11829500,14072900,-6.7660303
1995,524,14518900,17329200,-7.5766878
1996,524,15398400,19184900,-7.6508083
1997,524,17210800,20326400,-6.3314285
1998,524,18216500,22156300,-7.0311379
1999,524,20265600,23467800,-5.8722525
2000,524,21638900,28911400,-8.3983784
2001,524,23968900,33514900,-9.3412132
2002,524,26699800,36236200,-7.3665814
2003,524,28421600,37320200,-6.862123
2004,524,32004700,43303600,-7.3334603
2005,524,41230700,49536300,-6.9764047
1990,534,72173000,89206000,-3.4306364
1991,534,86279000,101584000,-2.2808979
1992,534,98492000,115066000,-2.3461657
1993,534,101518000,132670000,-3.9809363
1994,534,123240000,150059008,-2.4390533
1995,534,147786000,171744000,-2.205883
1996,534,171891008,196400000,-2.0424092
1997,534,185836992,225574000,-2.8234146
1998,534,195768992,255346000,-3.3426812
1999,534,233799008,294343008,-3.3419466
2000,534,251254000,329764992,-3.8783929
2001,534,257356000,359919008,-4.4172935
2002,534,290864992,400137984,-4.733922
2003,534,333160992,435008992,-3.4772491
2004,534,387564000,482203008,-3.2000406
2005,534,451048992,551137984,-3.1786454
1990,536,,,
1991,536,4158500096,2556000000,.67608386
1992,536,4749199872,3043300096,-.98443669
1993,536,5228000256,3531780096,.53945708
1994,536,6503000064,3751099904,.74093515
1995,536,7099200000,4318400000,2.4654901
1996,536,8247799808,5285000192,1.5573598
1997,536,10987370496,7869369856,.9133411
1998,536,15281370112,12596490240,-1.8172363
1999,536,18376499200,17794549760,-3.581583
2000,536,24219052032,21210650624,-3.0238786
2001,536,30061606912,24626749440,-2.4661741
2002,536,29851811840,28042850304,-.65235734
2003,536,34137700352,30793447424,-1.87362
2004,536,40058957824,36360806400,-.23166466
2005,536,48560062464,45874315264,-.12593247
1990,566,17672000,18058700,-2.8110805
1991,566,21661200,19959800,-1.497022
1992,566,24083600,21900400,-1.6954495
1993,566,25872800,23456300,-.81311291
1994,566,30624600,28468200,1.4932673
1995,566,33839100,33480100,1.0227965
1996,566,40478900,38492000,.43426973
1997,566,46241500,43503900,.18642271
1998,566,46079800,48515800,-1.6555009
1999,566,47431900,53527700,-3.2961314
2000,566,51011600,58539600,-3.6596608
2001,566,56255900,67252736,-3.666811
2002,566,56967500,75965864,-5.2153168
2003,566,64182000,84679000,-4.4941826
2004,566,70890000,90559000,-3.8328772
2005,566,81563000,97680000,-2.795826
1990,612,,,
1991,612,,,
1992,612,,,
1993,612,,,
1994,612,43419900,37645400,-3.9953513
1995,612,60084700,48130500,-1.2527715
1996,612,82515696,57499600,2.9629238
1997,612,92666800,66155500,2.2830274
1998,612,77451104,66385500,-3.5763409
1999,612,95049600,77469504,-.34543908
2000,612,157816096,85619296,9.7014103
2001,612,150550000,96360000,4.3646808
2002,612,160320000,109770000,1.1630032
2003,612,200071200,114406600,5.5067463
2004,612,225251008,128532096,5.309299
2005,612,311272512,129290200,13.39487

[LPP85]

the first line of code should read

sort fid year

instead of "id year"

[Nick Cox]

Thanks for more details. Please continue to consider changing your identifier.

It's documented (see the manual entry for tsfilter hp) that the sample may not contain gaps. So, you get thrown out as soon as the program finds a gap in the data. Hence, as surmised, the problem is in your dataset.

Missing values at the beginning and end of the sample are excluded from the sample. The sample may not contain gaps.

The following strategy will get you some of the way. Taking your data as above,

 
save hp_all

drop if missing(var1)
tsfilter hp var1_hp = var1 , smooth(100) trend(var1_tr)
save hp_1

use hp_all
drop if missing(var2)
tsfilter hp var2_hp = var2 , smooth(100) trend(var2_tr)
save hp_2

use hp_all
drop if missing(var3)
tsfilter hp var3_hp = var2 , smooth(100) trend(var3_tr)

merge 1:1 fid year using hp_2
drop _merge
merge 1:1 fid year using hp_1

The filter still fails for var3. You may want to consider your options, e.g. interpolating first, although there is some risk of circularity, i.e finding smooth structure you inserted yourself.