Hello everyone,
I have district level daily news reports from 2015-2015 and I want to identify days in each district that have unusually high reporting. I then want to use these outliers do investigate if people react to such a high media event.
Seasonal-trend decomposition seems a potential option to achieve that. However, for most districts, the fast majority of days have zero reports. For example, district 1 has 3774 observations (days), of which only 32 days have any reports at all. Applying the seasonal trend decomposition would therefor mark every day with a report as an outlier. While these days are technically outliers, I don't expect people to react to a single news report.
So far, I have used the following approach:
I included the condition of reports>=20 to circumvent that every day with >0 reporting is marked as an outlier, but it is quite arbitrary.
The level of reporting is also not stable across the years. 2015 has the highest level of reporting, 2021 the lowest. Following, using just one residual standard deviation would find outliers in 2016 but probably not in 2021. I was thinking of using a rolling window technique but that would loose observations at the beginning of the time series that I need for my further analysis later on.
Any ideas what the best way to go about this is?
Here is an example of my data:
I have district level daily news reports from 2015-2015 and I want to identify days in each district that have unusually high reporting. I then want to use these outliers do investigate if people react to such a high media event.
Seasonal-trend decomposition seems a potential option to achieve that. However, for most districts, the fast majority of days have zero reports. For example, district 1 has 3774 observations (days), of which only 32 days have any reports at all. Applying the seasonal trend decomposition would therefor mark every day with a report as an outlier. While these days are technically outliers, I don't expect people to react to a single news report.
So far, I have used the following approach:
Code:
*** Estimate Regression
gen yhat = .
gen resid = .
levelsof district, local(districts)
foreach d of local districts {
quietly reg reports date i.year i.month i.dow if district == `d'
quietly predict yhat_tmp if district == `d', xb
quietly predict resid_tmp if district == `d', residuals
quietly replace yhat = yhat_tmp if district == `d'
quietly replace resid = resid_tmp if district == `d'
drop yhat_tmp resid_tmp
}
*** Identify spike days
bysort district: egen resid_sd = sd(resid)
gen t_stat = resid / resid_sd
gen d_spike = (t_stat > 2.576) & reports>=20
lab var d_spike "Significantly Increased Reporting"
I included the condition of reports>=20 to circumvent that every day with >0 reporting is marked as an outlier, but it is quite arbitrary.
The level of reporting is also not stable across the years. 2015 has the highest level of reporting, 2021 the lowest. Following, using just one residual standard deviation would find outliers in 2016 but probably not in 2021. I was thinking of using a rolling window technique but that would loose observations at the beginning of the time series that I need for my further analysis later on.
Any ideas what the best way to go about this is?
Here is an example of my data:
Code:
* Example generated by -dataex-. For more info, type help dataex clear input float date double state long district float(reports year month dow) 20331 6 486 0 2015 8 1 20182 27 314 0 2015 4 6 20389 36 503 0 2015 10 3 20415 34 376 0 2015 11 1 20276 19 249 0 2015 7 2 20207 22 261 0 2015 4 3 20234 6 460 0 2015 5 2 20184 17 158 0 2015 4 1 20405 38 325 0 2015 11 5 20379 8 177 0 2015 10 7 20138 5 109 0 2015 2 4 20359 16 302 0 2015 9 1 20339 10 173 0 2015 9 2 20118 36 129 0 2015 1 5 20240 32 443 0 2015 6 1 20242 14 397 0 2015 6 3 20161 4 632 0 2015 3 6 20240 38 384 0 2015 6 1 20438 33 602 0 2015 12 3 20284 25 523 0 2015 7 3 20245 36 297 0 2015 6 6 20090 34 505 0 2015 1 5 20261 21 123 0 2015 6 1 20249 6 35 0 2015 6 3 20362 25 541 0 2015 10 4 20397 21 556 0 2015 11 4 20263 18 161 0 2015 6 3 20286 26 618 0 2015 7 5 20372 21 552 0 2015 10 7 20112 22 407 0 2015 1 6 20197 33 373 0 2015 4 7 20199 18 336 0 2015 4 2 20126 13 154 0 2015 2 6 20352 21 457 0 2015 9 1 20124 6 108 0 2015 2 4 20133 13 379 0 2015 2 6 20096 8 496 0 2015 1 4 20377 16 345 0 2015 10 5 20332 22 166 0 2015 9 2 20247 36 503 0 2015 6 1 20174 8 375 0 2015 3 5 20394 22 599 0 2015 11 1 20403 33 306 0 2015 11 3 20365 31 130 0 2015 10 7 20149 18 551 0 2015 3 1 20220 36 398 0 2015 5 2 20414 6 310 0 2015 11 7 20193 6 108 0 2015 4 3 20268 37 636 0 2015 6 1 20377 30 465 0 2015 10 5 20287 17 120 0 2015 7 6 20436 21 28 0 2015 12 1 20107 27 385 0 2015 1 1 20303 36 553 0 2015 8 1 20335 36 232 0 2015 9 5 20102 18 125 0 2015 1 3 20209 18 414 0 2015 5 5 20225 6 29 0 2015 5 7 20438 5 405 0 2015 12 3 20152 31 256 0 2015 3 4 20159 15 550 0 2015 3 4 20402 22 36 0 2015 11 2 20423 22 19 0 2015 12 2 20169 33 425 0 2015 3 7 20299 25 6 0 2015 7 4 20396 22 332 0 2015 11 3 20323 19 9 0 2015 8 7 20250 4 366 0 2015 6 4 20242 6 410 0 2015 6 3 20360 27 277 0 2015 9 2 20379 6 323 0 2015 10 7 20311 17 320 0 2015 8 2 20269 5 329 0 2015 6 2 20152 36 276 0 2015 3 4 20198 8 375 0 2015 4 1 20420 8 208 0 2015 11 6 20118 22 648 0 2015 1 5 20356 8 375 0 2015 9 5 20113 27 385 0 2015 1 7 20216 22 332 0 2015 5 5 20091 17 175 0 2015 1 6 20139 27 527 0 2015 2 5 20337 11 651 12 2015 9 7 20364 17 320 0 2015 10 6 20259 38 384 0 2015 6 6 20414 5 289 0 2015 11 7 20193 29 299 0 2015 4 3 20305 36 382 0 2015 8 3 20447 35 218 0 2015 12 5 20200 8 50 0 2015 4 3 20272 30 72 0 2015 7 5 20313 12 578 0 2015 8 4 20311 35 319 0 2015 8 2 20399 31 563 0 2015 11 6 20395 4 458 0 2015 11 2 20163 6 371 0 2015 3 1 20203 8 143 0 2015 4 6 20200 38 325 0 2015 4 3 20383 36 312 0 2015 10 4 20129 16 361 0 2015 2 2 end format %td date
Comment