Good evening,
I am working on my dissertation and have come to a wall. My dissertation is on the efficacy of postsecondary remediation for incoming freshman at North Dakota institutions. The following are my summary statistics.
summarize Math103Letter mathAct
Variable | Obs Mean Std. Dev. Min Max
-------------+---------------------------------------------------------
Math103Let~r | 6,793 3.98734 1.140518 1 5
mathAct | 6,793 23.66524 3.645204 13 36
I then did a histogram of my running variable.
The cutoff for remediation in North Dakota is a 22, those above a 22 are good, those below 22 will have to take remediation. With the histogram I notice that it is not continuous - there is a large decrease in scores at 21. My opinion is that students who get a 21 will retest to try to do better and then they get a 22 or higher which puts them out of remediation. I do not have data on if they retested or how many times they retested. I think because of that I need to utilize a fuzzy design? or does that fact make RD not a path I want to take?
After the histogram, I continued on the path and completed an RD Manipulation test
rddensity mathAct, c(22)
Computing data-driven bandwidth selectors.
RD Manipulation Test using local polynomial density estimation.
Cutoff c = 22 | Left of c Right of c Number of obs = 6793
-------------------+---------------------- Model = unrestricted
Number of obs | 1227 5566 BW method = comb
Eff. Number of obs | 15 1598 Kernel = triangular
Order est. (p) | 2 2 VCE method = jackknife
Order bias (q) | 3 3
BW est. (h) | 1.034 1.172
Running variable: mathAct.
-----------------------------------------------
Method | T P>|T|
-------------------+---------------------------
Robust | 87.0309 0.0000
-----------------------------------------------
This also gave the warning: bandwidth h may be too low. I am not sure what I would adjust the bandwidth to, any help here would be great.
After that, I did a rdplot
rdplot Math103Letter mathAct, c(22)
RD Plot with evenly spaced mimicking variance number of bins using spacings estimators.
Cutoff c = 22 | Left of c Right of c Number of obs = 6793
----------------------+---------------------- Kernel = Uniform
Number of obs | 1227 5566
Eff. Number of obs | 1227 5566
Order poly. fit (p) | 4 4
BW poly. fit (h) | 9.000 14.000
Number of bins scale | 1.000 1.000
Outcome: Math103Letter. Running variable: mathAct.
---------------------------------------------
| Left of c Right of c
----------------------+----------------------
Bins selected | 73 391
Average bin length | 0.123 0.036
Median bin length | 0.123 0.036
----------------------+----------------------
IMSE-optimal bins | 6 25
Mimicking Var. bins | 73 391
----------------------+----------------------
Rel. to IMSE-optimal: |
Implied scale | 12.167 15.640
WIMSE var. weight | 0.001 0.000
WIMSE bias weight | 0.999 1.000
---------------------------------------------
As you can see there shows a discontinuity in the data.
Then I conducted a rdbwselect to try to find the optimal bandwidth. This could not be completed, I receive the error below:
rdbwselect Math103Letter mathAct, c(22) all
Invertibility problem in the computation of preliminary bandwidth below the thresholdInvertibility problem in the computation of preliminary bandwidth ab
> ove the thresholdInvertibility problem in the computation of bias bandwidth (b) below the thresholdInvertibility problem in the computation of bias ban
> dwidth (b) above the thresholdInvertibility problem in the computation of loc. poly. bandwidth (h) below the thresholdInvertibility problem in the comp
> utation of loc. poly. bandwidth (h) above the threshold
Bandwidth estimators for sharp RD local polynomial regression.
Cutoff c = 22 | Left of c Right of c Number of obs = 6793
-------------------+---------------------- Kernel = Triangular
Number of obs | 1989 4804 VCE method = NN
Min of mathAct | 13.000 23.000
Max of mathAct | 22.000 36.000
Order est. (p) | 1 1
Order bias (q) | 2 2
Outcome: Math103Letter. Running variable: mathAct.
--------------------------------------------------------------------------------
| BW est. (h) | BW bias (b)
Method | Left of c Right of c | Left of c Right of c
-------------------+------------------------------+-----------------------------
mserd | . . | . .
msetwo | . . | . .
msesum | . . | . .
msecomb1 | . . | . .
msecomb2 | . . | . .
-------------------+------------------------------+-----------------------------
cerrd | . . | . .
certwo | . . | . .
cersum | . . | . .
cercomb1 | . . | . .
cercomb2 | . . | . .
--------------------------------------------------------------------------------
I did some reading and saw that the error could come because the running variable is not continuous at the cutoff? Any help here would be great as well. I did try the rdbwselect as a fuzzy RD and I received the same error.
I continued just to see what would happen. I did the rdrobust sharp design and I utilized bandwidth of 9 since that was what it showed in the RD plot, If I did not put a value for h the program did not run. Here are the results:
rdrobust Math103Letter mathAct, c(22) h(9)
Sharp RD estimates using local polynomial regression.
Cutoff c = 22 | Left of c Right of c Number of obs = 6793
-------------------+---------------------- BW type = Manual
Number of obs | 1227 5566 Kernel = Triangular
Eff. Number of obs | 1223 5416 VCE method = NN
Order est. (p) | 1 1
Order bias (q) | 2 2
BW est. (h) | 9.000 9.000
BW bias (b) | 9.000 9.000
rho (h/b) | 1.000 1.000
Outcome: Math103Letter. Running variable: mathAct.
--------------------------------------------------------------------------------
Method | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------------+------------------------------------------------------------
Conventional | .07855 .1176 0.6679 0.504 -.151937 .309032
Robust | - - 1.0388 0.299 -.248302 .808343
--------------------------------------------------------------------------------
According to these results, I could not reject the null hypothesis which would be that remediation does NOT have an effect on the next credit-bearing course, in this case, College Algebra. So to me, that would indicate that math remediation does not have an effect on the grade a student receives in college algebra? Am I on the right track, can I use the bandwidth of 9 or should it be something different? Any help would be appreciated.
Thank you,
Russ Ziegler
I am working on my dissertation and have come to a wall. My dissertation is on the efficacy of postsecondary remediation for incoming freshman at North Dakota institutions. The following are my summary statistics.
summarize Math103Letter mathAct
Variable | Obs Mean Std. Dev. Min Max
-------------+---------------------------------------------------------
Math103Let~r | 6,793 3.98734 1.140518 1 5
mathAct | 6,793 23.66524 3.645204 13 36
I then did a histogram of my running variable.
The cutoff for remediation in North Dakota is a 22, those above a 22 are good, those below 22 will have to take remediation. With the histogram I notice that it is not continuous - there is a large decrease in scores at 21. My opinion is that students who get a 21 will retest to try to do better and then they get a 22 or higher which puts them out of remediation. I do not have data on if they retested or how many times they retested. I think because of that I need to utilize a fuzzy design? or does that fact make RD not a path I want to take?
After the histogram, I continued on the path and completed an RD Manipulation test
rddensity mathAct, c(22)
Computing data-driven bandwidth selectors.
RD Manipulation Test using local polynomial density estimation.
Cutoff c = 22 | Left of c Right of c Number of obs = 6793
-------------------+---------------------- Model = unrestricted
Number of obs | 1227 5566 BW method = comb
Eff. Number of obs | 15 1598 Kernel = triangular
Order est. (p) | 2 2 VCE method = jackknife
Order bias (q) | 3 3
BW est. (h) | 1.034 1.172
Running variable: mathAct.
-----------------------------------------------
Method | T P>|T|
-------------------+---------------------------
Robust | 87.0309 0.0000
-----------------------------------------------
This also gave the warning: bandwidth h may be too low. I am not sure what I would adjust the bandwidth to, any help here would be great.
After that, I did a rdplot
rdplot Math103Letter mathAct, c(22)
RD Plot with evenly spaced mimicking variance number of bins using spacings estimators.
Cutoff c = 22 | Left of c Right of c Number of obs = 6793
----------------------+---------------------- Kernel = Uniform
Number of obs | 1227 5566
Eff. Number of obs | 1227 5566
Order poly. fit (p) | 4 4
BW poly. fit (h) | 9.000 14.000
Number of bins scale | 1.000 1.000
Outcome: Math103Letter. Running variable: mathAct.
---------------------------------------------
| Left of c Right of c
----------------------+----------------------
Bins selected | 73 391
Average bin length | 0.123 0.036
Median bin length | 0.123 0.036
----------------------+----------------------
IMSE-optimal bins | 6 25
Mimicking Var. bins | 73 391
----------------------+----------------------
Rel. to IMSE-optimal: |
Implied scale | 12.167 15.640
WIMSE var. weight | 0.001 0.000
WIMSE bias weight | 0.999 1.000
---------------------------------------------
As you can see there shows a discontinuity in the data.
Then I conducted a rdbwselect to try to find the optimal bandwidth. This could not be completed, I receive the error below:
rdbwselect Math103Letter mathAct, c(22) all
Invertibility problem in the computation of preliminary bandwidth below the thresholdInvertibility problem in the computation of preliminary bandwidth ab
> ove the thresholdInvertibility problem in the computation of bias bandwidth (b) below the thresholdInvertibility problem in the computation of bias ban
> dwidth (b) above the thresholdInvertibility problem in the computation of loc. poly. bandwidth (h) below the thresholdInvertibility problem in the comp
> utation of loc. poly. bandwidth (h) above the threshold
Bandwidth estimators for sharp RD local polynomial regression.
Cutoff c = 22 | Left of c Right of c Number of obs = 6793
-------------------+---------------------- Kernel = Triangular
Number of obs | 1989 4804 VCE method = NN
Min of mathAct | 13.000 23.000
Max of mathAct | 22.000 36.000
Order est. (p) | 1 1
Order bias (q) | 2 2
Outcome: Math103Letter. Running variable: mathAct.
--------------------------------------------------------------------------------
| BW est. (h) | BW bias (b)
Method | Left of c Right of c | Left of c Right of c
-------------------+------------------------------+-----------------------------
mserd | . . | . .
msetwo | . . | . .
msesum | . . | . .
msecomb1 | . . | . .
msecomb2 | . . | . .
-------------------+------------------------------+-----------------------------
cerrd | . . | . .
certwo | . . | . .
cersum | . . | . .
cercomb1 | . . | . .
cercomb2 | . . | . .
--------------------------------------------------------------------------------
I did some reading and saw that the error could come because the running variable is not continuous at the cutoff? Any help here would be great as well. I did try the rdbwselect as a fuzzy RD and I received the same error.
I continued just to see what would happen. I did the rdrobust sharp design and I utilized bandwidth of 9 since that was what it showed in the RD plot, If I did not put a value for h the program did not run. Here are the results:
rdrobust Math103Letter mathAct, c(22) h(9)
Sharp RD estimates using local polynomial regression.
Cutoff c = 22 | Left of c Right of c Number of obs = 6793
-------------------+---------------------- BW type = Manual
Number of obs | 1227 5566 Kernel = Triangular
Eff. Number of obs | 1223 5416 VCE method = NN
Order est. (p) | 1 1
Order bias (q) | 2 2
BW est. (h) | 9.000 9.000
BW bias (b) | 9.000 9.000
rho (h/b) | 1.000 1.000
Outcome: Math103Letter. Running variable: mathAct.
--------------------------------------------------------------------------------
Method | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------------+------------------------------------------------------------
Conventional | .07855 .1176 0.6679 0.504 -.151937 .309032
Robust | - - 1.0388 0.299 -.248302 .808343
--------------------------------------------------------------------------------
According to these results, I could not reject the null hypothesis which would be that remediation does NOT have an effect on the next credit-bearing course, in this case, College Algebra. So to me, that would indicate that math remediation does not have an effect on the grade a student receives in college algebra? Am I on the right track, can I use the bandwidth of 9 or should it be something different? Any help would be appreciated.
Thank you,
Russ Ziegler
Comment