Dear all
Thanks to Kit Baum, there is a package cgssm available on the SSC.
In the package, there is both a command for use with variables in a dataset (-cgssm-) and an immediate version (-cgssmi-).
Establishing a competency standard for when a certain level of expertise is reached by identifying cut-off points on different performance measures based on rating scores or simulator metrics is an important issue in competency-based learning.
One method to set such standards is the contrasting groups' standard-setting method. It is a participant-based method where the performance of a certain procedure is evaluated between participants of different expertise levels, e.g., novices and experts.
Using the contrasting groups' method, the cut-off point is set by identifying the intercept of two normally distributed curves that represent the score distributions of the groups defined by their level of expertise. After a pass/fail score is defined, the percentage of false positives and negatives can be calculated to explore the consequences of the test.
Traditionally, these false positives and false negatives are calculated based on the observed number of individuals who passes or fails a test. However, validity studies often include only a small number of participants. These small numbers make the rate of false positives and false negatives sensitive to outliers.
Instead, using the normally distributed curves that represent the score distributions of the groups defined by their level of expertise, the theoretical false negatives, and theoretical false positives can be calculated.
What is new in -cgssm- and -cgssmi- in contrast to the reference is that the cut-off is found using an exact formula based on solving a polynomial of first or second-order degree.
A reference is Jørgensen, M., Konge, L. & Subhi, Y. Contrasting groups' standard setting for consequences analysis in validity studies: reporting considerations. Adv Simul 3, 5 (2018). https://doi.org/10.1186/s41077-018-0064-7
Requires: Is developed and tested in Stata 12.
Example:
The experts' scores did have a mean of 50 and a standard deviation of 10. The mean and the standard deviation of the novices are 20 and 5, respectively.
The immediate command gives:
Enjoy
Thanks to Kit Baum, there is a package cgssm available on the SSC.
In the package, there is both a command for use with variables in a dataset (-cgssm-) and an immediate version (-cgssmi-).
Establishing a competency standard for when a certain level of expertise is reached by identifying cut-off points on different performance measures based on rating scores or simulator metrics is an important issue in competency-based learning.
One method to set such standards is the contrasting groups' standard-setting method. It is a participant-based method where the performance of a certain procedure is evaluated between participants of different expertise levels, e.g., novices and experts.
Using the contrasting groups' method, the cut-off point is set by identifying the intercept of two normally distributed curves that represent the score distributions of the groups defined by their level of expertise. After a pass/fail score is defined, the percentage of false positives and negatives can be calculated to explore the consequences of the test.
Traditionally, these false positives and false negatives are calculated based on the observed number of individuals who passes or fails a test. However, validity studies often include only a small number of participants. These small numbers make the rate of false positives and false negatives sensitive to outliers.
Instead, using the normally distributed curves that represent the score distributions of the groups defined by their level of expertise, the theoretical false negatives, and theoretical false positives can be calculated.
What is new in -cgssm- and -cgssmi- in contrast to the reference is that the cut-off is found using an exact formula based on solving a polynomial of first or second-order degree.
A reference is Jørgensen, M., Konge, L. & Subhi, Y. Contrasting groups' standard setting for consequences analysis in validity studies: reporting considerations. Adv Simul 3, 5 (2018). https://doi.org/10.1186/s41077-018-0064-7
Requires: Is developed and tested in Stata 12.
Example:
The experts' scores did have a mean of 50 and a standard deviation of 10. The mean and the standard deviation of the novices are 20 and 5, respectively.
The immediate command gives:
Code:
cgssmi, msd(50, 10 \ 20, 5) graph
Enjoy
Comment