advice on reporting for predicted probabilities with "new statistics" (CIs, effect sizes, no mention of statistical significance)

[MJ Smith]

Below is a chart generated after a multinomial logit model with: predicted probabilities, observed proportions, and the 95% CIs around predicted probabilities. Model used cluster SE's to account for the same person in multiple years.

The goal of my analysis is to determine whether changes in proportions of male, female, and unknown over time are practically significant (seems like there is movement away from asking if differences are statistically significant). I have 8 of these charts for different positions/departments so would take the same reporting approach for each.

I plan to present this chart and below a table which shows: predicted probabilities for 2014 v. 2008 for each category, difference in predicted probabilities between 2014 and 2008, and 95% CIs around the difference. Example below:

	Predicted Probabilities
	2014	2008	Diff: 2014-2008	95% CI
M	0.26	0.15	0.11	[0.07, .15]
F	0.66	0.71	-0.05	[ -.10, -.0008]
PNA	0.09	0.15	-0.06	[ -.09, -.03]

Then, I would add a sentence summarizing the finding, e.g. above "Conclusion: the differences in predicted probabilities for M, F, and PNA (0.11, -0.05, -0.06) are practically significant."

Question: any impressions/ comments/criticisms on this approach of reporting?

Any thoughts would be greatly appreciated.

[Mike Lacy]

I'd suggest you attend more to the substantive properties of your findings , i.e., do the point estimates indicate differences big enough that they would matter in the relevant real world context. I'd recommend saying something in the direction of: "The estimated difference is large enough that you could 1) get rich by betting on it; 2) use it to save 1,000 lives; or 3) whatever matters in your context." <grin> What you are terming "practical significance" here sounds to me close to "statistical significance" in disguise, and the differences you found might (for all I know) be so small as not to matter *practically* at all. Paraphrasing an apocryphal quote told to me years ago: Your point estimate is your finding, the CI is how much you believe it.

[MJ Smith]

Thank you Mike! I think you hit the nail on the head with your response. I will try to think about what matters in the real world context, although it's a bit of a challenge. I also realized that since I am interested on the trend, I should have reported the marginal effects for the year variable (difference in the probability of each of the outcome level associated with a unit change in each year)

For the 3 categories of gender, an increase in year is associated with changes of 0.02, -0.008, and -0.01 for male, female, and unknown. I would at least consider the 0.02 change per year "practically significant." I will ponder on it more. Thank you!!

	ME	95% CIs
M	0.02	[.01, 0.02]
F	-0.008	[-0.02, 0.0003 ]
U	-0.01	[-0.02,-0.005 ]