Discrepancies (statistical significance) between proportional (eydx) and absolute (dydx) differences in an identical model

[Jose Alcides Santos]

Dear statalist users,

In a study of social mobility in Brazil I am estimating returns to complete tertiary education (versus lower level) on children's income conditional on class origin, race, and cohort. I use a Generalized Linear Model (Gamma distribution; logarithmic link function) with simultaneous interactions between class origin, race, education and cohort and controls only for factors prior to the child's entry into the labor market. It should be considered that the Generalized Linear Model with a logarithmic link function has the advantage of generating predicted values at real levels before applying the link function, that is, the model avoids any transformation bias (Baum, 2021).

Estimates for the most recent ten-year cohort and origin at the social top (on the dimensions of capital, authority, and expert knowledge) on proportional returns to higher education (margins command eydx option) show this unexpected result: the white group reveals a disadvantage relative to the brown & black group of the order of -0.363 in log or -30.4% (p=0.028).

Estimates with the same model and specifications, but in terms of absolute differences (dydx option of margins) yield this seemingly contradictory result: the white group has a significant disadvantage of -980 Reais (Brazilian currency), which represents a difference of -27.9%. However, this difference is uncertain in the statistical significance criterion (p=0.137), even though the confidence interval is strongly inclined towards the negative direction. I am aware of the limitations of the p-value criterion that Clyde Schechter strongly emphasizes.
Note that the discrepancy between the estimates concerns a second-order difference, that is, a (racial) difference between differences (in returns to education) using the mlincom option for Stata.

Interpretation: The estimates of proportional and absolute differences do not differ in the size and direction of the effect, that is, the discrepancies between the estimates basically concern a problem of precision and statistical significance. It seems revealing that the problem with statistical significance only occurs in the post-estimation when bringing the predicted values to the absolute scale in Reais (currency in Brazil) and estimating the difference and standard error for this scale since both estimates were made with the same model with the same specifications. The discrepancies found in the results would arise purely from the fact that one measure expresses a relative change and the other an absolute change. As long as the effect is strong and there is no shortage of cases, it seems reasonable to assume that incomes in absolute values show greater variability (particularly at higher income levels), which may affect the standard error and statistical significance. This variability problem could be aggravated by an estimate of the difference between differences with the respective levels of variability underlying the differences.

Would this be the main conclusion to be emphasized? That's what I have concluded.
In substantive and statistical terms, the striking evidence is the size and direction of the effect, which is convergent in both estimates and shows that in origin at the social top, the returns to higher education are clearly unfavorable to the white group in the most recent cohort.

Would this alternative conclusion be more appropriate?
Both results are valid in what they are expressing and must be considered in the light of the characteristic of each measure (absolute and relative) and the type of manifestation of the phenomenon under investigation.

Would another interpretation be more adequate?
Could this discrepancy have another source? What would this source be and what is its implication?

Given the relevance and substantive implication of the evidence and the issue, all comments are welcome!

[Jose Alcides Santos]

As an elaboration of the issue discussed in the original post I include in the paper an additional counter-argument and a response to it in these terms:

“Although the logarithmic scale is used to obtain a better fit to the data and the estimates interpreted in relative terms, by contracting more strongly the higher values of the distribution, the scale artificially decreases the appearance of inequality. Furthermore, the linear function of a variable in logarithm changes the distribution of the error term since multiplication on the original scale becomes addition on the logarithmic scale (Hao and Naiman, 2007, p. 78-79). However, in the case in question, returns to complete higher education are estimated for the origin at the social top, where values are typically higher for both groups and, as seen, both proportional measures and in original values show a strong advantage unfavorable to the white group or favorable to the brown (pardo) and black group”.

Furthermore, in the study about to be finalized (if no other question arises) an argument had already been developed in order to make the result found and its implication intelligible in these terms:

“This benefited portion of the brown (pardo) and black group that has advantages to transmit to their children, however, represents a smaller proportion or a more selective group due to the inequalities given in the distribution of class origin, education and race. The emerging pattern of racial distance relative to the higher education premium, on the other hand, was more associated with a narrowing of discrepancies in educational returns within the white group. Those who come from a privileged background in the brown (pardo) and black group, in the comparison between peers, now have more to gain when they reach higher education and/or less to gain or more to lose if they end up with lower education. The opposite would have occurred in the white group with a privileged background, on average, as higher education guarantees less gains and/or lower education offers more to retain or less to lose in proportional terms. In origin at the social top, the proportional returns of higher education versus below were less unequal within the white group”.

Statistical, substantive and interpretation issues of empirical evidence are involved in the solution considering the resources and results provided by the Stata program and its margins command.

Has this phenomenon already been observed in a study of social mobility in contemporary society in a country with a considerable non-white racial or ethnic group in an equivalent situation of disadvantage?

Reference cited:
HAO, Lingxin; NAIMAN, Daniel Q. (2007), Quantile regression. Thousand Oaks, Sage.

[Sergey Alexeev]

Dear Jose, I have the same issue but in the opposite direction.

. margins, dydx(HN)

Average marginal effects    Number    of    obs     =    1,108
Model VCE    : Robust

Expression   : Linear prediction, predict()
dy/dx w.r.t. : HN

                
Delta-method
dy/dx   Std. Err.      t    P>t        [95% Conf.    Interval]
                
HN     265.909   123.5084     2.15    0.039        14.01202    517.806

Meanwhile

 
. margins, eydx(HN) 

Average marginal effects    Number    of    obs     =    1,108
Model VCE    : Robust

Expression   : Linear prediction, predict()
ey/dx w.r.t. : HN

                
Delta-method
ey/dx   Std. Err.      t    P>t        [95% Conf.    Interval]
                
HN    .4673261    11.0444     0.04    0.967        22.05788    22.99253

I have outliers in my data when OLS is used, I am thinking it is related to that somehow.

Also, if I replace the OLS estimator, which shows a statistically significant point estimate of 265.909 , with the PPML estimator, I have a statistically insignificant point estimate of 1.77. Here are the dydx margins after PPML.

ppmlhdfe ///
ED_Metro /// 
HN ///
, ///
a( ///
i.qt c.t##i.st ///
T_GS_HasLaw ///
pdmp doctorshopping painclinic ///
physicianexam pharmacistverification requireid ///
trpf ///
) ///
cluster(st) d
************************** 
margins, dydx(HN)


Average marginal effects                        Number    of    obs     =    1,108
Model VCE    : Robust

Expression   : Predicted mean of ED_Metro, predict()
dy/dx w.r.t. : HN

            
Delta-method
dy/dx   Std. Err.      z    P>z        [95% Conf.    Interval]
            
HN    1.770374   54.32453     0.03   0.974        104.7038    108.2445

So, I reckon OLS eydx and dydx are so different because of the outliers, but I am not sure.

I attached the data, and the code to replicate the issue is

clear all
use HCUP_Data.dta
*********************************************************
reg /// 
ED_Metro ///
HN ///
i.qt ///
c.t##i.st ///
T_GS_HasLaw ///
pdmp doctorshopping painclinic ///
physicianexam ///
pharmacistverification requireid ///
trpf, cluster(st)
estimates store X
estimates restore X
margins, dydx(HN)
margins, eydx(HN)