Issue with predicting efficiency using sftfe

[Alexander Kupfer]

Hello,

I'm wondering whether anyone has experienced a similar issue when using the sftfe command to run a stochastic frontier analysis and subsequently predict efficiency using predict xyz, jlms.

When I run the model with my data, the stochastic frontier output looks complete, and there are no missing standard errors. However, when I try to predict the efficiency scores, Stata only generates missing values ("xxx missing values generated") without providing any further error message.

Has anyone experienced a similar issue or know what might be causing this?

I'm also happy to share my data if that helps!

Thank you!

Alexander

[Choonjoo Lee]

In my case, the most
important cause is a numerical underflow in the JLMS calculation performed by
sftfe's prediction program.

For the half-normal and truncated-normal models, sftfe_p.ado calculates the
conditional inefficiency using a ratio of the following form:

normalden(-z) / normal(z)

When z is sufficiently negative, Stata evaluates normal(z) as exactly zero.
In the Stata version used for this experiment, this occurred at approximately
z < -37.52. The numerator can also underflow to zero, so the expression becomes
0/0 and the resulting JLMS efficiency prediction is missing.

1. With ordinary simulated panel data, sftfe produced no missing JLMS
efficiency predictions.

2. After introducing very large positive within-panel frontier residuals,
predict ..., jlms reported:

50 missing values generated

3. The z values for those missing observations had a mean of approximately
-344.64, far beyond the numerical limit at which normal(z) becomes zero.

4. A separate numerical check showed that normal(z) remained positive through
approximately z = -37.50 and became zero from approximately z = -37.52.

The numerical problem can arise under one or more of the following conditions:

* The dependent variable or an explanatory variable contains very large
outliers.

* Variables are expressed in unnecessarily large units.

* The estimated noise standard deviation, e(sigma_v), is very close to zero.

* The ratio of e(sigma_u) to e(sigma_v) is extreme.

* Some observations lie extremely far on the unexpected side of the estimated
frontier. For a production frontier, unusually large positive frontier
residuals are particularly relevant. The direction is reversed for a cost
frontier.

* The data were changed after estimation and predict was then run using the old
estimation results.

* predict was applied to observations outside e(sample). Missing predictions
outside the estimation sample are expected and do not necessarily indicate a
numerical failure.

Run these commands immediately after estimating the model:

predict double te if e(sample), jlms
count if missing(te) & e(sample)

display e(sigma_u)
display e(sigma_v)

predict double xb if e(sample), xb
predict double ahat if e(sample), alpha

gen double resid = DEPENDENT_VARIABLE - ahat - xb if e(sample)
summarize resid, detail

Replace DEPENDENT_VARIABLE with the actual dependent variable used in the
sftfe model.

If

count if missing(te) & e(sample)

returns zero, the missing values reported by predict are merely observations
outside the estimation sample.

If JLMS predictions are missing within e(sample), inspect the residuals for
extreme observations and compare e(sigma_u) with e(sigma_v). Missing values
within the estimation sample are consistent with numerical underflow in the
inverse-Mills-ratio calculation described above.

The behavior reported in the question can therefore be reproduced even when
the estimation output appears complete and all reported standard errors are
present. Complete estimation output does not guarantee that the subsequent
JLMS formula is numerically well behaved.

Without the questioner's data and exact estimation command, it is not possible
to identify the particular observation or parameter responsible in that case.
Nevertheless, the mechanism that produces the missing predictions has been
reproduced and can be diagnosed using the commands above.

Stata: Stata/MP 16.1
sftfe: version 1.2.9, 19 October 2022
sftfe_p.ado: version 1.0.1, 11 March 2015

Good luck,
Choonjoo