Statalist - Forums

Getting confused in selecting Optimal Lag

Radoyan Hossain — Fri, 07 Aug 2026 19:05:47 GMT

i am struggling in selecting optimal lag for Ardl model. my model is based on 33 observations . for this which command should i run for selecting optimal lag am getting confused Alot , i tried to take help from chatgpt but the provided ans are ambiguos.
therefore i would be glad if the precise command for lag selection is given to me keeping in mind of 33 observations of my model with 7 variables.best regards.

Method of using interactive dummy in PVAR model

Binita Dey — Fri, 07 Aug 2026 16:52:38 GMT

Dear Statalist members,

I am attempting to incorporate an interactive dummy variable into a Panel VAR (PVAR) model estimated using GMM, and I would appreciate some guidance on whether my approach and Stata commands are appropriate.

I consider the following equation:
Y_it = Y_(it-1) + D_it X_(it-1) + u_it
where (D_{it}) is a dummy variable constructed based on the change in (X). Specifically,

(D_{it}=1) if (\Delta X_{it}>0) (i.e., (X) increases), and
(D_{it}=0) if (\Delta X_{it}<0) (i.e., (X) decreases).

Thus, I intend (D_{it}X_{it-1}) to capture whether the effect of the lagged (X) on (Y) differs depending on whether (X) is increasing or decreasing.

I estimate the model as a Panel VAR using GMM, treating both (Y) and (X) as endogenous. My Stata commands are:
pvar y Dx1 Dx2 Dx3 Dx4, lags(1) instlags(1/6)
pvargranger
pvarirf, oirf mc(200) byoption(yrescale) porder(y Dx1 Dx2 Dx3 Dx4)
My questions are:

Is this specification an appropriate way to implement an interactive dummy variable in a PVAR framework?
Does the pvar command correctly estimate the model when the interaction term is constructed as (D_{it}X_{it-1}), while (Y) and (X) are treated as endogenous?
Are the pvarirf commands appropriate for obtaining impulse-response results for this specification?
If this approach is not appropriate, what would be the recommended way to incorporate and interpret an interactive dummy effect in a PVAR estimated using GMM?

I would particularly appreciate guidance on whether the commands above actually estimate the intended interactive effect, or whether a different specification or estimation procedure is required.

Thank you.

sgmediation installation failed - "connection timed out" and "r(199)" error

Ahmed Zayyanu Hudu — Fri, 07 Aug 2026 10:56:10 GMT

Dear Statalist members,

I am trying to run mediation analysis for my dissertation research using Stata 15/IC on Windows 7. I need to install `sgmediation` to perform Sobel-Goodman mediation tests.

I tried:

ssc install sgmediation

But I got the following error:

"connection timed out" / "package not found"

I also tried:

net from http://fmwww.bc.edu/RePEc/bocode/s/
net install sgmediation

Still the same issue.

My internet connection is working fine. I can browse websites without problems.

What I need:
- I want to test whether Tax Reform (TREF) mediates the relationship between Leverage, Profitability, and Liquidity on Corporate Tax Planning (CETR).
- I have 120 observations (12 firms × 10 years).

When I tried SEM as an alternative:

sem (CETR <- Leverage TREF) (TREF <- Leverage)

Stata gave this error:

"model not identified; no paths from latent variable CETR to observed variables" r(503)

I typed "estat teftects" (which was a typo) and got "r(321)" error. I now know it should be "estat teffects".

My questions:
1. How can I successfully install sgmediation?
2. If sgmediation cannot be installed, how do I correctly specify the SEM model for mediation?
3. Is there an alternative way to test mediation without sgmediation?

What I have tried:
- ssc install sgmediation
- ssc install sgmediation, replace
- net from http://fmwww.bc.edu/RePEc/bocode/s/
- net install sgmediation
- Checking Stata's ado path

Any advice would be greatly appreciated.

Thank you,
Hudu Zayyanu Ahmed
Stata 15/IC, Windows 7

panel dataset

Matt Piercy — Thu, 06 Aug 2026 21:58:28 GMT

I am looking for a good panel dataset to explain GEE. McCuloloch, Searle and Neuhaus mention a dataset modelling alcohol consumption and low birth weight. There is the stata dataset by Hosemer and Lemeshow called lbw but it is not panel data. Is anyone aware of a publically accessible teaching dataset along these lines?

Model for dataset with a very small number of incidents.

Luis Mijares Castaneda — Wed, 05 Aug 2026 20:36:56 GMT

Hello, I was wondering if anyone could provide advice on model selection for my dataset. I have data from a longitudinal survey with five waves: one baseline wave and four follow-up waves. My goal is to model post-baseline home eviction rates.

The challenge is that only 18 participants reported experiencing at least one home eviction during follow-up. I use the number of home evictions as the outcome variable and the total number of post-baseline waves completed as the exposure (offset) in a Poisson regression model, along with the covariates listed below.

My concern is that the number of non-zero outcomes is so small that the model appears to be overfit, resulting in very wide confidence intervals. Could anyone recommend an alternative modeling approach for count data with such a small number of events, or suggest strategies for handling this type of sparse outcome?

c.ppage ///
i.biosex ///
i.race_alt ///
i.education ///
i.region ///
i.income ///
i.Personal_debt ///
c.sf8pcs ///
c.sf8mcs ///
i.asud ///
i.asmi ///
i.housetype_alt ///
i.employment, ///
exposure(total_years) ///

Code:



 tab total_years

  Number of |
      waves |
 present in |
 the survey |      Freq.     Percent        Cum.
------------+-----------------------------------
          1 |         91       10.87       10.87
          2 |        110       13.14       24.01
          3 |        113       13.50       37.51
          4 |        523       62.49      100.00
------------+-----------------------------------
      Total |        837      100.00

. tab any_eviction_count

      Total |
  number of |
  evictions |
       from |
Year01-Year |
         04 |      Freq.     Percent        Cum.
------------+-----------------------------------
          0 |        819       97.85       97.85
          1 |         13        1.55       99.40
          2 |          1        0.12       99.52
          3 |          2        0.24       99.76
          4 |          2        0.24      100.00
------------+-----------------------------------
      Total |        837      100.00

Question about mixed-effects models after linear interpolation of longitudinal data

Oana Vaida — Tue, 04 Aug 2026 12:00:06 GMT

Dear all,
I would appreciate your advice regarding the appropriate analysis of a longitudinal dataset in Stata.

I have a cohort of 221 patients with repeated DXA Z-score measurements obtained at irregular follow-up times, resulting in 2,373 observed Z-score measurements. To standardize follow-up at annual intervals (as required for a publication), I linearly interpolated Z-scores between consecutive observed measurements, producing an anniversary-year dataset with 4,498 observations from 197 patients. Patients with only one observed measurement could not be interpolated and were therefore excluded from the anniversary-year dataset.

I am fitting the following mixed-effects model:

mixed z_score c.anniv_year##(i.z_sitespine1hip2radius3 i.gender1female i.splenectomy1yes i.n370s_homozygous1yes i.eversmoked i.typetype11type30 c.interval_symptom_tx c.zimran_score c.ert_date1 c.ageatthestartoftreatment c.anniv_bmi c.vitamind_anniv ib2.erttype1alglucerase2velagluceras c.ukg i.bisphosphonatesanniv_year0no1yes) || subject_id: c.anniv_year, covariance(unstructured) reml dfmethod(kroger)

My question concerns statistical inference. Since many of the Z-scores were created by deterministic linear interpolation rather than being directly observed, I am concerned that treating all anniversary-year observations as regular measurements may underestimate uncertainty.

I initially wondered whether mi estimate should be used, but I understand that mi estimate is intended for multiple imputation rather than deterministic interpolation

Could you please advise:

Is there any recommended way in Stata to account for the additional uncertainty introduced by deterministic linear interpolation when fitting a mixed-effects model?
Is using reml dfmethod(kroger) considered sufficient for this setting, or is there another recommended approach?
Should I I fit the mixed model using only the original observed measurements and compare it with the interpolated mixed model?

Any guidance or references to recommended practice would be greatly appreciated.

Thank you very much for your time and assistance

Best,
Oana

New commands on SSC: atip_reglas/atip_score/atip_volcano/atip_mahal (outlier detection) and mmd_2s (distributional comparison)

Andrés Lahur Talavera Cuya — Mon, 03 Aug 2026 20:01:46 GMT

Dear Statalist,
I would like to announce six new commands now available on SSC, developed while building a data-quality pipeline for Peru's National Agricultural Survey (ENA) at INEI. All are validated by simulation (results and methodology documented in each help file), and none require any commercial license beyond base Stata.
Outlier detection suite (four commands, designed to work together):
- atip_reglas: flags outliers within groups using an ensemble of five classic rules (mean+/-3SD, mean+/-2SD, z-score, Tukey IQR, univariate Mahalanobis). Uses median/MAD instead of mean/SD by default, which is far more robust under contamination -- simulation shows recall rising from 27.4% to 95.5% at 20% contamination.
- atip_score: generalizes atip_reglas into two separate continuous axes (evidence: how many rules fired; magnitude: how far the value is), instead of a single binary flag -- useful for selective-editing style triage rather than a hard cutoff.
- atip_volcano: volcano-style plot (magnitude vs. evidence) of atip_score output, without inventing a formal p-value for what are heuristic rules.
- atip_mahal: genuine multivariate outlier detection (2+ variables) via Mahalanobis distance, using the Minimum Covariance Determinant estimator (Rousseeuw and Van Driessen 1999) by default -- classical covariance collapses under masking (multiple simultaneous outliers), MCD does not. Simulation: 100% recall at 20% contamination with MCD vs. 0% with the classical estimator. Includes an optional tolerance-ellipse plot (k=2) that can overlay both classical and MCD ellipses to visualize masking directly.
Distributional comparison:
- mmd_2s: two-sample test based on Maximum Mean Discrepancy (RBF kernel), for comparing whether two samples come from the same distribution -- more general than a t-test (catches differences in variance/shape, not just location) and complementary to a Kolmogorov-Smirnov test. Weighting is fully optional via Stata's native [pweight/aweight/iweight] syntax.
Install any of them with:

HTML Code:

 ssc install atip_reglas

HTML Code:

 ssc install atip_score

HTML Code:

 ssc install atip_volcano

HTML Code:

 ssc install atip_mahal

HTML Code:

 ssc install mmd_2s

All help files include real-data examples (sysuse auto/nlsw88) plus a companion command, SRIINEI, for downloading public microdata directly from Peru's official statistics portal (also on SSC, ssc install sriinei) -- useful if anyone wants to reproduce the real-data examples themselves.
Comments, bug reports, and suggestions welcome.
Best regards,
Andres Talavera Cuya
atalaveracuya@gmail.com

SSC Archive

KitBaum — Sun, 02 Aug 2026 17:21:53 GMT

Details of SSC Archive downloads for July 2026 are now available via sec hot. As a reminder, you can access download statistics for a particular author's packages with sec hot, author(last name). That can be combined with the n(#) option to view more than 10 packages.

Rankings for the last three months for 1,555 authors are available at http://repec.org/docs/sscAuthors.html

Updates to the SSC Archive will be suspended August 5-19 during travels.

SSC Archive, July 2026

KitBaum — Sat, 01 Aug 2026 14:49:58 GMT

With wider use of AI translation tools, an amazing 64 packages were added to the SSC Archive in July, and 46 existing packages were revised. The details are available at ssc whatsnew. Download statistics have also increased; results for July will be available in a day or so.

Boot CV for 'fbaardl' under "aardl" package

Abhijit Lahiri — Sat, 01 Aug 2026 10:08:43 GMT

It would have been nice if the Boot CV for 1% and 10% are also given in addition to the 5% level (which is available) when we use the 'fbaardl' option under the "aardl" package. This is necessary for reporting in the research paper.

String variables with value labels!!

Krishanu Karmakar — Fri, 31 Jul 2026 20:59:35 GMT

As far as I understand string variables cannot have value labels. However, I have received a dataset with string variables which shows (output of -describe- command) that the string variables have value labels attached to them. Moreover, the value labels have two or more different labels (texts) attached to the same value 0 (while the variables have no zero values). Could anyone share any thought about what might be going on here? Or have any of you encountered this problem ever? I am sharing my code and its output below (I cannot even understand how to give a data example in this context as I do not know how to create string variables with value labels, so pardon me.) Note that the problem remains the same even if I save the data in the newer dta format 118.

Code:

. use "D:\L02_Demographic_particulars.dta" 

. version
version 19.5

. dtaversion "D:\L02_Demographic_particulars.dta"
  (file "D:\L02_Demographic_particulars.dta" is .dta-format 115 from Stata 12)

. describe Sector Gender

Variable      Storage   Display    Value
    name         type    format    label      Variable label
--------------------------------------------------------------------------------
Sector          str1    %-9s       Sector0    Sector
Gender          str1    %-9s       Gender2    Gender


. label list Sector0 Gender2
Sector0:
           0 urban
           0 rural
Gender2:
           0 Female
           0 transgender
           0 Male

. tab Sector

     Sector |      Freq.     Percent        Cum.
------------+-----------------------------------
          1 |    292,208       62.63       62.63
          2 |    174,319       37.37      100.00
------------+-----------------------------------
      Total |    466,527      100.00

. tab Gender

     Gender |      Freq.     Percent        Cum.
------------+-----------------------------------
          1 |    240,556       51.56       51.56
          2 |    225,897       48.42       99.98
          3 |         74        0.02      100.00
------------+-----------------------------------
      Total |    466,527      100.00

-reshapehelper- on SSC

eric_a_booth — Fri, 31 Jul 2026 14:37:16 GMT

-reshape- is the command everyone second-guesses or runs to the help file for support (or maybe it's just me)

-reshapehelper- helps with that. It's a command that works out the reshape for you: it reads your variable names, figures out the wide-or-long data structure, and dry-runs the command on a subset copy of your data.
You get back exact, copy/paste-ready -reshape- syntax (stored in `r(cmd)') with a pass/fail verdict. **No LLMs involved ... no, not an April Fool's prank**

• Detects i and j automatically (xtset / tsset / svyset aware) and most data layouts (e.g., long, wide (or even triply-wide) data)
• Tests the reshape before you run it, identifies duplicates ... avoids most r(198) surprises
• Hands you working syntax, not a guess or a panic-run to the .sthlp file

Thanks to KitBaum it's on SSC via:

Code:

ssc install reshapehelper

Some worked examples below. To pressure test this I dug through reshape questions from this community over the past few years and I tried to anticipate possible data shapes or conditions that might break -reshapehelper-. Where the program cannot recursively act to detect the structure or variable patterns (even with your help) it suggests some next steps to diagnose the problem in the output.

More examples in the help file:

Code:

* ---------------------------------------------------------------------------
* T1. Classic wide -> long, numeric suffixes ([D] reshape Example 1)
* ---------------------------------------------------------------------------
clear
input id sex inc80 inc81 inc82
1 0 5000 5500 6000
2 1 2000 2200 3300
3 0 3000 2000 1000
end
reshapehelper
assert "`r(status)'"    == "ok"
assert "`r(direction)'" == "wide2long"
assert r(tested)        == 1
assert strpos(`"`r(cmd)'"', "reshape long inc, i(id) j(year)") > 0
assert `"$reshapehelper_cmd"' == `"`r(cmd)'"'
assert _N == 3 & c(k) == 5          // data untouched

* ---------------------------------------------------------------------------
* T2. Two stubs at once: inc AND ue ([D] reshape Example 1, full)
* ---------------------------------------------------------------------------
clear
input id sex inc80 inc81 inc82 ue80 ue81 ue82
1 0 5000 5500 6000 0 1 0
2 1 2000 2200 3300 1 0 0
3 0 3000 2000 1000 0 0 1
end
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "reshape long inc ue, i(id) j(year)") > 0

* ---------------------------------------------------------------------------
* T3. Classic long -> wide ([D] reshape Example 1 reversed)
* ---------------------------------------------------------------------------
clear
input id year sex inc ue
1 80 0 5000 0
1 81 0 5500 1
1 82 0 6000 0
2 80 1 2000 1
2 81 1 2200 0
2 82 1 3300 0
3 80 0 3000 0
3 81 0 2000 0
3 82 0 1000 1
end
reshapehelper, to(wide)
assert "`r(status)'"    == "ok"
assert "`r(direction)'" == "long2wide"
assert r(tested)        == 1
assert strpos(`"`r(cmd)'"', "reshape wide inc ue, i(id) j(year)") > 0
assert _N == 9 & c(k) == 5

* ---------------------------------------------------------------------------
* T4. String suffixes after an underscore (sysuse bpwide)
* ---------------------------------------------------------------------------
sysuse bpwide, clear
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "reshape long bp_, i(patient) j(period) string") > 0

* ---------------------------------------------------------------------------
* T5. @ mid-name stubs beside plain stubs ([D] reshape Example 7: inc@r + ue)
* ---------------------------------------------------------------------------
clear
input id sex inc80r inc81r inc82r ue80 ue81 ue82
1 0 5000 5500 6000 0 1 0
2 1 2000 2200 3300 1 0 0
3 0 3000 2000 1000 0 0 1
end
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "inc@r") > 0
assert strpos(`"`r(cmd)'"', "ue") > 0
assert strpos(`"`r(cmd)'"', "j(year)") > 0

* ---------------------------------------------------------------------------
* T6. Unbalanced stubs: ue81 does not exist ([D] reshape Example 6)
* ---------------------------------------------------------------------------
clear
input id sex inc80 inc81 inc82 ue80 ue82
1 0 5000 5500 6000 0 0
2 1 2000 2200 3300 1 0
3 0 3000 2000 1000 0 1
end
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "reshape long inc ue, i(id) j(year)") > 0
assert strpos(`"`r(note)'"', "unbalanced") > 0

* ---------------------------------------------------------------------------
* T7. The inc2 trap: a stray same-stub variable ([D] reshape, j() values).
*     The dry run PASSES (reshape happily builds a j=2 group), so the
*     mixed-width caution is the safety net.
* ---------------------------------------------------------------------------
clear
input id sex inc80 inc81 inc82 inc2
1 0 5000 5500 6000 1
2 1 2000 2200 3300 0
3 0 3000 2000 1000 1
end
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(caution)'"', "widths differ") > 0
assert strpos(`"`r(caution)'"', "restrict j") > 0

* ---------------------------------------------------------------------------
* T8. Inconsistent stub names: inc80 / income81 / incm82 (UVA + Stata FAQ).
*     No family forms, so the helper must say so and coach a rename.
* ---------------------------------------------------------------------------
clear
input id sex inc80 income81 incm82
1 0 5000 5500 6000
2 1 2000 2200 3300
3 0 3000 2000 1000
end
reshapehelper
assert "`r(status)'" == "needinfo"
assert `"`r(cmd)'"' == ""

* ---------------------------------------------------------------------------
* T9. String j containing spaces (Statalist r(111) thread): forced j(state)
*     must trigger the pre-clean line, the string option, and a passing test
* ---------------------------------------------------------------------------
clear
input year str12 state pop
2020 "New York" 20.2
2020 "Texas" 29.1
2021 "New York" 19.8
2021 "Texas" 29.5
end
reshapehelper, to(wide) j(state)
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "j(state) string") > 0
assert strpos(`"`r(preclean)'"', "subinstr") > 0
assert _N == 4 & c(k) == 3          // caller's data untouched (incl. spaces)
assert strpos(state[1], " ") > 0

* ---------------------------------------------------------------------------
* T10. Prefix-as-j: qld_p nsw_p vic_p (Statalist "no xij variables found")
* ---------------------------------------------------------------------------
clear
input year qld_p nsw_p vic_p
2018 4.9 7.9 6.4
2019 5.0 8.0 6.5
2020 5.1 8.1 6.6
end
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "@_p") > 0
assert strpos(`"`r(cmd)'"', "string") > 0

* ---------------------------------------------------------------------------
* T11. Duplicate (i, j) pairs block reshape wide (Statalist / manual Ex. 3):
*      the helper must diagnose, count, and hand back the remedy menu
* ---------------------------------------------------------------------------
clear
input id year inc
1 2019 45000
1 2020 47000
2 2019 32000
2 2019 32000
2 2020 33500
end
reshapehelper, to(wide)
assert "`r(status)'" == "needinfo"
assert strpos(`"`r(diagnosis)'"', "duplicates report") > 0
assert strpos(`"`r(diagnosis)'"', "collapse") > 0
assert strpos(`"`r(diagnosis)'"', "concat") > 0

* ---------------------------------------------------------------------------
* T12. Two crossed factors (Statalist animal/level/delay): the helper finds a
*      compound i and widens ONE factor, and its note points to the rest
* ---------------------------------------------------------------------------
clear
input animal s1level s1s2delay s2peakvalue
1 0 50 12.1
1 0 100 13.4
1 0 200 15.2
1 1 50 18.3
1 1 100 19.9
1 1 200 22.4
2 0 50 11.8
2 0 100 12.9
2 0 200 14.7
2 1 50 17.5
2 1 100 19.2
2 1 200 21.8
end
reshapehelper, to(wide)
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "reshape wide s2peakvalue") > 0
assert strpos(`"`r(note)'"', "concat") > 0

* ---------------------------------------------------------------------------
* T13. Doubly wide (UCLA FAQ): two digit runs in the names -> two chained
*      reshapes, both dry-run tested
* ---------------------------------------------------------------------------
clear
input famid ht_k1_t1 ht_k1_t2 ht_k2_t1 ht_k2_t2
1 3.1 3.6 4.0 4.4
2 3.3 3.8 4.1 4.6
3 3.0 3.5 3.9 4.3
end
reshapehelper
assert "`r(status)'"    == "ok"
assert "`r(direction)'" == "doubly"
assert r(tested)        == 1
assert `"`r(cmd2)'"' != ""
assert strpos(`"`r(cmd)'"',  "reshape long ht_k1_t ht_k2_t, i(famid)") > 0
assert strpos(`"`r(cmd2)'"', "reshape long ht_k@_t, i(famid") > 0
assert _N == 3 & c(k) == 5

* ---------------------------------------------------------------------------
* T14. Long-long to wide-wide, step one ([D] reshape second-level nesting):
*      compound i() found, low-cardinality factor left in i() flagged
* ---------------------------------------------------------------------------
clear
input hid str1 sex year inc
1 "f" 90 3200
1 "f" 91 4700
1 "m" 90 4500
1 "m" 91 4600
2 "f" 90 3600
2 "f" 91 3800
2 "m" 90 5100
2 "m" 91 5300
end
reshapehelper, to(wide)
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "reshape wide inc") > 0
assert `"`r(note)'"' != ""

* ---------------------------------------------------------------------------
* T15. Already-long tidy panel, no to(): the helper reads it as long and
*      offers the wide command
* ---------------------------------------------------------------------------
clear
input id year inc
1 80 5000
1 81 5500
1 82 6000
2 80 2000
2 81 2200
2 82 3300
end
reshapehelper
assert "`r(status)'"    == "ok"
assert "`r(direction)'" == "long2wide"
assert r(tested)        == 1

* ---------------------------------------------------------------------------
* T16. Transpose, not reshape (Statalist xpose thread): metrics as rows
* ---------------------------------------------------------------------------
clear
input str12 metric alpha beta gamma
"n"       100 200 150
"mean"    52.1 48.9 50.3
"missing" 3 7 5
end
reshapehelper
assert "`r(status)'" == "needinfo"
assert r(xpose) == 1

* ---------------------------------------------------------------------------
* T17. Wide-long panel honoring xtset: county-year rows with sector columns;
*      i() must include the existing time variable
* ---------------------------------------------------------------------------
clear
input county year emp_manuf emp_retail emp_gov
1 2019 120 340 210
1 2020 115 330 215
2 2019  80 210 150
2 2020  78 220 155
3 2019  60 190 120
3 2020  61 200 118
end
xtset county year
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "reshape long emp_, i(county year)") > 0
assert strpos(`"`r(cmd)'"', "string") > 0
assert strpos(`"`r(note)'"', "SECOND long dimension") > 0

* ---------------------------------------------------------------------------
* T18. User-assisted bare string suffixes ([D] reshape Example 8: incm/incf):
*      stubs()+i()+j() supplied; the dry-run engine discovers the string
*      option by iterating on reshape's own r(498)
* ---------------------------------------------------------------------------
clear
input id kids incm incf
1 0 5000 5500
2 1 2000 2200
3 2 3000 2000
end
reshapehelper, to(long) stubs(inc) i(id) j(sex)
assert "`r(status)'" == "ok"
assert r(tested)     == 1
assert strpos(`"`r(cmd)'"', "reshape long inc, i(id) j(sex) string") > 0

* ---------------------------------------------------------------------------
* T19. Guardrails: shorthand tokens, bad options, empty data
* ---------------------------------------------------------------------------
sysuse bpwide, clear
reshapehelper long                    // bare-token shorthand for to(long)
assert "`r(status)'" == "ok"
capture reshapehelper, to(sideways)
assert _rc == 198
capture reshapehelper, sample(3)
assert _rc == 198
clear
capture reshapehelper
assert _rc == 2000

* ---------------------------------------------------------------------------
* T20. The SMCL suggestion file exists and holds the unwrapped command
* ---------------------------------------------------------------------------
clear
input id inc80 inc81
1 5000 5500
2 2000 2200
end
tempfile junk
reshapehelper, smcl("scratch_suggestion.smcl") replace
assert "`r(status)'" == "ok"
confirm file "scratch_suggestion.smcl"
file open fh using "scratch_suggestion.smcl", read text
local found 0
file read fh line
while r(eof) == 0 {
    if strpos(`"`macval(line)'"', "reshape long inc, i(id) j(year)") local found 1
    file read fh line
}
file close fh
assert `found' == 1
erase "scratch_suggestion.smcl"

* ---------------------------------------------------------------------------
* T21. Edge sweep: hostile small/weird data must never crash, never touch
*      the data, and never leave a stale global
* ---------------------------------------------------------------------------
* (a) one observation, one variable -> graceful checklist
clear
set obs 1
gen x = 1
reshapehelper
assert inlist("`r(status)'", "needinfo", "ok")
assert _N == 1 & c(k) == 1

* (b) all-string dataset -> graceful
clear
input str5 a str5 b
"x" "y"
"z" "w"
end
reshapehelper
assert inlist("`r(status)'", "needinfo", "ok")
assert _N == 2 & c(k) == 2

* (c) strL beside wide stubs -> still suggests, strL skipped in probes
clear
input id inc80 inc81
1 5000 5500
2 2000 2200
end
gen strL comment = "free text " + string(id)
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested) == 1
assert strpos(`"`r(cmd)'"', "reshape long inc, i(id)") > 0
assert c(k) == 4

* (d) a variable named year already exists beside inc80-inc82: the proposed
*     j must dodge the collision and the dry run must still pass
clear
input id year inc80 inc81 inc82
1 1999 5000 5500 6000
2 1999 2000 2200 3300
end
reshapehelper
assert "`r(status)'" == "ok"
assert r(tested) == 1
assert strpos(`"`r(cmd)'"', "j(year)") == 0

* (e) bare stub variable v beside v1-v3 (reshape r(110) territory): composed
*     but honestly reported as blocked after the dry run cannot fix it
clear
input id v v1 v2 v3
1 9 10 11 12
2 8 20 21 22
end
reshapehelper
assert inlist("`r(status)'", "blocked", "needinfo")
assert _N == 2 & c(k) == 5

* (f) widening would build names longer than 32 chars: caution + honest fail
clear
input id year abcdefghijklmnopqrstuvwxyzabcde
1 2018 1.1
1 2019 1.2
2 2018 2.1
2 2019 2.2
end
reshapehelper, to(wide)
assert inlist("`r(status)'", "blocked", "ok")
if "`r(status)'" == "blocked" assert `"`r(caution)'"' != ""

* (g) missing values in the j candidate -> no crash, data untouched
clear
input id year x
1 2018 1
1 2019 2
1    . 3
2 2018 4
2 2019 5
end
reshapehelper, to(wide)
assert inlist("`r(status)'", "ok", "blocked", "needinfo")
assert _N == 5 & c(k) == 3

* (h) an all-missing variable beside normal stubs -> still ok
clear
input id inc80 inc81
1 5000 5500
2 2000 2200
end
gen ghost = .
reshapehelper
assert "`r(status)'" == "ok"

* (i) mixed-case stubs Inc_80/Inc_81: case preserved in the suggestion
clear
input id Inc_80 Inc_81
1 5000 5500
2 2000 2200
end
reshapehelper
assert "`r(status)'" == "ok"
assert strpos(`"`r(cmd)'"', "Inc_") > 0

* (j) value-labeled integer j -> to(wide) works
clear
input id year x
1 1 10
1 2 11
2 1 20
2 2 21
end
label define yl 1 "wave one" 2 "wave two"
label values year yl
reshapehelper, to(wide)
assert "`r(status)'" == "ok"
assert r(tested) == 1

* (k) REGRESSION (adversarial finding): a chained run followed by a
*     single-step run must CLEAR the stale second-step global
clear
input famid ht_k1_t1 ht_k1_t2 ht_k2_t1 ht_k2_t2
1 3.1 3.6 4.0 4.4
2 3.3 3.8 4.1 4.6
end
reshapehelper
assert `"$reshapehelper_cmd2"' != ""
clear
input id inc80 inc81
1 5000 5500
2 2000 2200
end
reshapehelper
assert `"`r(cmd2)'"' == ""
assert `"$reshapehelper_cmd2"' == ""
* ... and a checklist run clears the first global too
sysuse auto, clear
reshapehelper
assert "`r(status)'" == "needinfo"
assert `"$reshapehelper_cmd"' == ""

* (l) varlist restriction scopes the SCAN but not the id hunt
clear
input id inc80 inc81 ue80 ue81
1 5000 5500 0 1
2 2000 2200 1 0
end
reshapehelper inc80 inc81
assert "`r(status)'" == "ok"
assert strpos(`"`r(cmd)'"', "reshape long inc, i(id)") > 0
assert strpos(`"`r(cmd)'"', "ue") == 0

* (m) user errors arrive as clean return codes
capture reshapehelper, i(no_such_var)
assert _rc == 111
capture reshapehelper, to(wide) j(no_such_var)
assert _rc == 111
clear
input id z1 z2
1 1 2
2 3 4
end
reshapehelper, to(long) stubs(qqq) i(id)
assert inlist("`r(status)'", "blocked", "needinfo")

* ---------------------------------------------------------------------------
* T22. REGRESSION (adversarial round 2): a double-quote inside a string j
*      VALUE must not crash the caller (it broke the 32-char length test)
* ---------------------------------------------------------------------------
clear
input id str8 size measure
1 "5in" 10
1 "6in" 20
2 "5in" 11
2 "6in" 21
end
replace size = subinstr(size, "in", char(34), .)   // values become  5"  6"
capture reshapehelper, to(wide) j(size)
assert _rc == 0                                     // no escaped r(132)/r(198)
assert inlist("`r(status)'", "ok", "blocked", "needinfo")
assert _N == 4 & c(k) == 3                          // data untouched
* a backtick in a j value must also be safe
clear
input id str8 code measure
1 "a" 10
1 "b" 20
2 "a" 11
2 "b" 21
end
replace code = "a" + char(96) + "b" if code == "a"
capture reshapehelper, to(wide) j(code)
assert _rc == 0
assert inlist("`r(status)'", "ok", "blocked", "needinfo")

* ---------------------------------------------------------------------------
* T23. REGRESSION: the "i() still contains the factor" note must NOT fire on a
*      terminal single-id panel (it was misreading the id as a leftover factor)
* ---------------------------------------------------------------------------
clear
input id year inc
1 80 5000
1 81 5500
2 80 2000
2 81 2200
3 80 3000
3 81 3300
end
reshapehelper, to(wide)
assert "`r(status)'" == "ok"
assert strpos(`"`r(cmd)'"', "reshape wide inc, i(id) j(year)") > 0
assert `"`r(note)'"' == ""                          // no spurious factor note
* but a genuine compound-i leftover factor STILL earns the note (T14 shape)
clear
input hid str1 sex year inc
1 "f" 90 3200
1 "f" 91 4700
1 "m" 90 4500
1 "m" 91 4600
2 "f" 90 3600
2 "f" 91 3800
2 "m" 90 5100
2 "m" 91 5300
end
reshapehelper, to(wide)
assert "`r(status)'" == "ok"
assert `"`r(note)'"' != ""

* ---------------------------------------------------------------------------
* T24. REGRESSION: r(direction) only ever returns a DOCUMENTED value
* ---------------------------------------------------------------------------
clear
input a b c
1 2 3
4 5 6
end
reshapehelper, to(long)
assert inlist("`r(direction)'", "wide2long", "long2wide", "doubly", "unknown")
clear
input id x y
1 1 2
2 3 4
end
reshapehelper, to(wide)
assert inlist("`r(direction)'", "wide2long", "long2wide", "doubly", "unknown")

* ---------------------------------------------------------------------------
* T25. Sparse panel: an id that DOES uniquely identify rows with year but
*      barely repeats is set aside by the plausibility bar; the checklist must
*      name it as a possible sparse-units cause and expose it in r(sparse)
* ---------------------------------------------------------------------------
clear
input id year x
1 2019 10
2 2019 20
3 2019 30
4 2019 40
5 2020 50
1 2020 11
end
reshapehelper, to(wide)
assert "`r(status)'" == "needinfo"
assert "`r(sparse)'" == "id"
assert `"`r(cmd)'"' == ""
* forcing the flagged id makes it resolve
reshapehelper, to(wide) i(id) j(year)
assert "`r(status)'" == "ok"
assert strpos(`"`r(cmd)'"', "reshape wide x, i(id) j(year)") > 0
* a normal (non-sparse) panel must NOT set r(sparse)
clear
input id year x
1 80 5
1 81 6
2 80 7
2 81 8
end
reshapehelper, to(wide)
assert "`r(status)'" == "ok"
assert "`r(sparse)'" == ""

Data file not opening

Sumit Mahajan — Fri, 31 Jul 2026 11:22:52 GMT

Hello!
I am new to STATALIST. I have subscription of STATA 19 SE. When I am trying to open a STATA file from another source, it is not opening and giving this error message "dataset too large
This dataset was created by Stata/MP and has more variables than your Stata can handle."
How can I overcome this problem with the current STATA subscription?

logistic_scores is now available on SSC

Nick Cox — Fri, 31 Jul 2026 09:32:22 GMT

Thanks as always to Kit Baum, a new command logistic_scores is now available from SSC. Stata 9 is required.

It's all about generating logistic scores, a transformation of graded ordinal variables best publicised in Frederick Mosteller and John W. Tukey's book Data Analysis and Regression (1977). The idea can also be seen in a long paper by Tukey written in 1961 but not published until 1986.

This is a slow-running project, I posted some Mata code on Statalist in 2007, got around to writing a command in 2015, and tweaked it in 2022, and finally wrote up a help file this month.

The definitions and even the derivation of the recipe are all in the help file and an ancillary document and its references.

Take any example of such a variable. Many Stata users will know that the auto dataset includes rep78 -- which is repair record 1978 on a five-point scale with numeric values 1 to 5.

Here is a graph to show what this command produces for that variable.
Array

And here as a second example are some anonymous or at least unexplained data from Mosteller and Tukey, included so that I could check that the command produced the same results.
Array
People not interested in this command might nevertheless be interested in the small graphical tricks for producing such displays. Complete code is included in the help file.

In two sentences, perhaps not intelligible until you've read the longer accounts: Cumulative probabilities define breakpoints between slices of an underlying distribution taken be standard logistic. Scores are then returned as the centres of gravity of those slices.

How far the original variables say something about whatever is being graded, or something about the people doing the grading, is a topic for discussion. Either way or both ways, they are the data we have.

Many people reading this will have been exposed to a classification of measurement scales or levels into nominal, ordinal, interval, and ratio, That classification was revisited repeatedly in articles (and a book) by the psychologist Stanley Smith Stevens from 1946 to 1975 (the last publication was posthumous). The way it is cited varies across the entire range from purely descriptive to strongly prescriptive -- or even strongly proscriptive.

One common argument runs that for variables like this with values 1 2 3 4 5 are, as said, just ordinal grades conventionally reported as successive integers, but not otherwise to be taken literally, which means numerically. We don't have licence, the argument runs, to treat intervals such as 2 minus 1 or 3 minus 2 as equal or even as meaning anything in particular. So the scale lacks the property that intervals can be taken literally. Still less can we take it that 2 is twice 1, 4 is twice 2, and so on. Ratios can be formed, but not legitimately. So far, so correct, but does that affect what we can do (should do) statistically? Have you heard the line that only medians make sense for such variables, and that means can't be used validly? Do you agree in principle? In your statistical practice?

Despite this caution much of the practice of data analysis ignores or even defies any purism about measurement scales. summarize (for example) will without a murmur of discontent take such a variable and calculate not just means, but also SD, skewness and kurtosis, and so forth. Is that valid or useful? Stata's attitude, as with most comparable software, is that yours is the choice to make, and yours is the responsibility. More broadly, any number of websites post average review scores, not to mention what many universities do with student grades or marks.

On a less abstract level: How do people use such variables? Perhaps the easiest decision to make is to treat a variable like rep78 through a set of indicator variables when it is regarded as a predictor or covariate. That answers, or avoids, qualms about measurement scale. It is letting the data show how the variable works in practice. Depending on results, researchers might leave the variable as indicators, to be emboldened to replace it with the data as it came, to use a quadratic instead, and so on.

Logistic scores are also a variant on the letting the data show how the variable works in practice. The easy part to buy is that cumulative probabilities are well defined if anything is with such a variable. The last cumulative probability is identically 1. Otherwise we have (usually) one fewer cumulative probability than categories acting as breakpoints between slices of a distribution.

More arbitrary is positing an underlying logistic distribution. The argument for that pivots on simplicity and tractability: the scores for each slice fall out as a result of a moderately simple calculation, as now done for you by this command.

I wouldn't usually use logistic scores as a predictor without checking whether alternatives, notably indicator variables, give broadly similar results.

What about logistic scores as an outcome? Recently I heard of a variable like rep78 being used as an outcome in multiple regression as it came. I had to suppress as commentary what I suppose is a fairly standard line that you should be trying something quite different instead, say ordinal logit -- or ordinal probit, or any number of alternative ordinal models from categorical data analysis. I readily confess to a chicken and egg reluctance to get involved with these models. That is, you need to use them routinely before you can wrap your head around which one to use, what to look for, and how to explain them. So, although there is always a need to check whether linearity works well enough I don't want to write off that kind of application, and now have an alternative to suggest.

New version of kmest on SSC

Roger Newson — Fri, 31 Jul 2026 00:39:53 GMT

Thanks as always to Kit Baum, a new version of the kmest package is now available for download from SSC. In Stata, use the ssc command to do this, or adoupdate if you already have an old version of kmest.

The kmest package is described as below on my website. The new version returns the results e(wtype) and e(wexp), containing the type and expression of the weights used (if any). It has also been given additional test sequences in its certification script.

Best wishes

Roger

-----------------------------------------------------------------------------------------------------------------------------------------------------------
package kmest from https://www.rogernewsonresources.org.uk/stata16
-----------------------------------------------------------------------------------------------------------------------------------------------------------

TITLE
kmest: Compute Kaplan-Meier survival probabilities and/or percentiles as estimation results

DESCRIPTION/AUTHOR(S)
kmest is intended for use in a survival time dataset set up by stset.
It computes Kaplan-Meier survival probabilities (as computed by sts
generate) for a list of times (sorted in ascending order), and/or
Kaplan-meier percentiles for a list of percents (srted in ascending
order), and saves them as estimation results, without a variance
matrix. kmest is intended for use with the bootstrap prefix, or
possibly with the jackknife prefix, to create confidence intervals for
the Kaplan-Meier survival probabilities and/or percentiles, possibly
allowing for clustering and/or sampling-probability weighting.
Alternatively, kmest can be used with the SSC packages parmest and
esetran to compute Greenwood confidence intervals, or delta-Greenwood
confidence intervals using a variety of Normalizing transforms.

Author: Roger Newson
Distribution-Date: 30 July 2026
Stata-Version: 16

INSTALLATION FILES (click here to install)
kmest.ado
kmest_p.ado
kmest.sthlp
-----------------------------------------------------------------------------------------------------------------------------------------------------------
(click here to return to the previous screen)