svy: proportion not displaying Design-based F test

[Inaamul Haq]

I am running

svy: prop pvp01, percent over(partworked01)

I expect the result to display a Design-based F test, but it does not. What can the problem be.
Here is my svyset command

svyset clusterid [pweight = weightsvy], strata(strata) singleunit(scaled)

Kindly help.

[Inaamul Haq]

Even though I tried a workaround with

svy: prop pvp01, percent over(partworked01) citype(jeff)
quietly svy: tab pvp01 worked01
display e(p_Pear)

I still want to know why svy: prop did not display design-based p by default.

[Richard Williams]

I googled this and AI responded as follows. For anything important I never trust AI answers unless I can verify with the original sources — all too often answers sound plausible but have serious errors — so determine for yourself if this answer is correct or at least partially correct.

AI Overview
When the design-based F-test is not displayed in Stata after a svy command, it typically indicates a problem with the variance-covariance matrix being less than full rank. This can occur for several reasons in the context of complex survey data:

• Insufficient Design Degrees of Freedom:
  The number of coefficients being estimated in your model may exceed the design degrees of freedom, which are determined by the survey design (e.g., number of PSUs minus number of strata). If this is the case, the F-statistic cannot be computed.
• Near-collinearity of Predictors:
  While perfect collinearity would lead Stata to drop a variable, near-collinearity can result in a singular or near-singular variance-covariance matrix, preventing the calculation of the F-statistic. This can be investigated using commands like estat vif after your regression.
• Singleton Strata:
  If any stratum in your survey design contains only one primary sampling unit (PSU), the variance estimation formulas will fail, leading to a missing F-statistic. You can check for singleton strata using svydes and, if found, address this by reassigning the singleton stratum to another appropriate stratum.
• Incorrect Use of if Condition:
  When analyzing subsets of survey data, using the if condition instead of the subpop() option within the svy prefix can lead to incorrect variance estimation and potentially a missing F-statistic, as it does not properly account for the survey design on the subpopulation.
• Specific Variance Estimator Limitations:
  In some cases, certain variance estimators, like the bootstrap variance estimator with svy: tabulate, might not display design-based tests by default, as the uncorrected testing results may be displayed instead.

Troubleshooting Steps:

• Check svydes output:
  Identify any singleton strata that might be causing issues.
• Investigate collinearity:
  Use estat vif after your svy estimation to check for high variance inflation factors, indicating potential collinearity.
• Review svyset:
  Ensure your svyset command correctly specifies all design characteristics, including weights, PSUs, and strata.
• Use subpop() for subsets:
  If analyzing a subpopulation, always use the subpop() option within the svy command instead of if.
• Consider alternative tests:
  If the F-statistic remains unavailable, explore using test after your svy estimation to perform Wald tests on specific coefficients or groups of coefficients, which will produce an F-test appropriate for survey data.

[Joseph Coveney]

I googled this and AI responded as follows.

Richard, from the first line of the response, I'm guessing that your Google prompt was something like, "Why would the design-based F-test not be displayed in Stata after a svy command?"

Maybe for FAQ consideration: would it in general be helpful to readers to know what the prompts were for AI responses posted to the list?

[Richard Williams]

Joseph, good idea. I'll do that in the future. I don't remember my exact wording, but the prompt "stata design based f test not displayed site:www.statalist.org" gives similar results.

With AI, I don't think you get identical results every time.

I'll stress again that I never trust AI unless I can verify from other sources. I've seen it give plausible sounding but wildly incorrect answers too many times. But in this case, I wasn't having the problem myself and I didn't have a reproducible example, so I thought I'd toss out what AI said and let the OP decide if it was helpful. In this case, I suspect the AI answer addresses various problems you can have with svy but not necessarily the specific problem the OP was having.

[Joseph Coveney]

With AI, I don't think you get identical results every time.

Yeah, I wasn't thinking in terms of reproducibility, but rather that there's an art to composing prompts to get the best results from AI. The benefit to users on the list would be to get an idea of what kind of wording works best for getting results from AI that most closely meets their needs.

I completely agree on the need to vet what one gets from AI.