Need help performing Statistical tests comparing differences between 3 Sample Survey Data and between Sample and Population

[Roland Getor]

Dear all,
I am currently attending to comments of a reviewer for a paper I'm working on. I have three residents' survey results for 2016, 2017 and 2018. One of the comments from the reviewer states:

"Statistical tests should be done between the three samples and between these and the population, to test whether the 3 samples are not statistically significant and if each of them or all are not different from the population".

Will be grateful if someone can give me direction as to how to go about it.
Thanks

Kind regards,
Roland

[Roland Getor]

Dear all,
I am currently attending to comments of a reviewer for a paper I'm working on. I have three residents' survey results for 2016, 2017 and 2018. One of the comments from the reviewer states:

"Statistical tests should be done between the three samples and between these and the population, to test whether the 3 samples are not statistically significant and if each of them or all are not different from the population".

Will be grateful if someone can give me direction as to how to go about it.
Thanks

Kind regards,
Roland

Hi,
Will be grateful if anyone can be of assistance regarding the above.
Thanks

[Clyde Schechter]

There are plenty of people out here who would be happy to help, but you provide so little information about your problem that it just isn't possible. You and your reviewer are both familiar with your study; Forum readers have been told nothing about it other than it has, somehow, three groups. That's why you have gone three days without an answer.

Please read the Forum FAQ. It will give you excellent advice on how to pose your question in a way that is likely to attract a timely and helpful response.

[Roland Getor]

There are plenty of people out here who would be happy to help, but you provide so little information about your problem that it just isn't possible. You and your reviewer are both familiar with your study; Forum readers have been told nothing about it other than it has, somehow, three groups. That's why you have gone three days without an answer.

Please read the Forum FAQ. It will give you excellent advice on how to pose your question in a way that is likely to attract a timely and helpful response.

Thanks Clyde for the feedback. Will do.

[Roland Getor]

Dear all,
I am currently working on a paper on the "Social Acceptability of A Wind Turbine Blade Assembly Facility" located in a city in UK. We conducted three residents surveys over a three year period (1st Survey- during the Construction Phase, 2016 [82 responses]; 2nd Survey- during the Beginning of the Operational Phase, 2017 [63 responses] and the 3rd Survey-One year Post Operational Phase, 2018 [61 responses] of the facility).

The surveys sought the views of residents residing in the catchment area of the blade facility of the on issues like Noise Pollution, Air Quality, effect of erection of noise barrier on the local scenery, loss of local amenities like footpath/bicycle lanes, views on impact of offshore wind on the environment. We also collected background information on how long residents have lived in the area, employment status, age, education. The catchment area has about 160 households with at least 1 household member.

Now, one of the questions from a reviewer states:
"Statistical tests should be done between the three samples and between these and the population, to test whether the 3 samples are not statistically significant and if each of them or all are not different from the population".

Will be grateful for any assistance as to how to carry out the tests.
Thanks
Roland

[Clyde Schechter]

The reviewer's language is a bit vague, but here's my take on it.

The purpose of opinion surveys is to estimate the opinion of a target population. The target population is, in your case, all residents of the catchment area. That target population is typically too large or too inaccessible to survey everybody, so we get samples. Ideally, random samples, though, in practice, most often they are at least partially samples of convenience. So there is always the concern whether the samples are adequately representative of the population And when you use three different samples over time, there is the concern whether they are similar in composition to each other. So I think what the reviewer wants you to do is identify a reference source of information about the target population's distribution on the same demographic variables you measured in your study: duration of residency in the area, employment status, age, and education. (I would throw in sex as well as this is often very salient in opinion surveys.) Here in the US we would usually use recent Census data or similar government statistics--I don't know what the equivalent in the UK would be. Then create a table with separate columns for each of the three surveys and a fourth column for the target population. Each row of the table would represent one of the aforementioned variables.

Then carry out appropriate statistical tests for equality of distributions of each variable across the three surveys as a whole, and for each survey with respect to the target population. What are appropriate tests? That depends on the way the variables are represented in your study. For example, if education was measured as years of formal schooling completed, then it could be treated as a continuous variable or a count variable. If you categorized it into groups such as primary education only, secondary, higher education, post-graduate (these are categories we might use in the US, I'm not familiar with how it's done in the UK, sorry.) then you could treat it as ordinal or categorical.

So, for sake of disscussion, let's suppose that age is a continuous variable. You could do a regression of age on indicator ("dummy") variables for the sample groups, and then, for each sample separately, do a one-sample t-test comparing the age in that sample to the reference source's mean age in the target population. For categorical variables, chi-square tests would probably be the most appropriate. And so on. For ordinal variables, Kruskal-Wallis ANOVA can be done to compare the three samples to each other (-help kwallis-), and each sample's comparison to the target population might be done with a median regression (-help qreg-) of the variable as dependent variable (with no explanatory variables) followed by a test of the equality of the constant term to the reference-source's median value (if that's the statistic the reference source gives.)

Evidently this is going to be a fair amount of work. And the details will depend on exactly how your own data is structured and what kind of summary statistics are available from reference sources about that catchment area. There may be additional difficulties if the form in which reference statistics are provided is not equivalent to the form in which you have your data.

I think that's about as much as can be said at this point. Happy to advise later with more details as you progress and have a clearer picture of the detailed situation.

[Roland Getor]

Hi Clyde,
Thanks so much for the very detailed response. I agree the reviewers language is a bit vague. I will proceed as suggested and carry out the exercises and the tests outlined. These should help address the issues raised by the reviewer. Very grateful for your assistance.
Thanks
Roland

[Roland Getor]

Hi Clyde,
Wanted to say a big thank you for your assistance which helped immensely in revising the paper and finally got it published, "Social Acceptability of a wind turbine facility in Kingston upon Hull" https://doi.org/10.1016/j.jclepro.2022.134859.
Much appreciated.