[Jem Lane]

Excellent - thanks a lot Carlo. Really appreciate all your help.

Jem

[Clyde Schechter]

3. You should only move from one model to a more complex one (e.g. involving an extra explanatory variable) if the -lrtest- for the two produces p<0.05.

P-values were developed for a specific purpose: to test hypotheses. If the experiment in question is, for example, designed to confront two different theories about the (probably not directly observable) disposition of some substance, and one theory predicts that the relationship between concentration and cv (whatever that is) is linear, and the other theory predicts a quadratic relationship (a parabola with a peak or nadir within the range of concentrations to be studied), then relying on the p-value to endorse one of the models over the other is an appropriate way to do it, presuming that the experiment has been properly designed and executed, and that the design provided adequate a priori power to really distinguish the hypotheses.

But not all data analysis is carried out in a hypothesis testing context. And when we are not in that modus operandi, we should generally ignore p-values. Sometimes we are working with things that have seldom or never been really studied before and there is little or no theory to guide our expectations. We may be in the position of, in effect, asking "I wonder what happens if I..." In those situations we generally are in no position to frame sharp hypotheses, and lack the preliminary data needed to properly design any good experiments. We're just groping and trying to get a sense of what relationships may exist among certain variables we are interested in. This is exploratory work. Graphs, descriptive summaries and tabulations are the principal tools to use here. These may in turn suggest certain types of models might fit the data well, and we might give them a try. But in this circumstance, the p-values that those models provide us do not really deserve serious consideration. (If nothing else, in this context our choice of models has been driven by the data itself, so that the p-values do not have their nominal meaning.) We should focus, at this stage of investigation, more on the non-inferential statistics, such as parameter estimates, and explore graphs of observed vs predicted values to get a sense of things. But we are not yet in a position to draw sharper conclusions. That requires additional data and additional research.

So to come down to Jem's specific situation, I would disagree with the quotation above. If this was a designed experiment, then either it was designed without consideration of statistical power (not a good idea), or the measurements obtained are much more variable than was anticipated when the study was designed. My impression, though, is that this was not really a designed experiment with sharp prior hypotheses--more of an exploration. In any case, I would attach no importance to statistical significance here. The only clear conclusion from this data is that they are too scanty and imprecise to distinguish between a linear and quadratic model.

If the underlying research questions warrant the expense and effort, the data in hand can be provide a basis for further research and development. Perhaps the cv measure has too much variation and the measurement process needs to be somehow refined--a different technology, or using the average of several replicate measures, or.restricting sampling to a subpopulation in which it varies much less, or other alternatives. Or perhaps there are auxiliary variables that should be measured to allow for variance reduction by statistical adjustment. Or perhaps we need to plan a study with more samples, enough samples that we will have power to distinguish the linear and quadratic models.

But at this point, I would draw no conclusions based on p-values.

[Carlo Lazzaro]

I hope someday I'll become as clear as Clyde in explaining such theoretical stuff (I mean in Italian, too).
I find also interesting Clyde's bridging the gap between frequentist and Bayesian approach (

...sharp prior hypotheses...

).

[Jem Lane]

Dear Clyde,

Thank you for this really helpful explanation. I echo Carlo's sentiments - a very clear overview of different experimental approaches and how best to approach them statistically.
I have once again learnt a lot from this post and all the responses; my future work will be much better informed. Grateful as always to all contributors.

Jem