Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    Thank you. I think longitudinal surveys are not controlled experiments, therefore conclusions draw from them are not causal.
    I have another question about fixed effects models. I was once told by an economist that to use fixed effects models, you need to have at least four waves of data. I haven't been able to find any literature supporting that, but wonder if it is true. Thank you.

    Comment


    • #17
      No, it is not true. I'm not sure what that person was thinking of.

      Comment


      • #18
        Thank you. I have another question: when odds ratio is bigger than 2, how do we interpret it? I use estout to transform coefficients from my logistic regressions to odds ratios, but I got a result of 2.46 and sometime even bigger. I don't think I can use percentage now. And my understanding is odds ratio should be betwen -2 and 2. So I don't know where the problem is. Is esttab or estout transforming my coefficients to odds ratio or something else?
        Last edited by Meng Yu; 11 Jun 2021, 22:19.

        Comment


        • #19
          There is no reason an odds ratio can't be greater than 2. On the other hand, an odds ratio can never be less than 0. The possible values for an odds ratio range from 0 to + infinity. And the interpretation does not depend on the size of the odds ratio. In your case, assuming the predictor is dichotomous, it means that the odds of a positive outcome when the predictor is 1 is 2.46 times the odds of a positive outcome when the predictor is 0. If the predictor is continuous, it means that a unit difference in the value of the predictor is associated with the odds of a positive outcome increasing by a factor of 2.46.

          It's important to remember that odds ratios are not probability ratios. The relationship between odds and probability is odds = probability/(1-probablity). So as the probability approaches 1, the odds goes to infinity. That's why you can have infinite odds ratios--there is infinite room in the odds metric; odds are not bounded by 1 the way probabilities are.

          It is true that if the predictor variable that odds ratio represents is a dichotomous variable, it is, in the real world, rather uncommon to find odds ratios greater than 2 (or less than 1/2): there just aren't that many things that have effects that large. But in principle it's possible, and it does happen from time to time. It is much less common, in the real world, to see an odds ratio greater than 4 for a dichotomous predictor (or, less than 1/4) and when you see a result like that you should suspect that there has been a mistake in the data or in the analysis. But even this does occur: the odds ratio associating cigarette smoking with lung cancer is about 10! But few real world effects are that powerful. And, of course, if you are dealing with a continuous predictor variable, then the magnitude of the odds ratio is going to depend on the units of measurement of the predictor, and so anything is possible.

          Comment


          • #20
            Thank you very much. I understand how to interpret the effect of a dichotomous predictor now. Can I assume, if the predictor is a categorical variable, the interpretation would be: the odds of a positive outcome when the predictor is (one of the categories) is 2.46 times the odds of a positive outcome when the predictor is the reference group? And if the odds ratio is smaller than 2, I simply state it is a certain percentage either lower or higher odds than the reference group. I don't need to use the term "a positive/negative outcome" in my interpretation, right?

            In my analysis, the predictor is a continuous variable, a score ranging from 0 to 1. I don't understand what you meant "by a factor of 2.46". 2.46 has many factors.

            Thank you.

            Comment


            • #21
              Can I assume, if the predictor is a categorical variable, the interpretation would be: the odds of a positive outcome when the predictor is (one of the categories) is 2.46 times the odds of a positive outcome when the predictor is the reference group?
              Yes.

              And if the odds ratio is smaller than 2, I simply state it is a certain percentage either lower or higher odds than the reference group.
              Yes, but you can also do it with an odds ratio bigger than 2. With OR = 2.46 you can say that the odds are 146% higher than the odds for the reference group.

              I don't understand what you meant "by a factor of 2.46". 2.46 has many factors.
              The term factor is overloaded in English. I did not mean it in the sense of 3 and 5 are the factors of 15. When y = a * x we can say that y differs from x by a factor of a. It's a different use of the term. Admittedly it's confusing.

              Comment


              • #22
                Thank you for your explanation. If the outcome variable is dichotomous (mental health problem with 0 meaning healthy) and the predictor variable is a continuous score, can I say "a one unit increase in the score is associated with an increase of 2.46 times the odds of having of mental health problems?"

                I use three waves of survey data in this analysis, and I examine the contemporaneous relationship between the predictor and the outcome variable. I included wave as a covariate in the regression. I wonder if it is necessary to do so.

                Comment


                • #23
                  Thank you for your explanation. If the outcome variable is dichotomous (mental health problem with 0 meaning healthy) and the predictor variable is a continuous score, can I say "a one unit increase in the score is associated with an increase of 2.46 times the odds of having of mental health problems?"
                  Not exactly. You can say that a unit increase in the score is associated with an increase to 2.46 times the odds of having mental health problems. Or you could say it is associated with an increase of 1.46 times the odds of having mental health problems. Of, perhaps clearer than either of those, the odds of having mental health problems is 2.46 times as high with each unit increase in the predictor.

                  I use three waves of survey data in this analysis, and I examine the contemporaneous relationship between the predictor and the outcome variable. I included wave as a covariate in the regression. I wonder if it is necessary to do so.
                  That depends. First, there is the question whether theory or prior research suggests that this variable either has trends over time ("secular trend") or is subject to random shocks in time. If so, it is important to include it. If not, it might be omitted. Clearly, that is not a statistical question, but one that requires knowledge in your domain. If unsure about this, consult a colleague with expertise in mental health. Next, since you already included it, you should look at the results you got for that variable. If the coefficients of the wave indicators (or coefficient of the single wave variable if you treated it as continuous) are large enough that wave actually makes a noticeable impact on the predicted outcome, then, even if there were no a priori reason to include it, the findings would suggest that it should be kept in the model.

                  If you have neither an a priori reason, nor a suggestion in your findings that wave makes a difference in the outcome, then it can be omitted, and doing so would give you a simpler, cleaner model.

                  Comment


                  • #24
                    Thank you for answering both my questions. I appreciate it.

                    The coefficients of the wave variable are big enough for me to keep it and time point does make a difference to the relationship under examination. I have been thinking what a covariate does to a regression model. I don't understand how variables "constrain" each other in a model.

                    Comment


                    • #25
                      I don't understand how variables "constrain" each other in a model.
                      That's very complicated, and really has to be understood as a property of the model in its entirety. You can't even correctly discuss things in terms of pairs or triads of variables. I don't know anybody who has an intuitive understanding of this in the sense of being able to look at, say, a correlation matrix, and being able to perceive the impact of including or excluding some subset of the variables. Ultimately it boils down to the matrix algebra, including matrix inversion, that underlies regression, and that's not simple. In the end, I think the only effective way to know what a covariate does is to run the model with and without it and then compare the results.

                      Comment


                      • #26
                        Thank you. I thought it would boil down to matrix algebra, but thought there was a way to visually depict the change in X1, X2, X3... and Y.

                        What I am really interested in is the differences between having wave as a covariate, interacting wave with the predictor variable, and interacting wave with all of the independent variables. When the survey data was collected at waves 2 and 3, the interview question for my outcome variable changed somewhat from wave 1 and I am concerned it may have contributed to the change in the distribution of the variable. So when I run a xtlogit model across the three waves, I don't know how different the predictor coefficient will be from if I run the model separately for the three waves.

                        I believe if I interact all independent variables with the wave variable, the results will equal to if I run the model separately for the three waves. Correct me if I am wrong.

                        But it is not always feasible to interact all independent variables with the wave variable as I also need to interact the predictor with another variable. So I am thinking if it is good enough to just add wave as a covariate to deal with data limitation.
                        Last edited by Meng Yu; 18 Jun 2021, 16:26.

                        Comment


                        • #27
                          When the survey data was collected at waves 2 and 3, the interview question for my outcome variable changed somewhat from wave 1 and I am concerned it may have contributed to the change in the distribution of the variable.
                          Yes, that can be a serious problem, and it is definitely appropriate to begin on the assumption that wave needs to be represented in the model.

                          I believe if I interact all independent variables with the wave variable, the results will equal to if I run the model separately for the three waves. Correct me if I am wrong.
                          That is almost completely correct. The coefficients will be the same. But the standard errors, and the confidence intervals and test statistics that derive from it, will usually be different. This is largely due to sample size. When you run separate models, each model's sample size is just the observations for that wave. When you run a single model where wave interacts with all of the model variables, then the sample size is the complete sample for all three waves combined. It is, in part, for this reason that such models allow the individual waves to "borrow strength" from each other.

                          But it is not always feasible to interact all independent variables with the wave variable as I also need to interact the predictor with another variable.
                          That shouldn't, in general, be a problem. It just means you have a three way interaction. That's not a big deal--just do it. The only issue it might raise is if your sample size is marginal for the number of variables to start with, in which case adding in a bunch of interaction terms can cut the data too finely and you don't have enough observations to support the analysis properly.

                          So I am thinking if it is good enough to just add wave as a covariate to deal with data limitation.
                          You need to consider from a real-world, not a statistical, perspective what the effect of the change in wording of the question is likely to be. If it is to simply shift the proportions of positive and negative responses to the question, including i.wave as a model covariate will adequately model that. However, it you think that the nature of the change in question wording is such that it may have changed the relationship between your predictor variable(s) and the response, that is an entirely different matter and requires an interaction to properly model. It is necessary, and sufficient, to interact wave with those predictors and covariates whose relationships to the response variable are likely to have changed as a result of the wording change. You do not have to interact with every model variable unless you think that all of those relationships will have been changed as a result of the wording change.

                          Let me give an example. I will base it on knowledge testing rather than opinion survey, but the principles are the same. Suppose I am studying performance of students in an American primary school on a standardized test over time. Suppose that in a certain year, a question is changed to make it more difficult, e.g. a problem involving multiplying 3 digit numbers is changed to multiplying 5 digit numbers. That is likely to shift the scores downward, but whatever attributes of the student I am looking at in relationship to performance, say sex and whether they are from an English speaking household, their predictiveness of test score is likely to remain as it was. The change in the item should affect all the students equally; it's just harder to answer for everyone. So in my model, I would include "wave" as a covariate in the model and leave it at that. On the other hand, suppose a word problem with a short, simple stem is replaced by a word problem whose mathematical solution is essentially the same but the context and stem are lengthened considerably and more difficult vocabulary words are included in the problem. Now, this is a bad item because it confounds mathematical ability with reading comprehension and differentially makes the problem more difficult for students from non-English speaking households than for students from English speaking households. In terms of modeling the scores in terms of those predictors, I would expect this change to increase the association of being from an English speaking household with score. On the other hand, there is no reason to think that the increased difficulty of this bad item would differ between boys and girls. So I would model this by including "wave" and its interaction with English speaking household, but not its interaction with sex.

                          If you do not have a good sense of how the change in wording is likely to have affected things, I suggest consulting a colleague with expertise in survey question design or a psychometrician.
                          Last edited by Clyde Schechter; 18 Jun 2021, 19:09.

                          Comment


                          • #28
                            Thank you for your detailed answering. When I interact every independent variable with wave variable, the sample size becomes three times as large. Do you mean that will increase the statistical power and therefore be more likely to produce statistically significant results?

                            The change in the wording of the outcome variable is unlikely to change the relationship between the predictor variable and the outcome variable in my study. I think it is more likely to affect every respondent similarly. So I will just include wave as a model covariate.

                            In another study of mine using the same data and outcome variable, I examine the relationship between the independent variable and the outcome variable in the subsequent wave. In this situation, is adding wave variable as a covariate sufficient to deal with the data limitation? As the change in the outcome variable is more dramatic from waves 1 to 2 than from waves 2 to 3, it may be more likely to achieve statistically significant result from waves 1 to 2.




                            Comment


                            • #29
                              When I interact every independent variable with wave variable, the sample size becomes three times as large.
                              It isn't clear to me what you are saying here. If you go from 3 separate models, one for each wave, to a single model of all three waves (with or without interactions of any kind), yes the sample size will increase, by a factor of 3 if all three waves have the same number of observations. If you mean that you started with a model of all three waves having no interactions and then you added interactions and that changed the sample size, then something is radically wrong. Adding an interaction (or any other kind of variable) should not change the sample size--if it does, it will only reduce the sample size due to missing values in the newly added variable(s).

                              I am one of those people who endorses the American Statistical Association's recommendation to abandon the concept of statistical significance. See https://www.tandfonline.com/doi/full...5.2019.1583913 for the "executive summary" and https://www.tandfonline.com/toc/utas20/73/sup1 for all 43 supporting articles. Or https://www.nature.com/articles/d41586-019-00857-9 for the tl;dr.So I don't think about these things in terms of their effect on significance. The sample size becomes larger, which, all else equal, would increase the precision of your estimates. Of course, sometimes all else is not equal and the pooling of the data can result in adding noise, which would have the opposite effect. How those balance out is hard to predict.

                              The change in the wording of the outcome variable is unlikely to change the relationship between the predictor variable and the outcome variable in my study. I think it is more likely to affect every respondent similarly. So I will just include wave as a model covariate.
                              That sounds right.

                              In another study of mine using the same data and outcome variable, I examine the relationship between the independent variable and the outcome variable in the subsequent wave. In this situation, is adding wave variable as a covariate sufficient to deal with the data limitation? As the change in the outcome variable is more dramatic from waves 1 to 2 than from waves 2 to 3, it may be more likely to achieve statistically significant result from waves 1 to 2.
                              I'm not sure I understand the situation. If you are analyzing the data with the next wave's outcome, there will be no observations from wave 3 in the estimation sample, although the outcome variable from wave 3 will appear in the estimation (and the outcome variable from wave 1 will not). So your sample size will be two wave's worth of data, not three. Is that the "data limitation" you are referring to? Including wave as a covariate does not compensate for that. What it does do is absorb the effect of the changed wording on the outcome so that that effect does not get misattributed to something else. It avoids a bias, it doesn't affect precision (or, in your terms, power).

                              Comment


                              • #30
                                What it does do is absorb the effect of the changed wording on the outcome so that that effect does not get misattributed to something else.
                                That is what I want to achieve. Sorry I should have made it more clear.

                                I have a question on how to interpret marginal effect percentages (if they are percentages) when I interact a continuous variable with a categorical variable in a xtlogit regression.

                                Code:
                                margins, dydx (score) at (province=(0 1 2 3 4))
                                And below are the results: A to E stand for the five provinces
                                A B C D E
                                0.28 0.07 0.34* 0.91*** -0.27
                                (0.21) (1.28) (0.15) (0.26) (0.22)
                                Delta-method standard errors in parentheses.
                                *p<0.05, **p<0.01, ***p<0.001

                                I wonder if I can say something like "In province D, a one unit increase in the score is associated with 91 percentage points higher chance of having mental health problems, whereas in province C, it is associated with only 34 percentage points higher chance."

                                Thank you.

                                Comment

                                Working...
                                X