I have a dataset of over 60 million observations. The dataset is of individual people with 300 or so need variables, some supply variables and a dependent variable of their healthcare resource use in £.
The dataset is for a single year of resource use, it is not panel data, an individual appears only once.
A healthcare organisation that a person is registered with is considered a supply variable. These are usually entered into the model as dummy variables. A person can only be registered with one healthcare organisation, so that where the dummy variable is 1 for that person, all other healthcare organisation dummies are 0: People are nested within healthcare organisations.
I usually create a linear cost model. The effect of supply is later “sterilised”: that is, after the model has been ran and coefficients have been estimated, the underlying data for the supply variables are set to their mean, before creating the model predictions. The idea is that the effect of supply is averaged out across observations. the coefficients for all other variables (representing need) are not sterilised.
In the latest model, the number of health care organisations has increased from 199, to around 6000. It doesn’t seem right to estimate a model for 5999 dummy variables (plus the 300 or so need variables), and I am wondering if there is an alternative method to follow, that would allow to measure the effect of the 6000 healthcare organisations, but then allow me to sterilise this effect, in the same way I sterilise the effect of supply in the linear regression model.
Any advice would be much appreciated
The dataset is for a single year of resource use, it is not panel data, an individual appears only once.
A healthcare organisation that a person is registered with is considered a supply variable. These are usually entered into the model as dummy variables. A person can only be registered with one healthcare organisation, so that where the dummy variable is 1 for that person, all other healthcare organisation dummies are 0: People are nested within healthcare organisations.
I usually create a linear cost model. The effect of supply is later “sterilised”: that is, after the model has been ran and coefficients have been estimated, the underlying data for the supply variables are set to their mean, before creating the model predictions. The idea is that the effect of supply is averaged out across observations. the coefficients for all other variables (representing need) are not sterilised.
In the latest model, the number of health care organisations has increased from 199, to around 6000. It doesn’t seem right to estimate a model for 5999 dummy variables (plus the 300 or so need variables), and I am wondering if there is an alternative method to follow, that would allow to measure the effect of the 6000 healthcare organisations, but then allow me to sterilise this effect, in the same way I sterilise the effect of supply in the linear regression model.
Any advice would be much appreciated
Comment