Iteration backed up problem

[Berend Nijhuis]

Dear Statalisters,

Unfortunately after running my regression in Stata this pops up:

mixed price in_out inhabitants density crimerate travel size employment income greenspace socialhousing || neighborhood:

Iteration 0:   log likelihood = -968156.44  
Iteration 1:   log likelihood = -968156.44  
Iteration 2:   log likelihood = -968156.44  (backed up)
Iteration 3:   log likelihood = -968156.44  (backed up)
Iteration 4:   log likelihood = -968156.44  (backed up)
Iteration 5:   log likelihood = -968156.44  (backed up)
Iteration 6:   log likelihood = -968156.44  (backed up)
Iteration 7:   log likelihood = -968156.44  (backed up)

When I try the same regression with a specification of -iterate(5)-, these are my results:

mixed price in_out inhabitants density crimerate travel size employment income greenspace socialhousing || neighborhood:, iterate(5)

-------------------------------------------------------------------------------
        price |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
--------------+----------------------------------------------------------------
       in_out |    6585.13   6838.532     0.96   0.336    -6818.147    19988.41
  inhabitants |  -.0231968   .0564559    -0.41   0.681    -.1338484    .0874548
      density |   785.4646   275.0119     2.86   0.004     246.4511    1324.478
    crimerate |  -29.97358   62.35665    -0.48   0.631    -152.1904    92.24321
       travel |  -6707.326   2810.795    -2.39   0.017    -12216.38   -1198.268
         size |   7876.235   23151.28     0.34   0.734    -37499.43     53251.9
   employment |  -1526.386   559.3436    -2.73   0.006     -2622.68   -430.0931
       income |   8831.107   1172.088     7.53   0.000     6533.856    11128.36
   greenspace |   838.0578   417.8564     2.01   0.045     19.07436    1657.041
socialhousing |  -421.7408   457.5617    -0.92   0.357    -1318.545    475.0637
        _cons |  -40347.05   112098.6    -0.36   0.719    -260056.2    179362.1
-------------------------------------------------------------------------------

From these numbers I can see that the variable inhabitants is extremely low, but excluding this variable does not solve the problem. Also another possible solution by just running the regression with one variable gives the iteration problem for every single variable. However, by chance I found that dropping 20.000 of the 77.000 observations solves the problem. I know that this is the way to perform a valid regression, but maybe it helps finding a solution for the problem.

Thanks in advance.

Berend

Edit: forgot something to mention.

[Maarten Buis]

With a random effects model you add a model for the constant; each neighborhood has its own constant, which a draw from a normal distribution with mean 0 and a to be estimated variance. The constant is the predicted price when all explanatory variables are equal to 0. If that far outside the range of the data, then Stata can run in the kind of trouble you describe. The solution is simple change the explanatory variables such that the value 0 is some meaningful value within the range of the data. This meaningful value could be the overal mean, but does not have to be (actually I prefer it not to be the mean, but tastes differ). So you create all new explanatory variables which are the old variables minus its meaningful value, and estimate your model with these new explanatory variables instead.

[Berend Nijhuis]

Dear Maarten,

Thanks for your advice! Since I'm not a Stata specialist I think this goes beyond my knowledge of Stata. In your response you say that it's just a simple change, but I don't understand it yet. Is there a simple command to make this 'simple' change?

With kind regards,

Berend.

[Maarten Buis]

generate

[Berend Nijhuis]

Ah sure. I've generated a new variable sizenew by

summarize size
generate sizenew= size-2.33

Where 2.33 is the mean of the variable size.

After this I still get the iteration problem

[Maarten Buis]

I am sure you are quite happy that we don't have direct access to your computer, but the downside is that you need to tell us exactly what you did if you want us to diagnose your problem. So show us exactly what you did and what Stata told you in return.

[Berend Nijhuis]

Dear Maarten,

That's indeed a downside. I will try to elaborate on what I've done so far.

First an example of my data with some of the independent variables.

* Example generated by -dataex-. To install: ssc install dataex
clear
input long price str22 local_authority double(density crimerate travel size) byte employment double income
 69950 "WANDSWORTH"                 76 192.92572394193104  9.73 2.21 85 36.385
178000 "EALING"                   54.2  77.48182410250276 10.86 2.53 73 29.729
185000 "SOUTHWARK"               84.86  45.11447077176905   8.1 2.26 73 29.165
124300 "HAMMERSMITH AND FULHAM" 100.73  170.9311192069813  8.58 2.16 74 33.635
535000 "CITY OF WESTMINSTER"     84.41  166.8689253444833  7.42 1.92 71 37.789
 54000 "CROYDON"                 38.21  84.81882227673805 11.02 2.36 77 30.487
 58500 "BARNET"                  36.27  81.76396536157985 10.88 2.44 74 32.129
 81950 "KINGSTON UPON THAMES"    39.54   241.205108879428 11.46 2.34 82  34.36
195000 "CITY OF WESTMINSTER"     84.41  166.8689253444833  7.42 1.92 71 37.789
 65000 "CAMDEN"                  90.85 152.44924755075246   8.1 2.06 73 32.921
131000 "EALING"                   54.2  77.48182410250276 10.86 2.53 73 29.729
109750 "GREENWICH"               45.28 136.02888019290774 11.35 2.28 72 28.661
106000 "SOUTHWARK"               84.86  45.11447077176905   8.1 2.26 73 29.165
 68500 "HARROW"                  40.98  84.57841345363467 11.82 2.59 78 30.827
685000 "CITY OF WESTMINSTER"     84.41  166.8689253444833  7.42 1.92 71 37.789
 94950 "HAMMERSMITH AND FULHAM" 100.73  170.9311192069813  8.58 2.16 74 33.635
137500 "KINGSTON UPON THAMES"    39.54   241.205108879428 11.46 2.34 82  34.36
132500 "TOWER HAMLETS"            99.2 109.56319541472469  8.17 2.47 64 29.383
185000 "HAMMERSMITH AND FULHAM" 100.73  170.9311192069813  8.58 2.16 74 33.635
 84450 "LEWISHAM"                70.81  53.18131784253702 10.19  2.3 77 29.156
end

The following actions are then performed:

drop if local_authority =="CITY OF LONDON"
drop if price < 52000
drop if price > 750000

drop id transaction_id data_processed quarter month year year_month post_code
drop address1 address2 address3 address4 town county record_status post_code_clean
drop borough_code borough_name ward_code ward_name msoa11 lsoa11 oa11

encode local_authority, gen(neighborhood)
encode inner_outer, gen(in_out)

destring greenspace socialhousing, replace

mixed price in_out inhabitants density crimerate travel size employment income greenspace socialhousing || neighborhood:

Which gives me the following result:

Iteration 0:   log likelihood = -968156.44  
Iteration 1:   log likelihood = -968156.44  
Iteration 2:   log likelihood = -968156.44  (backed up)
Iteration 3:   log likelihood = -968156.44  (backed up)
Iteration 4:   log likelihood = -968156.44  (backed up)
Iteration 5:   log likelihood = -968156.44  (backed up)
Iteration 6:   log likelihood = -968156.44  (backed up)
Iteration 7:   log likelihood = -968156.44  (backed up)
Iteration 8:   log likelihood = -968156.44  (backed up)
Iteration 9:   log likelihood = -968156.44  (backed up)
Iteration 10:  log likelihood = -968156.44  (backed up)

I hope this makes clear what I've done so far.

I could attach my dataset, but I don't know if that's what is needed.

Berend

[Maarten Buis]

So you did not do as I suggested in #2?

[Berend Nijhuis]

Maarten,

Your suggestion is included in the following code where the hard numbers in the generation lines are the mean values of the variables:

drop if local_authority =="CITY OF LONDON"
drop if price < 52000
drop if price > 750000

drop id transaction_id data_processed quarter month year year_month post_code
drop address1 address2 address3 address4 town county record_status post_code_clean
drop borough_code borough_name ward_code ward_name msoa11 lsoa11 oa11

encode local_authority, gen(neighborhood)
encode inner_outer, gen(in_out)

destring greenspace socialhousing, replace

generate in_outnew= in_out - 1.605745
generate inhabitantsnew= inhabitants - 231751.2
generate densitynew= density - 59.21304
generate crimeratenew= crimerate - 111.9356
generate travelnew= travel - 10.3065
generate sizenew= size - 2.339448
generate employmentnew= employment - 75.17517
generate incomenew= income - 31.50995
generate greenspacenew= greenspace - 33.27645
generate socialhousingnew= socialhousing - 23.61307

mixed price in_outnew inhabitantsnew densitynew crimeratenew travelnew sizenew employmentnew incomenew greenspacenew socialhousingnew || neighborhood:

This gives me the following result:

Iteration 0:   log likelihood = -968156.44  
Iteration 1:   log likelihood = -968156.44  
Iteration 2:   log likelihood = -968156.44  (backed up)
Iteration 3:   log likelihood = -968156.44  (backed up)
Iteration 4:   log likelihood = -968156.44  (backed up)
Iteration 5:   log likelihood = -968156.44  (backed up)
Iteration 6:   log likelihood = -968156.44  (backed up)
Iteration 7:   log likelihood = -968156.44  (backed up)
Iteration 8:   log likelihood = -968156.44  (backed up)
Iteration 9:   log likelihood = -968156.44  (backed up)
Iteration 10:  log likelihood = -968156.44  (backed up)
Iteration 11:  log likelihood = -968156.44  (backed up)