Why doesn’t the logical operator != (not) work in if-expression?

[Minna Tuominen]

I am using European Social Survey data (wave 9) and working on the variable ‘wkhtot’ (with 41,630 valid observations) measuring average weekly working hours. The variable includes 419 observations with weekly working hours greater than 80.

I want to create a copy of the original variable replacing all extreme values with 80, excluding those with missing values. To build a new variable, I ran the following commands:


gen wkhtot_mod = wkhtot
replace wkhtot_mod = 80 if (wkhtot > 80 & wkhtot != . )


As a result, Stata replaced all extreme values AND the missing values with 80. Why is that?


I did realise that I can reach the intended result with the following:


replace wkhtot_mod = 80 if (wkhtot > 80 & wkhtot < . )


I would still like to understand, why didn't the != (not) operator work in my first attempt.

I would appreciate any insight on this.

[Andrew Musau]

gen wkhtot_mod = wkhtot
replace wkhtot_mod = 80 if (wkhtot > 80 & wkhtot != . )


As a result, Stata replaces all extreme values AND the missing values with 80. Why is that?

Extreme values if meaning values greater than 80 are replaced because you asked for that:

... if (wkhtot > 80 ...

But you have to present a data example that shows that missing values are replaced. Unless you are referring to extended missing values (.a, .b, ..., .z) which are different from system missing values. See

help missing

[Minna Tuominen]

Dear Andrew,

Many thanks for the quick response. Indeed, I have both system missing (.) and extended missing values (.a, .b, .c etc.) in the data. I have not realized before that the extended missing values (.a, .b, .c etc.) would behave differently from the system missings (.). I read the help-file, but still I cannot see why they all do not obey the same commands.

I wonder, if I wanted to apply the operator != on the extended missing case, how should I then write my command?

[Andrew Musau]

The help file is clear as far as I am concerned. Two points:

3. The ordering of missing values is

all nonmissing numbers < . < .a < .b < ... < .z

and

The relational operators (see operators) interpret missing values as large positive numbers (see above). All the following thus evaluate to true

73 < .
. == .
.a == .a
.a != .
.a < .b
.a <= .b

Here, it is explicit that extended missing values are not equal to system missing. These are distinct.

I wonder, if I wanted to apply the operator != on the extended missing case, how should I then write my command?

You will have to enumerate them.

local list
foreach l in `c(alpha)'{
    local list `list' & wkhtot !=.`l'  
}
display "`list'"

Res.:

. display "`list'"
& wkhtot !=.a & wkhtot !=.b & wkhtot !=.c & wkhtot !=.d & wkhtot !=.e & wkhtot !=.f & wkhtot !=.g & wkhtot !=.h & wkhtot !=.i & wkht
> ot !=.j & wkhtot !=.k & wkhtot !=.l & wkhtot !=.m & wkhtot !=.n & wkhtot !=.o & wkhtot !=.p & wkhtot !=.q & wkhtot !=.r & wkhtot !
> =.s & wkhtot !=.t & wkhtot !=.u & wkhtot !=.v & wkhtot !=.w & wkhtot !=.x & wkhtot !=.y & wkhtot !=.z

So it is apparent that it is just easier to specify

& wkhtot < .

[Minna Tuominen]

Thank you for the clarification. It's really the first time I run into this kind of a problem. I appreciate your patience!

[Andrew Musau]

Another way to capture all missing values (system missing and extended missing) is to use the -missing()- function.

help missing()

So I often write

if !missing(var1)

instead of

if var1!=.

In your case:

gen wkhtot_mod = wkhtot
replace wkhtot_mod = 80 if (wkhtot > 80 & !missing(wkhtot))