Using detring to remove special characters

[Xinruo Wang]

Hi all,

I'm trying to remove special characters from a string variable. I have two variables. One contains only "-", another contains "-" and "+".
I successfully removed "-" from the first variable. When I try to remove "-" and "+" together, The Stata shows 'option "+ not allowed'. Here is my code: destring var, gen(new) ignore("-", "+"). How can I fix this? Thank you!

[William Lisowski]

Replace

destring var, gen(new) ignore("-", "+")

with

destring var, gen(new) ignore("-" "+")

removing the comma between the two strings to ignore. See the ouput of help desetring for further details on the syntax of the ignore option on the destring command.

[Xinruo Wang]

Hi William,

I tried the code. Now the error is "var contains characters not specified in ignore(); no generate". I don't see any other special characters. How can I find out what the characters are? Thank you!

[Nick Cox]

tab whatever if missing(real(whatever))

will show you values that can't be converted to numeric. See also chartab (SSC).

[William Lisowski]

Building on Nick's advice, I'm concerned about ignoring "-". Consider the following example.

. destring x, ignore("-" "+") generate(y)
x: characters - + removed; y generated as byte
(2 missing values generated)

. list, noobs

  +--------+
  |  x   y |
  |--------|
  |  1   1 |
  | -2   2 |
  | +3   3 |
  |  -   . |
  |  +   . |
  +--------+

. generate xx = x

. replace xx = "" if trim(xx)=="-"
(1 real change made)

. destring xx, ignore("+") generate(z)
xx: character + removed; z generated as byte
(2 missing values generated)

. list, noobs

  +------------------+
  |  x   y   xx    z |
  |------------------|
  |  1   1    1    1 |
  | -2   2   -2   -2 |
  | +3   3   +3    3 |
  |  -   .         . |
  |  +   .    +    . |
  +------------------+

We see that by ignoring "-" the negative 2 was destringed as a positive 2.

Certainly Nick's advice is the place to start. Systematically review your data. Be careful about ignoring non-numeric characters that are in fact part of numbers.

[Zanda Bullock]

Good afternoon.

My data contain both special characters such as "%" "--" but also negative numbers.

I am trying to remove "--" but stata also removes minus signs "-".

destring var_old, gen(var_new) ignore("%" "--")

Q: How to remove "--" without removing regular minus signs "-"

Any help appreciated.

[Rich Goldstein]

you don't show any actual data (see the FAQ and "help dataex") so the following is just a guess; negative numbers may start with the "-" and, if so, you can use the "strpos()" function to help such that you are only getting rid of "-" that are not in position 1; however, this may not be true and you may want to download "moss" (use search to find, download and install) to help esp since you may be able to use the moss results to keep if the count is 1 and drop/delete/ignore if the count is >1

[Zanda Bullock]

-12.81%	-29.71%	--
-14.89%	-23.15%	--
1.13%	-3.05%	-12.34%
0.63%	-12.41%	-27.21%
1.26%	5.17%	-11.72%
0.31%	-60.48%	--
0.64%	--	--
0.80%	--	--
-0.93%	--	--
-0.05%	--	--
0.61%	--	--
0.46%	--	--
2.14%	--	--

This is an example of my data (changes to previous period).

[Andrew Musau]

You can use the -force- option of destring, but this may mask other problems in the data. Otherwise, regular expressions work well here.

* Example generated by -dataex-. For more info, type help dataex
clear
input str7(var1 var2 var3)
"-12.81%" "-29.71%" "--"     
"-14.89%" "-23.15%" "--"     
"1.13%"   "-3.05%"  "-12.34%"
"0.63%"   "-12.41%" "-27.21%"
"1.26%"   "5.17%"   "-11.72%"
"0.31%"   "-60.48%" "--"     
"0.64%"   "--"      "--"     
"0.80%"   "--"      "--"     
"-0.93%"  "--"      "--"     
"-0.05%"  "--"      "--"     
"0.61%"   "--"      "--"     
"0.46%"   "--"      "--"     
"2.14%"   "--"      "--"     
end

destring var1- var3, ignore(%) force replace

Res.:

. l, sep(0)

     +--------------------------+
     |   var1     var2     var3 |
     |--------------------------|
  1. | -12.81   -29.71        . |
  2. | -14.89   -23.15        . |
  3. |   1.13    -3.05   -12.34 |
  4. |    .63   -12.41   -27.21 |
  5. |   1.26     5.17   -11.72 |
  6. |    .31   -60.48        . |
  7. |    .64        .        . |
  8. |     .8        .        . |
  9. |   -.93        .        . |
 10. |   -.05        .        . |
 11. |    .61        .        . |
 12. |    .46        .        . |
 13. |   2.14        .        . |
     +--------------------------+

[Zanda Bullock]

This works perfectly. Many thanks, Andrew.

. destring change7d, ignore(%) force replace
change7d: contains nonnumeric characters not specified in ignore()
change7d: character % removed; replaced as double
(7214 missing values generated)