A quick question about weird question marks in a string variable

[Yapeng Wang]

Hello, I am using Stata 15.1. I have a string variable - student-teacher ratio (see data below). for some responses, they have the sign �� rather than a regualr expression. I think it should be a colon. Does anyone know how I can change �� into a colon or make it show as it should be rather than ��? Thanks!
----------------------- copy starting from the next line -----------------------

* Example generated by -dataex-. To install: ssc install dataex
clear
input str9 plc0107
"12:1"     
"16��1"    
"6.6:1"    
"3:1"      
"16.3:1"   
"10.7��1"  
"1173��93" 
"11��1"    
"23��1"    
"16.15��1" 
"1��14"    
"16.15��1" 
"15��1"    
"12��1"    
"1��13"    
"17.7:1"   
"16.3:1"   
"14.5��1"  
"1��19.6"  
"100��9"   
"9��1"     
"09��1"    
"12.5:1"   
"12��1"    
"1��10.4"  
"17.7:1"   
"15��1"    
"11:1"     
"11.4:1"   
"16.3:1"   
"9��1"     
"1��11"    
"18:1"     
"1��15"    
""         
""         
"6:5"      
"12:1"     
"11:1"     
"10.7��1"  
"14��1"    
"16.3:1"   
"12:1"     
"16.3:1"   
"1��14"    
"12��1"    
"10��1"    
"10��1"    
"16��1"    
"17:1"     
"6.6:1"    
"100��5.38"
"6.6:1"    
"12.8��1"  
"16.3:1"   
"7:1"      
"15.6��1"  
"11:1"     
"20��1"    
""         
"12.5:1"   
"5.5��1"   
"1��19.6"  
"18:1"     
"7:1"      
"1��15"    
"1��10.4"  
"11:1"     
"11��1"    
"15:1"     
""         
"12.5:1"   
"11��1"    
"16.8:1"   
"14.5��1"  
"1��11"    
"22��1"    
"10.7��1"  
"17:1"     
"21��1"    
"6.6:1"    
"7:1"      
"17:1"     
"7��1"     
"14��1"    
"1007��60" 
"12.5:1"   
"6:5"      
"18��1"    
"16.15��1" 
"11.6��1"  
"17:1"     
"3:1"      
"1��10.4"  
"13��1"    
"16��1"    
"7��1"     
"11:1"     
"14.5��1"  
"1��11.3"  
end

------------------ copy up to and including the previous line ------------------

[Clyde Schechter]

So the first thing you need to do is identify what underlies the "weird question marks." If you install Robert Picard's -chartab- from SSC and run -chartab plc0107- you will get a complete listing of all the characters that occur in the variable, along with their Unicode numeric values in both decimal and hexadecimal, as well as the characters themselves. When I do this using your example here, I find that the underlying Unicode character for the weird question marks is Unicode 65,533. But you have to do this yourself in your real data because the process of transfering the data from your data set to your clipboard to the Forum editor to my clipboard to my Stata may have changed it. Anyway, find out what the numeric representation of that character is, then run

[code]
gen cleaned_plc0107 = usubinstr(plc0107, "`=char(whatever)'", ":")
[code]
replacing whatever by the decimal numeric value that -chartab- showed you, and you will get a new variable with colons wherever the weird question marks were.

[Yapeng Wang]

So the first thing you need to do is identify what underlies the "weird question marks." If you install Robert Picard's -chartab- from SSC and run -chartab plc0107- you will get a complete listing of all the characters that occur in the variable, along with their Unicode numeric values in both decimal and hexadecimal, as well as the characters themselves. When I do this using your example here, I find that the underlying Unicode character for the weird question marks is Unicode 65,533. But you have to do this yourself in your real data because the process of transfering the data from your data set to your clipboard to the Forum editor to my clipboard to my Stata may have changed it. Anyway, find out what the numeric representation of that character is, then run

[code]
gen cleaned_plc0107 = usubinstr(plc0107, "`=char(whatever)'", ":")
[code]
replacing whatever by the decimal numeric value that -chartab- showed you, and you will get a new variable with colons wherever the weird question marks were.

Hi Clyde, Thank you very much for the prompt response! I tried the method. The results showed that the question marks are 65,533. I replaced whatever by the value 65,533, but Stata told me 65,533 was invalid name. I used 65533, and Stata told me "invalid syntax". I do not know why it did not work. Any more advice? Thanks!

[William Lisowski]

Since your results from chartab suggestged that the representation in post #1 was probably correct, I tested the code from post #2 on your example data. That led to two changes to the code, and it now works as intended.

. gen cleaned_plc0107 = usubinstr(plc0107, "`=uchar(65533)'", ":", .)
(4 missing values generated)

. list cleaned_plc0107 plc0107 in 1/10, clean abbreviate(16)

       cleaned_plc0107   plc0107  
  1.              12:1      12:1  
  2.             16::1     16��1  
  3.             6.6:1     6.6:1  
  4.               3:1       3:1  
  5.            16.3:1    16.3:1  
  6.            10.7::    10.7��  
  7.            1173::    1173��  
  8.             11::1     11��1  
  9.             23::1     23��1  
 10.           16.15::   16.15��

[Yapeng Wang]

Since your results from chartab suggestged that the representation in post #1 was probably correct, I tested the code from post #2 on your example data. That led to two changes to the code, and it now works as intended.

. gen cleaned_plc0107 = usubinstr(plc0107, "`=uchar(65533)'", ":", .)
(4 missing values generated)

. list cleaned_plc0107 plc0107 in 1/10, clean abbreviate(16)

cleaned_plc0107 plc0107
1. 12:1 12:1
2. 16::1 16��1
3. 6.6:1 6.6:1
4. 3:1 3:1
5. 16.3:1 16.3:1
6. 10.7:: 10.7��
7. 1173:: 1173��
8. 11::1 11��1
9. 23::1 23��1
10. 16.15:: 16.15��

Yes, it works. Thank you so much, William!

[William Lisowski]

The thanks go to Clyde, I would never have remembered the chartab command since I'm fortunate enough to spend my time analyzing well-mined public use data from large surveys that don't have that sort of data problem.