How to detect and delete "invisible/hidden" elements within strings

[Benno Schoenberger]

Dear statalist members,
after half an hour of unsuccessfully scrolling through the results of various searches in the forum, I have come to the conclusion that my problem is apparently not so common as to have been discussed here too often, and I am turning to you.

I'm having trouble with seemingly hidden or invisible elements within strings. As part of a larger project for automatically checking filter conditions within survey datasets, I'm extracting more or less human-readable filter conditions from a database using ODBC and processing them further in Stata. In doing so, these conditions are sometimes broken down into multiple individual conditions, and elements like from ... to are recoded to inrange() or var = 1 & 2 & 3 & 4 to inlist(var,1,2,3,4), etc.
One of the final steps is to test which error messages Stata outputs when these filter conditions are executed, for example, as count if `filter'
There are some filters that appear to be Stata-compliant (Stata functions are correct, all parentheses and quotation marks are complete, etc.), but still return error code 133.
After exporting a data sample via Dataex for this post, I saw what was probably the crucial clue to the actual problem. This is the unmodified Dataex output:

* Example generated by -dataex-. For more info, type help dataex
clear
input float id str697 filter float errorcode
1 "inlist(td00352,1)|inlist(td00353,2)" 111
2 "inlist(td00353,1)
"                   133
end

As you can clearly see here, Dataex also seems unable to handle my problem correctly, as some kind of line break is also being output for observation 2. These hidden line breaks are apparently also the problem that leads to a Stata error code 133 when they are part of a filter condition.

In short: is there a simple and quick way to identify and delete these hidden elements within strings?

As the example above reveals, I am unfortunately unable to generate a comprehensible and executable data example for you, so I hope that someone will still take pity on me and help me out.
Kind regards,
Benno Schönberger

[Benno Schoenberger]

Problem solved!

For anyone else who might be struggling with the same problem: I've been able to find and successfully replace a set of potentially problematic ASCII elements with the following syntax:

// remove ASCII Codes 96 (back tick), 9 (tab stop), 13 (carriage return), 10 (line feed) and 160 (non-breakable space)
        local asciireplace `" 96="'" 9=" " 10=" " 13="" 160=" " "'
        while (!missing(trim(`"`asciireplace'"'))) {
            gettoken asciicode asciireplace : asciireplace , parse("=")
            confirm integer number `asciicode'
            gettoken delim asciireplace : asciireplace , parse("=")
            assert (`"`delim'"'=="=")
            gettoken replacetext asciireplace : asciireplace
            quietly : count if strmatch(filter,"*"+char(`asciicode')+"*")
            if (r(N)>0) {
               display as text  in smcl `"{tab} substituting {it:`r(N)'} special character(s) with ASCII code {it:`asciicode'} in {it:filter} with "{it:`macval(replacetext)'}""'
                quietly : replace filter=subinstr(filter,char(`asciicode'),`"`macval(replacetext)'"',.)
            }
        }