Hidden Characters - Escape Sequences in JSON

Hello everyone,

I’m sending data to a webservice, via POST, PUT request…
I’m having problems with hidden characters that originated in the registrations.
I need to delete them because the webservice returns “no data sent”
To see them I send the json to Webhook.site. See data in part of the json:

"phone": "Office: 27999172225 \n M.V Aline: 27999015331\n M",
"email": "janaina\u001f\[email protected]",
"phone": "27 999195266 (call) 14 98220-8842 (WhatsApp)",
"phone": "(27 ) 3223-1880 - 99915-7066 B?rbara",
"email": "Name/ EMAIL Phone\r\[email protected]",
"social_ratio": "\u0081Endocrinovet (Sarah Soukef)",
"login": "Aline Vianna de Almeida \n Private",
"email": "[email protected]\r",

I believe they are enters, or copies from somewhere that Clarion did not handle the character, and pasted into the fields.( \n, \r, ?, \u00…,)
Nothing identifies this, the only way to fix it was to identify the field, delete it and retype it.
But since there are so many, this is very confusing.
Can anyone help me?

Thank you very much.

Hi Jorge

In json, normally things like CRLF (‘<13,10>’) are represented \r\n and tabs (‘<9>’) as \t and so on. But taking your requirement to remove ( \n, \r, ?, \u00…,) at face value, you could do this simply using StringTheory or similar.

For example, something like:

st    stringTheory
x     long,auto

  code
  st.setvalue(myText)
  st.remove('\n')
  st.remove('\r')
  st.remove('?')
  x = 1
  loop
    x = st.findchars('\u', x)
    if ~x or x > st.length() - 5 then break.
    st.removeFromPosition(x, 6)  ! remove \u1234
  end
  myText = st.getValue()

That is off the top of my head and not tested, and there are ways to make that faster but I have tried to keep it simple.

You may wish to replace \n \r and ? with a space instead of removing them.

  st.replace('\r\n',' ')
  st.replace('\n',  ' ')
  st.replace('\r',  ' ')
  st.replace('?',   ' ')

anyway let us know how you go

cheers

Geoff R

#edit 1. I originally was checking the chars after \u were digits, but realize that they are hex chars so removed that.

#edit 2. Suggested option of replacing with space rather than removing

#edit 3. Cleaned code a little and made \r\n replaced by one space instead of two on suggestion re replacing with space rather than removing

#edit 4. If you decide to replace by space (rather than remove) but are worried about extraneous spaces, then you could only put a space if there wasn’t already one either before or after:

! I'll use '<32>' for a space for clarity
  st.replace('\r\n', '?')
  st.replace('\r',   '?')
  st.replace('\n',   '?')
  st.replace('<32>?','?')
  st.replace('?<32>','?')
  st.replace('?',    '<32>')

You haven’t said what tools you are using to generate the JSON and POST/PUT it to the server?

The reason I mention it is because I thought that some of the available Clarion tools had functions, or automatically, removed invalid JSON characters?

Having said that, it would still be a good idea to clean up your data and install some process to prevent saving these unnecessary characters when they are entered.

If you think of this in revers. the solution might be better creating a filter of included characters and not the excluded ones. ie A-Z, a-z, 0-9 and any other special characters. I know this is rather easy using string theory.

KeepChars(string Alphabet)

If the source is RTF then that could be a different cause and will require a different solution.

The other option is encoding

Hi Jeff

those are certainly useful StringTheory methods and these days you can use st.KeepCharRanges() and st.KeepRemoveCharRanges() that were introduced in ST version 370.

for example to keep alphabetic chars and digits (and remove the rest):

st.KeepCharRanges('A-Za-z0-9') 

However this is not appropriate in this situation as Jorge wants to remove ‘\n’ and ‘\r’ strings which are not single characters. And removing backslashes would still leave the ‘n’ and ‘r’. Hence my answer above to either remove (or replace with a space) the unwanted strings.

cheers

Geoff R

1 Like

Hi Geoff, you always help me!
Thank you very much!
I’ll show you what happens:
1 - I have a TPS with data that is sent via cUrl with a POST by the magnificent cjson Mike


Code:

    if Loc:Id=7
        Loop i# = 1 to Records(Que:TPS)
            Get(Que:TPS, i#)
            If Qtps:tagIco=1
                ?PROGRESSpac{prop:progress} = ?PROGRESSpac{prop:progress} + 1
                DISPLAY()       
                MCLI:COD_CLIN = Qtps:codMv
                If Access:MCLI.Fetch(MCLI:Key_CodClin) = level:benign
                    arry:dadoCadastro[1] = EliminaOculto(MCLI:NOME)
                    arry:dadoCadastro[2] = EliminaOculto(MCLI:ENDERECO)
                    arry:dadoCadastro[3] = EliminaOculto(MCLI:BAIRRO)
                    arry:dadoCadastro[4] = EliminaOculto(MCLI:COMPLEMENTO)
                    arry:dadoCadastro[5] = EliminaOculto(MCLI:CIDADE)
                    arry:dadoCadastro[6] = EliminaOculto(MCLI:CEP)
                    arry:dadoCadastro[7] = Email_NComercial(MCLI:EMAIL)
                    arry:dadoCadastro[8] = Email_Comercial (MCLI:EMAIL)
                    arry:dadoCadastro[9] = EliminaAcentos_(MCLI:CELULAR)
                    arry:dadoCadastro[10]= EliminaAcentos_(MCLI:CNPJ)
                    arry:dadoCadastro[11]= EliminaAcentos_(MCLI:IE)
                    arry:dadoCadastro[12]= EliminaAcentos_(MCLI:REF)
                    arry:dadoCadastro[13]= EliminaAcentos_(MCLI:FONE)
                    arry:dadoCadastro[14]= EliminaAcentos_(MCLI:LOGIN)
                    
                    Send:PostParams=|
                        '{{'&|
                        '"codmv":"'&MCLI:COD_CLIN&'",'&|
                        '"inicio":"'&format(MCLI:DATAS,@d10-)&'",'&|
                        '"razaosocial":"'&clip(json::ToUtf8(arry:dadoCadastro[1]))&'",'&|
                        '"nomefantasia":"'&clip(json::ToUtf8(arry:dadoCadastro[1]))&'",'&|
                        '"endereco":"'&clip(json::ToUtf8(arry:dadoCadastro[2]))&'",'&|
                        '"bairro":"'&clip(json::ToUtf8(arry:dadoCadastro[3]))&'",'&|
                        '"comp":"'&clip(json::ToUtf8(arry:dadoCadastro[4]))&'",'&|
                        '"cidade":"'&clip(json::ToUtf8(arry:dadoCadastro[5]))&'",'&|
                        '"uf":"'&clip(MCLI:UF)&'",'&|
                        '"cep":"'&clip(arry:dadoCadastro[6])&'",'&|
                        '"ref":"'&clip(json::ToUtf8(arry:dadoCadastro[12]))&'",'&|
                        '"fone":"'&clip( arry:dadoCadastro[13])&'",'&|
                        '"wsap":"'&clip(arry:dadoCadastro[9])&'",'&|
                        '"cnpj":"'&clip( arry:dadoCadastro[10])&'",'&|
                        '"ie":"'&clip(arry:dadoCadastro[11])&'",'&|
                        '"email":"'&clip(arry:dadoCadastro[7])&'",'&|
                        '"emailfin":"'&clip(arry:dadoCadastro[8])&'",'&|
                        '"cod_cid":"'&clip(MCLI:COD_CID)&'",'&|
                        '"dia_fatura":"'&clip(MCLI:DIA_FATURA)&'",'&|
                        '"login":"'& clip(arry:dadoCadastro[13])&'",'&|
                        '"senha":"'&clip(MCLI:SENHA)&'",'&|
                        '"logo":"'&clip(MCLI:Logo)&'",'&|
                        '"ativo":"'&clip(MCLI:Ativo)|
                        &'"}'

                    Loc:Id    = 0                        
                    Send:CustomRequest  ='GET'
                    Send:Url    = clip(Glo:Url)&clip(Loc:cadastro)&'/'&Qtps:codMv
                    DO SendRequest!codMV
                    jRoot &= jParser.Parse(stReturn)
                    !=============== Se encontrar codMV                        
                    IF NOT jRoot &= NULL
                    !=============== Retorna o ID
                        Loc:Id = jRoot.GetValue('id')
                        MCLI:CodSite = Loc:Id
                        put(MCLI)
                    End
                   !=============== Se não encontrar codMV                        
                    If Loc:Id = 0
                        Send:CustomRequest='POST'
                        Send:Url    = clip(Glo:Url)&clip(Loc:cadastro)
                        DO SendRequest
                        jRoot &= jParser.Parse(stReturn)
                        IF NOT jRoot &= NULL
                            MCLI:CodSite = jRoot.GetValue('id')
                            put(MCLI)
                        End    
                    End 
                    jRoot.Delete()
                End
            End

!

Where: EliminaOculto()

Data
str                 StringTheory

Code
str.SetValue(Par:Str)
str.Remove('<9>')  !tab
str.Remove('<13>') !cr 
str.Remove('\r') 
str.Remove('?') 
str.Remove('\n')
return(str.GetValue())

Note: json::ToUtf8() does not cause these characters. I have already tested without it, so I can understand.
In fact, everything was on the same command line
json::ToUtf8(EliminaOculto()) the result is the same
The character was inserted in the field, when I open the registration screen, if I copy and paste it into a .TXT they also appear.

It looks like a TAB or enter…?

2 - I do the test by sending it to the website webhook.site, in this case the one selected from the list above:
Doc Vet

See the character appears in the email field

"email": "[email protected]\r,[email protected]",

Here we have the system screen:

And here is the data in the TPS…

and finally the TPS itself… lol

MCLI.zip (209,9,KB)

Note: The same happens with the characters I listed at the beginning of my question, that is, they do not appear in the field even with the TPS open but when pasted or sent…

I sent all this because sometimes there is missing data in my question, so I think everything is there.

I had already tried replacing it with eliminaOculto(), but I couldn’t, so I looked for help here on ClarionHub

Thank you very much in advance!!!

The solution is to make sure your data is clean before converting to json.

You are correct and I meant to delete that comment. :-). that was when I realized it was something embedded.

1 Like

There is a published list of JSON Escapes in RFC 4627. The \u#### will present some challenges.

\" - Escapes a double quote (")
\\ - Escapes a backslash ()
\/ - Escapes a forward slash (/), optional
\b - Escapes a backspace
\f - Escapes a form feed
\n - Escapes a newline
\r - Escapes a carriage return
\t - Escapes a tab
\u - Escapes a Unicode character

These links may help:

Hello again Jorge

pleased to be able to help.

as Jeff has shown, the email field has a carriage return ‘<13>’ (‘0D’ in hex)

your EliminaOculto() includes

str.Remove('<13>') !cr 

HOWEVER you are NOT using that on your email fields

arry:dadoCadastro[7] = Email_NComercial(MCLI:EMAIL)
arry:dadoCadastro[8] = Email_Comercial (MCLI:EMAIL)

hence it is not getting cleaned.

maybe clean it all when you construct your PostParams. So instead of

arry:dadoCadastro[5] = EliminaOculto(MCLI:CIDADE)

<snip>

'"cidade":"'&clip(json::ToUtf8(arry:dadoCadastro[5]))&'",'&|

you instead forget about having the array and just:

prepareField('cidade',MCLI:CIDADE) &|

where prepareField() incorporates the code from your EliminaOculto() plus a bit more:

prepareField  procedure(string Par:fieldName, string Par:Str)
str    StringTheory
  code
  str.SetValue(Par:Str)
  str.Remove('<9>')  !tab
  str.Remove('<13>') !cr 
  str.Remove('\r') 
  str.Remove('?') 
  str.Remove('\n')
  return '<34>' & par:fieldName & '<34>:<34>' & |
         clip(json::ToUtf8(str.GetValue())) & '<34>,'

(I am using ‘<34>’ to show double quotes for clarity)

or if you want to keep your EliminaOculto(), then simply:

prepareField  procedure(string Par:fieldName, string Par:Str)
  code
  return '<34>' & par:fieldName & '<34>:<34>'       & |
         clip(json::ToUtf8(EliminaOculto(Par:Str))) & |
         '<34>,'

I am not sure what your other functions

Email_NComercial
Email_Comercial
EliminaAcentos_

do, but they could perhaps include similar code to prepareField() and call your EliminaOculto.

hth & cheers

Geoff R

Good morning everyone
The solution was here but I couldn’t figure it out because the code is before it.
The cjson component from friend Mike Duglas handles this Escape character handling.
Our attempts using the filters didn’t work, apparently st.remove() doesn’t see these Escapes, neither inside the field nor in the complete json string.
Making json without processing generates Escapes.

Solution using Mike Duglas cjson

 Data

 tabSt                 &cJson
 lenSt                  long
 sndSt                 &String

 Code

 tabSt &= json::CreateObject()

 tabSt.AddStringToObject('inicio',format(Loc:datInicio,@d10-))
 tabSt.AddStringToObject('nome',json::ToUtf8(MZP:NOME))
 tabSt.AddStringToObject('rg',json::ToUtf8(MZP:RG))
 tabSt.AddStringToObject('nasc',format(Loc:Nasc,@d10-)
 tabSt.AddStringToObject('sexo',json::ToUtf8(MZP:SEXO))
 tabSt.AddNumberToObject('codmv',MZP:Cod_Pac)

 lenSt# = len(tabSt.ToString(TRUE))
 sendPams &= NewString (lenSt#)
 sendPams = tabSt.ToString(TRUE)

I sent it to the webservice and the bank recorded an error return, which was what I needed.
Thank you all very much in advance!