Greetings!
read the release highlights here:
the post has links to the full docs, lots of good stuff
Hello
does anybody know what need to be set to use utf-8 inside Clarion editor?
I changed settings on the IDE (Tools->Options->Text Editor->Choose Encoding->UTF-8)
but I still can not save utf-8 text inside Clarion editor code.
When I use include(‘./myclw.clw’) and when myclw.clw is saved as utf-8, it compiles and works fine.
Thanks.
I think the line;
!UTF-8
at the top of the file may help you.
I pasted some utf-8 text from the clipboard into the editor, and it offered to save the file as utf-8 (ie with a BOM) but I also added the above comment at the top of the file so that I’d know (for my reference) the file was UTF-8. But from the docs this comment makes the editor treat the file as utf-8, even if the BOM is missing.
Which build are you on? Only the most recent build supports unicode in MESSAGE.
And, if you’re in an app, is this code in an embed point, according to the original release notes you can’t have raw text like that in an embed point.
I’m on 12.0.14148, check below.
Not sure what is tgst but I presume that utf-8 is not allowed in the app’s embed?
sorry - phone type - was supposed to be “that”.
Correct, a this point utf-8 text won’t survive in an embed point. See
Known limitation — embeds
Generated code from the native template engine is always ANSI. Non-ANSI text typed into an embed cannot survive as raw characters — write U'...' with unit metachars, or put the code in a UTF-8 INCLUDE file.
That may change in the future.
You can specify the chars using the CHR() syntax, or the <nn,nn,nn> syntax.
Hi
I just tested here
Just add a U before the string
MESSAGE(U'今日 は素晴らしい日だ。')
That should work.
Mark
OK thanks for the explanation.
Hmm it does not save it (using app embed code).
Bruce answered this point, but I’ll give more details
Known limitation — Unicode text in embeds
The native template engine always generates ANSI .clw files — no BOM, no !UTF8 line. That’s a property of code generation, not of your handwritten source.
So when you type non-ANSI text (emoji, Japanese, Greek, accented glyphs) directly into an embed point, those characters are written into an ANSI-encoded generated file. The compiler then reads them through the active codepage and mangles them — e.g. U’
’ comes back with LEN = 4 instead of 2. The raw glyphs simply can’t survive the round-trip through generation.
Two ways around it — both keep the source of the embed pure-ASCII so nothing gets mis-decoded:
- Use metacharacters instead of raw glyphs. Write the literal with numeric code units rather than the character itself:
- U’<9731>’ →
(BMP character) - or build it with CHR(1F4A9h,1) →
(astral, returns a surrogate pair)
Because the embed text is all ASCII digits, it passes through the ANSI generator untouched, and the runtime produces the correct wide character.
- Move the code to a UTF-8 INCLUDE file. Put the Unicode literals in a file you control the encoding of (UTF-8 with BOM, or !UTF8 on line 1), then just reference it from the embed:
INCLUDE(‘MyUnicode.clw’),ONCE
The template never rewrites that file, so the raw characters stay intact.
Rule of thumb: For now, anything the template generates is ANSI — keep typed embeds to ASCII, and express Unicode either numerically (metachars / CHR) or in a hand-authored UTF-8 include.
@RZaunere I have not been able to figure out how to enter Surrogate Pairs in String Literals. I thought it was <UShort><UShort>. I would also like to be able to enter UTF-32 into Strings.
For Pile of Poo the UTF code is U+1F4A9 or Surrogates D83D + DCA9. You can see in the below screen capture it works fine as CHR(1F4A9h,1) or CHR(0D83Dh,1) & CHR(0DCA9h,1) but the 3 tries like '<0D83Dh>' & '<0DCA9h>' do not work?
I really do NOT like the look or feel of CHR( #, 1) to get Unicode. I hate the",1". I would prefer you add something like ChrW( # ) that I have in this sample code.
PROGRAM
MAP
ChrW PROCEDURE(LONG inUtf32),USTRING !Wrap CHR(,1) for Wide Text 'cause CHR(,1) is UGLY
END
CapUStr USTRING(200) !Wide String Caption
MsgUStr USTRING(8000) !Msg Bidy
wLf EQUATE('|')
wTab EQUATE('<9>')
CODE
SYSTEM{PROP:MsgModeDefault}=MSGMODE:CANCOPY ! ; Message('Poo '& 0D83Dh &' '& 0DCA9h,'Dec') ; return
CapUStr = 'Surrorgate Test UTF-32' !https://en.wikipedia.org/wiki/List_of_emojis
MsgUStr = U'' & | !U=tell compiler make this Unicode textin the EXE
wLf & CHR(1F4A9h,1) & wTab & 'U+1F4A9 Poo 1 Utf-32 (no surrogates)' &| !Pile of Poo 1 UTF-32
wLf& CHR(0D83Dh,1) & CHR(0DCA9h,1) &wTab & 'D83D,DCA9 Poo 2 Surrogates CHR(,1)' &| !Pile of Poo 2 Surrogates using CHR()
wLf& ChrW(0D83Dh) & ChrW(0DCA9h) &wTab & 'D83D,DCA9 Poo 2 Surrogates ChrW()' &| !Pile of Poo 2 Surrogates using ChrW() that calls CHR()
wLf& '<55357>' & '<56489>' &wTab & '<<55357,56489> Poo 2 Surrogates Literal Decimal' &|
wLf& '<0D83Dh>' & '<0DCA9h>' &wTab & '<<D83D,DCA9> Poo 2 Surrogates Literal BE Hex' &|
wLf& '<03DD8h>' & '<0A9DCh>' &wTab & '<<3DD8,A9DC> Poo 2 Surrogates Literal LE flipped' &|
wLf & |
wLf & ChrW(0E9h) & wTab & 'U+00E9 Caf' & ChrW(0E9h) &' (precomposed)' &| !U+00E9 LETTER e WITH ACUTE ACCENT https://en.wikipedia.org/wiki/Precomposed_character
wLf & ChrW(65h) & ChrW(301h) & wTab & 'U+0301 Cafe'& ChrW(301h) &' (e + combining`)' &| !e + U+0301 COMBINING ACUTE ACCENT https://en.wikipedia.org/wiki/Combining_character
wLf & ChrW(20ACh) & wTab & U'U+020AC Euro' &|
wLf & ChrW(1F603h) & wTab & 'U+1F603 Smile' !Smiling Face with Open Mouth UTF-21
Message(MsgUStr,U'*RTL* '& CapUStr,ICON:Clarion, | !Clarion Msg does not show Emoji Button
U'Close|' & CHR(1F4A9h,1) &'|'& '<03DD8h><0A9DCh>' &'|'& '<0D83Dh,0DCA9h>') !Poo a Surrogates in String so Flip LE :( ugg
ChrW PROCEDURE(LONG inUtf32)!,USTRING Chr(U,1) is ugly
CODE
RETURN Chr(inUtf32,1)
!CHR(,1) Syntax for UTF is UGLY and I would prefer a single parameter like WChar(LONG)
!note: I think the CHR(Number,1) is processed by the Compiler
! while the CHR(Variable,1) is a runtime call to UCHR()
It occurred to me to look at a Hex Dump of the UString. You see the first 2 lines that work using CHR(#,1) result in the UString having the correct Surrogate Pair 3DD8+a9dc.
But for the lines with string <hex short> that the <HHLL> in the U string has the LL as 00.
E.g. <55357> and <0D83Dh> get flipped to Little Endian <3DD8h> but changes the D8h to 00 resulting in <3D00h> ? The RTL is just seeing the first Byte of a <UShort>.
HexDumpW PROCEDURE(USTRING inUtf, BOOL BigEnd=0) !Utf 16 in Hex formated for Message() to Debug and verify code, to see Surrogates
Dmp ANY
X LONG
AUnit USTRING(2)
Wide2 STRING(2),OVER(AUnit) !How to access the USHORT ? This Over should
HHHHs STRING(5)
CODE
LOOP X=1 TO LEN(CLIP(inUTF)) !not BY 2
AUnit = InUTF[X] !was Wide2 = InUTF[X]
CASE Wide2
OF '<0,0>' ; BREAK !Null is the End, don't show 0000
OF '<13,0>' ; CYCLE !Just 10s no 13s
OF '<10,0>' ; Dmp=Dmp & '|' ; CYCLE !No need to see 0A00 for Debug
END
HHHHs=CHOOSE(~BigEnd, HexS1(Wide2[1])&HexS1(Wide2[2]) , HexS1(Wide2[2])&HexS1(Wide2[1]))
CASE IsSurrogate(Wide2)
OF 1 ; HHHHs[5] = '+' !Surrogate 1 aka High with +Low instead of Space
OF 2 ; HHHHs=lower(HHHHs) !Surrogate 2 aka Low in lower case
END
Dmp = Dmp & HHHHs
END
RETURN Dmp
HexS1 PROCEDURE(STRING InStr1) !,STRING !HEX digits of 1 byte String
In BYTE,OVER(InStr1)
Hx STRING('0123456789ABCDEF')
CODE
RETURN Hx[BSHIFT(in, -4) + 1] & Hx[BAND(in, 0FH) + 1]
IsSurrogate PROCEDURE(STRING Utf16WideChar) !,BYTE !1=High (1st); 2=Low (2nd); 0=Not Surrogate
ValUtf16 USHORT,OVER(Utf16WideChar) !Better pass in STRING(2) or ????
HiLoNo BYTE
CODE
CASE ValUtf16
OF 0D800h TO 0DBFFh ; HiLoNo=1 !High 1st
OF 0DC00h TO 0DFFFh ; HiLoNo=2 !Low 2nd
END
RETURN HiLoNo
@RZaunere the RTL needs an IsSurrogate() function. The one above needs to take a USTRING and overload with USHORT. I like that it returns 1, 2 or 0 so all 3 conditions are covered by one function.
The one-literal form you’re looking for already exists. What’s missing is only that the docs need to expand on it.
The literal you want is U’…’ with the metachars inside it. In a U’’ string literal, values are UTF-16 code units (decimal), not bytes. So Pile of Poo as a single literal is:
clarion
MsgUStr = U’<55357,56489>’ ! D83D,DCA9 as decimal units
MsgUStr = U’Poo <55357,56489> done’ ! composes like any literal
Verified this today: U’<55357,56489>’ produces exactly the same two units as CHR(1F4A9h,1) and as
CHR(0D83Dh,1) & CHR(0DCA9h,1).
The reason your three attempts didn’t work — they were all narrow literals, and in a narrow (non-U) literal has meant one byte since the beginning. The value is silently truncated to its low byte: your ‘<55357>’ actually emits byte 3Dh (=) and ‘<56489>’ emits A9h (©) — which is exactly the junk you saw. (archaeology from your hex attempts: the narrow lane does parse 0D83Dh-style hex — always has, undocumented — but it byte-truncates all the same. The U’’ lane is currently decimal-only, and rejects values above FFFFh rather than truncating.)
Your note at the bottom of the code is right: constant CHR(n,1) is folded by the compiler into the literal’s units; only a variable argument becomes a runtime call.
On the “UGLY” front — two things are now queued:
- A single-parameter builtin taking the code point straight (as in your ChrW shape), so the ,1 disappears.
- Hex — and full code points — in U’’ metachars, so U’<0D83Dh,0DCA9h>’ works as you wrote it, and _ideally U’<1F4A9h>’ expands to the surrogate pair by itself.
On the emoji in MESSAGE buttons: the units get there intact — the whole MESSAGE surface (text, caption, icon, buttons) is wide in this beta. What you’re seeing is rendering: dialog buttons are drawn by GDI, and GDI has no color-emoji path and only spotty font fallback for above-BMP glyphs (same story as TEXT controls). So it’s cosmetic, not data loss — copy the text off the dialog and the surrogates are all there.
One more thing: you have a passing comment in the example that RTL Message() “stopped showing all the content.” The argument path has no length caps (checked), so if you have a case where content that used to display now doesn’t, we’d like the repro — that would be its own bug, not part of the literal issue.The one-literal form you’re looking for already exists. What’s missing is only that the docs need to expand on it.
The literal you want is U’…’ with the metachars inside it. In a U’’ string literal, values are UTF-16 code units (decimal), not bytes. So Pile of Poo as a single literal is:
clarion
MsgUStr = U’<55357,56489>’ ! D83D,DCA9 as decimal units
MsgUStr = U’Poo <55357,56489> done’ ! composes like any literal
Verified this today: U’<55357,56489>’ produces exactly the same two units as CHR(1F4A9h,1) and as
CHR(0D83Dh,1) & CHR(0DCA9h,1).
The reason your three attempts didn’t work — they were all narrow literals, and in a narrow (non-U) literal has meant one byte since the beginning. The value is silently truncated to its low byte: your ‘<55357>’ actually emits byte 3Dh (=) and ‘<56489>’ emits A9h (©) — which is exactly the junk you saw. (archaeology from your hex attempts: the narrow lane does parse 0D83Dh-style hex — always has, undocumented — but it byte-truncates all the same. The U’’ lane is currently decimal-only, and rejects values above FFFFh rather than truncating.)
Your note at the bottom of the code is right: constant CHR(n,1) is folded by the compiler into the literal’s units; only a variable argument becomes a runtime call.
On the “UGLY” front — two things are now queued:
- A single-parameter builtin taking the code point straight (as in your ChrW shape), so the ,1 disappears.
2. Hex — and full code points — in U’’ metachars, so U’<0D83Dh,0DCA9h>’ works as you wrote it, and _ideally U’<1F4A9h>’ expands to the surrogate pair by itself.
On the emoji in MESSAGE buttons: the units get there intact — the whole MESSAGE surface (text, caption, icon, buttons) is wide in this beta. What you’re seeing is rendering: dialog buttons are drawn by GDI, and GDI has no color-emoji path and only spotty font fallback for above-BMP glyphs (same story as TEXT controls). So it’s cosmetic, not data loss — copy the text off the dialog and the surrogates are all there.
One more thing: you have a passing comment in the example that RTL Message() “stopped showing all the content.” The argument path has no length caps (checked), so if you have a case where content that used to display now doesn’t, we’d like the repro — that would be its own bug, not part of the literal issue.