Can ZWNJ and ZWJ be marked as invalid characters in Burmese (my)?

Question

We have a project in Burmese project that has Zero Width Non-Joiner (u+200C) invisible characters which I would like to mark as Invalid in the Characters inventory, but we cannot. We get a message:

However it does not exist in the Alphabetic Characters tab.or any other tab of the Language settings.
I also found the the same was true for Zero Width Joiner (u+200D).

When I examined the ldml.xml file and found these curious lines:

<!--sil:punctuation-pattern pattern="‌" context="medial" /-->
<sil:punctuation-pattern pattern="‍" context="medial" />

Inside the quotes were the ZWNJ 200C and ZWJ 200D characters. I tried deleting these lines but that did not change Paratext’s behavior.

I also did an experiment in an English project and tried adding ZWNJ and ZWJ to the language settings. Even though they were added as non-standard diacritics, I was not able to see them in the character inventory.

So I am wondering is this a feature or a Bug? Is the behavior of ZWNJ and ZWJ built into the behavior of certain languages and cannot be changed? If not, how can I mark ZWNJ Invalid in my Burmese project? (Note: The language code for Burmese is my.)

Answer 1

CrazyRocky:

We get a message

To get that message, the item you’re wanting to mark as invalid has to be in either the characters list or the medial punctuation list. It might be worth copy/pasting those fields into another application that can show you invisible Unicode characters.

Comments

That is what I thought as well. But that does not seem to be the case. After repeatedly searching for those characters, including doing a search in the Settings and ldml files, the problem persisted.

I did an experiment today and added ZWNJ and ZWJ (200C and 200D) to an English project then opened the Character Inventory. The both were marked as Valid and could not be marked as Invalid. The same error message as above appeared. I will file a Bug report…

---

Proceeding with caution, if you really want to get rid of them, you could try seeing if they are in the project’s settings file, either the Settings or ldml xml files and removing them from there. Backup the files somewhere safe first, of course.

Blessings,

Answer 2

Hey CrazyRocky, I ran into an issue with \u200f (which Paratext inserted in some of my RTL projects after the verse numbers.) To remove it I did this:
Search: (\v [\d-]*)\u200f( )
Replace: \1\2

I don’t know if this will help with your situation, because it is a little different.

Answer 3

CrazyRocky:

Burmese project that has Zero Width Non-Joiner

I wonder if the zero characters exist in any of the alphabet characters themselves. When you look in characters inventories and select “show combinations,” does the Unicode value “09CD” show up in the “Unicode value” column? This occurs in the Bangla font as a zero-width joiner character.