Unrecognized Characters?

Question

Hi, I am using the latest version of Paratext, WIN10, PTXPrint. The Satere (SAT) language uses nasalized vowels. I have a special font installed to convert to nasalized vowels. The straight apostrofe (') is used to represent the glottal. When I print Genesis or Exodus using PTXPrint all these characters show up normally in the PDF document. But, Leviticus shows the apostrofe as a square (i.e. unrecognized character?) in the PDF? NUM DEUT MATT show vowels as squares? Config doesn’t help? Grateful if anyone can help me to correct this behaviour! Also, just tried converting the PDF to a Word document (DOCX) in order to replace the squares with the appropriate character, but where the squares appear suddenly half the sentence goes missing in the conversion? Just checked character and punctuation inventories where there a few problems, but the issue with unrecognized characters when I try to print with PTXPrint continues?

Answer 1

Hi anon165192.
If the layout is already defined as NFC, then the only thing I can think of from the XeTeX message is that you are not using the same font in Paratext and PTXprint.

From the XeTeX message you seem to want to use the font “MarkTNR Satere”. Is it possible that you are using another font in Paratext?
You can check if you have the same font configured in both programs.

image1341×423 91.7 KB

If not, try going to this option in PTXprint. Tab “Fonts + Sripts” and change these 4 fields to your font (Regular, Bold, Italic, Bold Italic). Then try to generate your PDF.

Saludos,
anon689242.

Comments

Hi anon689242,

Thanks for your help! I did try that but both Paratext and PTXprint are configured to use the MarkNTR Satere font!
After a lot of looking around and asking questions I finally found a temporary solution. Automatic keyboard switching
somehow got disabled in Paratext. So, Genesis and Exodus print out fine, but other books contained characters which
anon421222’t belong to the font I use (i.e. wrong keyboard = wrong input font)! After enabling keyboard switching I had to manually
find and replace each character and now the books print out just fine!

However, this is only a temporary solution. Some people have suggested that it is possible to configure Paratext so that I no
longer need to use a special font? But so far no one has explained how to achieve this? I have checked Convert Project and
Project Properties dialogs. Convert Project dialog doesn’t have the option for Composed Characters(NFC), but Project Properties
dialog is already configured to unicode and Composed Characters(NFC). Therefore, I believe the project is already configured?
But how do I configure Paratext to be able to use any font and still be able to input nasalized vowels, nasalized y and straight
apostrophe which represents the glottal?

Grateful for your help!

anon165192

Answer 2

It sounds to me like you might have a mixed encoding where parts of the project are using different codepoints. You may need someone to sit with you and get it straightened out.

You mention a “special” font. That word scares me. I assume that means it is custom-encoded and will only work with that font. You should get someone to help you completely convert your whole project to Unicode if it’s not already. Then you won’t need a special font, you can use almost any common Latin font.

Comments

Hi Lorna, thank you for your reply. I just added some things I’m trying in Paratext but which haven’t helped. Not sure I understand your point about mixed encoding and codepoints - could you explain more? I’m afraid I anon421222’t have anyone to sit with me as I’m and independent translator. If you could point me to someone who can help I would greatly appreciate it. Ironically, I’m using the same configuration for Leviticus as I used for Genesis and Exodus. Hence the difficulty in understanding how Paratext or PTXPrint is working to interpret Leviticus, Numbers, Deuteronomy and Matthew differently? The “special font” was created by the New Tribes print shop specialist years ago so I anon421222’t see any problem there. It works just fine in Word or Paratext except for the nasalized y which I can’t get to work in Paratext? Also, using the straight apostrophe (') to represent the glottal sometimes causes the word to be interpreted as two separate words (i.e. word break)? However, this same font has been used for Genesis and Exodus which print out just fine? I would greatly appreciate someone’s help on what you mention about converting the whole project to unicode (if it’s not already)! I anon421222’t know how to do that, but I will take a look into it and see if I can understand how to do that. Thanks for now! God bless!

---

Of potential use:
How do I make my Paratext project data consistent?

For a new project:

To avoid problems caused by having project data with a mix of composed and decomposed characters, when you create a new project in Paratext 8.0 (and later), there is a “Normalization” drop-down on the Advanced tab of the Project properties dialog.

The default Normalization is Composed characters (NFC). This means that when Paratext saves the project, it converts all project data to composed characters.
If you prefer to have Paratext convert all project data to decomposed characters when it saves the project, select Decomposed characters (NFD) when you create the project.

If you prefer to allow Paratext to store the project data as a mix of Composed and Decomposed characters, select Off (no normalization) when you create the project.
We do NOT recommend selecting “Off (no normalization)” unless the orthography of the project language uses diacritics in a non-standard way and the order in which the diacritics occur is important.

For an existing project:

If your project has data with a mix of composed and decomposed characters (likely due to using different keyboard layouts), do the following to force all the data to be consistent.

1. Click the tab of your project to make it the active tab.
2. Click the tab menu icon in your project, then from the Project menu, point to Project settings, and select Project properties.
3. On the Advanced tab, select one of the following in the Normalization drop-down list.

• Composed characters (NFC) (for data to be saved as composed characters).
• Decomposed characters (NFD) (for data to be saved as decomposed characters).

4. Click OK.
   Paratext changes all the data in your project to the normalization option you chose.
   Once the data of the project with the mix of composed and decomposed characters has been made consistent, have all team members Send/Receive the project.

If you prefer to allow Paratext to store the project data as a mix of Composed and Decomposed characters, select Off (no normalization) from the Normalization drop-down list.
We do NOT recommend selecting “Off (no normalization)” unless the orthography of the project language uses diacritics in a non-standard way and the order in which the diacritics occur is important.

---

Hi anon421222,

Thank you for that information! As far as I can tell my project is already configured as unicode and
Composed Characters (see screenshot below)! On the left is the Convert Project dialog which doesn’t
show the option for Composed Characters. But on the right the Project Properties appear to show that
the project is configured as unicode (i.e. unicode appears and button is disabled) and Composed Characters?

image.png1605×492 136 KB

Someone from PTXPrint told me to look at the end of the XeTeX Log and as per their suggestion I found the
following:

Missing character: There is no ꞌ (U+A78C) in font MarkTNR Satere! > ꞌ = U+0027
Missing character: There is no ỹ (U+1EF9) in font MarkTNR Satere! > ỹ = U+00FF
Missing character: There is no ẽ (U+1EBD) in font MarkTNR Satere! > ẽ = U+00E6
Missing character: There is no ĩ (U+0129) in font MarkTNR Satere! > ĩ = U+00F0

The Satere-Mawe (SAT) language uses nasalized vowels, nasalized y and the straight apostrophe for the glottal.
MarkTNR Satere is the font I use, but the missing character’s unicode number is different from the same character
in the font I use (i.e. after the > symbol).

Therefore, the retangular unrecognized character symbols which appear when I print using PTXPrint are the missing
characters above. Somehow the input in Paratext appears not to be using the MarkTNR Satere font although this is
the font which is configured in the project?

A couple of people have suggested that the project can be converted in some way so that I anon421222’t need to use a special
font. But no one has actually explained how to achieve this? I’m willing to do the work, but I anon421222’t know what to do! I
have tried looking into converting the project, but it apparently is already configured for unicode and Composed Characters?

Below is a screenshot of the error I get when using PTXPrint to print to PDF. The missing characters in Leviticus appear to be
the glottal, but in other books it is the nasalized vowels which are not being recognized?

image.png1298×800 122 KB

Your help is greatly appreciated! God bless!

anon165192

Answer 3

From what you say above, your MarkNTR Satere font isn’t Unicode compliant. It has replaced characters like y diaeresis with y tilde and those need fixing. I believe a simple find/replace routine would work.
Find U+0027 Replace U+A78C
Find U+00FF > U+1EF9
Find U+00E6 > U+1EBD
Find U+00F0 > U+0129

You may have other hacked codepoints though. You need to do that with every nasal vowel and any other special character in your orthography. Then you should be able to switch to using Charis SIL, but you’ll also need to switch keyboards. If you have the source file for your keyboard I could help you change it to output Unicode compliant characters.

Comments

Hi Lorna, not sure what I said that indicates that the font isn’t unicode compliant? Automatic keyboard switching somehow got disabled which means that a regular font was being used for input rather that the font I use. When I enabled keyboard switching I was able to Find and Replace characters manually. Also, not sure where you can Find and Replace using unicode numbers as you suggest? I’m afraid I anon421222’t know how to use unicode numbers in that way? However, I can send you the font for you to look at.
fontsetc_.zip (99.9 KB)

Answer 4

Thanks for the font! That’s helpful. What you have is what we sometimes call a hacked font or a “custom-encoded” font. We used to do that a lot pre-Unicode because that was the only way we could get the characters we needed. So, although it feels like Unicode to you, some of the characters were redefined from the actual Unicode. Thus, yours is probably the only font in the world that has this mapping. If you were to lose the association of your data from this font, no one would know what the characters should be (thus the square boxes)

For example, the Unicode character ÿ (U+00FF) was redefined to be a y with a tilde. In Unicode numbers, that would be ỹ (U+1EF9).

At this point you may not want to go through the process of converting your data to Unicode. But, if you were willing to do that and if someone could help you, I’d be willing to create a mapping file that would map your custom codepoints to the correct Unicode codepoints. I could do that based on what I see in the font.

You would need to do the data conversion and keyboard update at the same time.

For long-term data stability you really should convert to Unicode, but I anon421222’t know what process in the translation you are.

Comments

Hi Lorna,

You and I reached the same conclusion. I also believe this is the result of a hacked font. I’ll email you and anon165192 off-list to see what we can do.

james_post

Answer 5

You can find more details in the Paratext Help topic

My project text uses a “custom” font. How do I convert it to Unicode?

Referring to that item, Lorna is offering to provide a mapping (.map) file that could be used for the conversion. You’ll need a revised keyboard that will output Unicode characters (or you could find a keyboard that produces the characters you need and use that). After having switched to Unicode, you’ll have the choice of a great number of fonts, so that shouldn’t be an issue (except that the custom font that you have been using probably won’t work).

Comments

Hey anon806807, thank you for that! Yes, both Lorna and james_post are communicating with me and Lorna’s working on the .map. After converting to unicode I anon421222’t think I will need the font I’ve been using anymore.