"Converting" a semi-Unicode project to full Unicode

Question

We have a project with a partial Unicode mapping. The user has a custom font, which displays non-Unicode characters for normal Unicode values.

The Help topic: My project text uses a “custom” font. How do I convert it to Unicode? says to apply a mapping file (which we have), and import the existing books into a new project.

However, there are also project notes that need to be converted, perhaps some Biblical Terms, and maybe other files. The Help topic doesn’t seem to cover those files.

Is there a way to ensure all of these files get converted properly? Will “Convert Project” apply a mapping to the new files?

Thank you,

james_post

Answer 1

Convert project does not allow for a mapping file. The recommendation would be to convert the text files within Paratext using Find/Replace and then manually do the Find/Replace on the other files (mostly the.xml flies in the project folder).
The other option would be to make use of a tool like CC where you create a table (map) of the changes to be made and then apply that to the various flies that need to be fixed. This would likely be most of the .xml flies in the project folder. The advantage of CC is that you are pretty sure what changes are being made. Since these are all unusual characters there should not be issues with accidently corrupting a file.
Before doing any conversion of those files I would make sure there is a good backup of the project.

Comments

Sounds like this is a job suited for a Transliteration project using Encoding converters. The advantage here is that you do not actually change your original project and can experiment with making any number of regular changes to the project until you get the conversion perfect. Once you have perfected the conversion you can copy the transliteration project text back over to the base project and delete the transliteration project.
The downside is that there is a bit of a learning curve to using Encoding Converters. It would be good to get help from an experienced person to make this as painless as possible…

---

Unfortunately I don’t believe that the transliteration project does anything more than the text itself.

---

I agree, it does not change any of the USFM markers, nor will it touch any non-publishable text such as \id lines and \rem remarks. But it can change all publishable text in all books, including peripherals, and text in footnotes, headings, etc. If that is what @james_post is looking for then I think it is a good option.

Answer 2

If the project is Unicode (in Paratext) and the issue is that certain characters are not the correct code points then you should be able to do some basic find/replace to change the characters. I would set the font to an appropriate Unicode font and then make the changes as needed. This shouldn’t be about a conversion since the project is already Unicode.

​Blessings,

Comments

I understand where you’re coming from, but I am more concerned about the supplemental files (not the main SFM files). Though I guess we could also do a find/replace in those too.

That being said, I’m not sure which files to do the change in…

Also, if it were possible via Convert Project, it would maintain the project history. But that may not be possible in this case.

Thanks,

james_post

Answer 3

I think james_post is wanting the changes to include PT Notes, Biblical Terms renderings, Wordlist entries (including morphology), etc.
I think these are mainly stored in xml files that can be directly edited if one has the courage. But can anyone state definitively what would happen with decisions and approvals related to these tools?

Comments

The same TechKit file that modifies the sfm files can also be used to modify the XML control files, including the Wordlist, Parallel passages and Biblical Terms files, even the Notes files and other files that one might want to update. The encoding converter tools include an XML data converter, which should help. So this is technically feasible and will eliminate the need to redo those checks wholesale, though some legitimate errors may arise due to the conversion.
That said it will take time and someone with a good grasp of the tool and what files need to be modified. For example:

• I would export the Wordlist to a single XML file, make the conversion on that, and re-import it rather than working on the three Wordlist control files (hyphenatedWord.txt, SpellingStatus.xml and WordAnalysis.xml).
• For notes you should only change the <SelectedText>, <ContextBefore> and <ContextAfter> fields, not the <Contents> (I would imagine)
• If you have interlinear texts that will be another level of complexity I am not very familiar with