On Sat, 26 Oct 2019 at 03:11, Zach Laine via Boost-users < boost-users@lists.boost.org> wrote:

About 14 months ago I posted the same thing. There was significant work that needed to be done to Boost.Text (the proposed library), and I was a bit burned out.

Now I've managed to make the necessary changes, and I feel the library is ready for review, if there is interest.

This library, in part, is something I want to standardize.

It started as a better string library for namespace "std2", with minimal Unicode support. Though "std2" will almost certainly never happen now, those string types are still in there, and the library has grown to also include all the Unicode features most users will ever need.

Github: https://github.com/tzlaine/text Online docs: https://tzlaine.github.io/text

If you care about portable Unicode support, or even addressing the embarrassment of being the only major production language with next to no Unicode support, please have a look and provide feedback.

I gave a talk about this at C++Now in May 2018, and now it's a bit out of date, as the library was not then finished. It's three hours, so, y'know, maybe skip it. For completeness' sake:

https://www.youtube.com/watch?v=944GjKxwMBo&index=7&list=PL_AKIMJc4roVSbTTfHReQTl1dc9ms0lWH

https://www.youtube.com/watch?v=GJ2xMAqCZL8&list=PL_AKIMJc4roVSbTTfHReQTl1dc9ms0lWH&index=8

Zach

(as a power user) I would be interested to have such library in Boost and already had plan to try Boost.Text in my next C++ project with text. I am following the discussions happening in SG16 and understand that there are some differences with the parts that will be proposed for standardisation (as ThePHD explains in his talk). Though honestly both approaches seems to solve my problems so I'm open to trying both. If boost.text is stable today, I'm happy to use it (at least to replace ICU and have a proper unicode text type). A. Joël Lamotte

I saw that talk (on video) and was quite impressed but never got to try your library out. Would be happy to use a library like this! Can you tell me how it relates to the proposed std? /Peter On Sat, Oct 26, 2019 at 3:11 AM Zach Laine via Boost wrote:

About 14 months ago I posted the same thing. There was significant work that needed to be done to Boost.Text (the proposed library), and I was a bit burned out.

Now I've managed to make the necessary changes, and I feel the library is ready for review, if there is interest.

This library, in part, is something I want to standardize.

It started as a better string library for namespace "std2", with minimal Unicode support. Though "std2" will almost certainly never happen now, those string types are still in there, and the library has grown to also include all the Unicode features most users will ever need.

Github: https://github.com/tzlaine/text Online docs: https://tzlaine.github.io/text

If you care about portable Unicode support, or even addressing the embarrassment of being the only major production language with next to no Unicode support, please have a look and provide feedback.

I gave a talk about this at C++Now in May 2018, and now it's a bit out of date, as the library was not then finished. It's three hours, so, y'know, maybe skip it. For completeness' sake:

https://www.youtube.com/watch?v=944GjKxwMBo&index=7&list=PL_AKIMJc4roVSbTTfHReQTl1dc9ms0lWH https://www.youtube.com/watch?v=GJ2xMAqCZL8&list=PL_AKIMJc4roVSbTTfHReQTl1dc9ms0lWH&index=8

Zach

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On Sat, 26 Oct 2019 at 02:11, Zach Laine via Boost-users wrote:

About 14 months ago I posted the same thing. There was significant work that needed to be done to Boost.Text (the proposed library), and I was a bit burned out.

Now I've managed to make the necessary changes, and I feel the library is ready for review, if there is interest.

This library, in part, is something I want to standardize.

It started as a better string library for namespace "std2", with minimal Unicode support. Though "std2" will almost certainly never happen now, those string types are still in there, and the library has grown to also include all the Unicode features most users will ever need.

Github: https://github.com/tzlaine/text Online docs: https://tzlaine.github.io/text

I would start by removing the superlative statements about Unicode being "hard" or "crazy". It's not that complicated compared to the actual hard problems that software engineers solve everyday. The only thing is that people misunderstand what the scope of Unicode is, it's not just an encoding, it's a a database and a set of algorithms (relying on said database) to facilitate natural text processing of arbitrary scripts, and does compromises to integrate with existing industry practices prior to all those scripts being brought together under the same umbrella. Now the string/container/memory management, this is quite irrelevant. That sort of stuff has nothing to do with Unicode and I certainly do not want some Unicode library to mess with the way I am organizing how my data is stored in memory. Your rope etc. containers belong in a completely independent library. What's important is providing an efficient Unicode character database, and implementing the algorithms in a way that is generic, working for arbitrary ranges and being able to be lazily evaluated (i.e. range adaptors). I already did all that work more than 10 years ago as a two-month GSoC project, though there are some limitations since at that time ranges and ranges adaptors were still fairly new ideas for C++. It does however provide a generic framework to define arbitrary algorithms that can be evaluated either lazily or eagerly. To be honest I can't say I find your library to be much of an improvement, at least in terms of usability, since the programming interface seems more constrained (why don't things work with arbitrary ranges rather than this "text" containers) and verbose (just look at the code to do transcoding with iterators), the set of features is quite small, and that the database itself is not even accessible, and last I remember your implementation was ridiculously bloated in size. It also doesn't provide the ability to do fast substring search, which you'd typically do by searching for a substring at the character encoding level and then eliminating matches that do not fall on a satisfying boundary, instead suggesting to do the search at the grapheme level which is much slower, and the facility to test for boundary isn't provided anyway. I'm pretty sure I made similar comments in the past, but I don't feel like any of them has been addressed.

I have been meaning to have a closer look at that library for weeks now, but unfortunately I don't sem to find the time to do so. So I'm just writing down some thoughts: First of all, I hihgly support the inclusion of this library into boost (assuming there aren't any major design problems). C++ needs unicode support easy text processing - the former being a prerequisite for the latter. The only way we get those things into the standard is if the different approaches get battle tested in anger, in real-life projects. What the average programmer like me (who is not working on a text editor) imho needs is the ability to read text from one interface, do some basic input validation, normalize it, (potentially) split it up and or parse parts of it, store other parts through a different interface, retrieve all of that later and send it on through a different text based interface. During all that I don't want to have to worry too much about whethernon-asccii text got broken in betweeen. And in particular I should not have to worry about the particularities of how unicode works to o that. For that, I think it is much more important to have encoding aware interface types and easy to use transcoding functions than a full set of unicode text processing algorithms. Not that I'm not happy when they are there, but at least for the standard, lets please focus on what standards are for: Facilitate and enable components from different 3rd pary vendors to work seemlessly together! I'm also very happy to see that the library focuses on the 90% cases instead of trying to be flexible enough to be usable for every possible use case and under every possible circumstance. Best Mike