On 10/7/15 9:14 AM, Artyom Beilis wrote:

...

________________________________ From: Robert Ramey

...

Independently of Boost.Nowide I would like to propose replacement of boost/detail/utf8_codecvt_facet by one thatactually takes in account proper Unicode handling.

This is a worthy undertaking. Consider putting it somewhere else other than "detail" directory/namespace. This should have been an officially supported boost component years and years ago. Also I'd like to see it look more like a boost library - tests, documentation ,etc.

...

First of all I can add it as header only component to Boost.Locale with its unit test as it would be a proper place - without any review process (also code is partially based on boost-locale's one)

Do this ! Note the test in the serialization library for the current facet. I don't know if it's useful - it has been in the past. I presume you can either move that test in or supply one that's at least as good.

And than other libraries should replace the broken one by a new one and remove boost/detail/utf8_codevt* all together.

You'll have to give us some warning - maybe skip a release or maybe just use header re-direction.

Not every library provides this (libstdc++ isn't for example)

Hmmm - I'll be damned. I've been including it conditionally on a macro from Boost config. Now that I think about it, since the test matrix uses older compilers it must still be used.

It is also does some other stuff that does not really fit the context.

It was written a long time ago when library support was all over the place. It's served us pretty well. I'm sure you can do much better now and I'll be damned disappointed if you don't!

Major use of utf8_codecvt facet is to "connect" narrow and wide APIs. C++11's <codecvt> is just too long for weight especially when C++03 would be with us for a loooooooong period.

It will be good to have an option.

...

Robert Ramey

Artyom Beilis

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

On 10/8/15 2:52 AM, Artyom Beilis wrote:

There is also small but significant difference, my UTF-8 <-> UTF-16/32 facets are template based and header only implementation.

The current one is also header only and template based

Regarding the test - I have my own that the old facet obviously wouldn't pass.

That is not my question. Will your new version pass the current test?

...

You'll have to give us some warning - maybe skip a release or maybe just use header re-direction.

It can't be simple redirection as it works in little bit different way.

Hmmm - very suspicious. I expect to just be able to include a header and create an instance of the codecvt type and start using it. If I can't do this, we might have a problem.

I think just keeping old utf8 facet until all library maintainers replace it - and it would likely take more than one release :-)

Artyom Beilis

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

I meant:

https://github.com/boostorg/serialization/blob/develop/test/test_utf8_codecv...

I adopted it and It passes - no problem there.

creating of code convert facets including documentation about what they are and how to use them.  Examples of usage would be the new utf8codecvt facet we're talking about.

How can I say this gently. codecvt facet is used there because it is "Standard" way of doing things and it is far from being flawless [1] but it exists and it is the ultimate way to convert between encodings in C++. std::locale is complex stuff with many issues by design including its codecvt facet - they are both hard to create and use. Most of users don't really need them - ideally you just run std::locale::global(std::locale("")) And everything just works for any stuff that needs to handle encoding. But in reality it does not. So you need to put workarounds and create stuff like utf8 facet because some standard libraries on some very well known operating system do not support UTF-8 locales. And consider it "confusing" that std::string is accidentally becomes utf-8 encoded string. All the char const *str = u8"привет-שלום" stuff to encode UTF-8 string was born in sin and probably die this way because of some specific vendors that ignore what the world had learned well. So ideally end users should not care about codecvt - that is why originally in nowide there is just a function called boost::nowide::nowide_filesystem() And magic happens. The problem to understand how the magic works you need to learn a lots of things and simple tutorial isn't enough - even entire library like Boost.Nowide isn't always enough. Regards, Artyom Beilis [1] One of the sings it does not allow to implement stateful encodings (such that can compose and decompose some characters the way iconv does)

From: Vladimir Prus On 07-Oct-15 8:37 PM, Robert Ramey wrote:

...

Do this !

Note the test in the serialization library for the current facet. I don't know if it's useful - it has been in the past. I presume you can either move that test in or supply one that's at least as good.

...

And than other libraries should replace the broken one by a new one and remove boost/detail/utf8_codevt* all together.

You'll have to give us some warning - maybe skip a release or maybe just use header re-direction.

I would prefer for any such replacement to be still part of boost/detail. Depending on boost.locale just to get one header seems not perfect.

- Volodya

The current utf8_codecvt can't be used as is and designed to be "copied" to specific libraries (with its own namespace) My codecvt in generic header only one that can and should be used outside boost specific libraries, i.e. if I want to read/write utf8 file using std::wfstream I can install this facet independently of rest of the boost libraries. Details is just like boost-private namespace. So isn't best place, on the other hand Unicode facilities is good place, especially that it is a component independent of entire locale library i.e. header only part similar to We can create a "Separate" codecvt library with its own formal review and it would be ready in best case in a year - or just use it in a place unicode processing belongs to. Artyom

----- Original Message -----

From: Peter Dimov Artyom Beilis wrote:

...

We can create a "Separate" codecvt library with its own formal review and it would be ready in best case in a year...

One option is to put it into utility; another is to use a mini-review if the new codecvt library is an implementation of the standard <codecvt> interface.

std::codecvt_utf8 is not quite the same as boost::utf8_codecvt_facet, but on the other hand, from your previous message it seems that your utf8_codecvt_facet is not std::codecvt_utf8 but std::codecvt_utf8_utf16, or perhaps it's the latter when wchar_t is 16 bit and the former when it's 32 bit.

[BEGIN: Long description regarding <codecvt> ] To be honest I don't know what guys who designed <codecvt> in first place thought of - I feel string influence of broken MS Unicode policies std::codecvt_utf8 is actually quite misleading - it converts between utf8 and ucs-2/ucs-4 i.e. using it under windows with wchar_t you wouldn't get support of utf-16 at all. It basically does what boost::XXX:utf8_codecvt_facet does for std::codecvt_utf8. Basically broken and useless as UCS-2 is subset of proper encoding. Now <codecvt>'s std::codecvt_mode is clear Microsoftism as for example using UTF-8 BOM is one of the many Unicode crimes Microsoft created - as storing UTF-16 files on disk. Another hilarious stuff is Maxcode = 0x10ffff template parameter for the facet... It is like creating template<double Pi_Value=3.14159> class circle; 0x10FFFF IS max value for Unicode codepoint, not 0xFFFF not anything else. std::codecvt_utf16 is an attempt to build "narrow" utf-16 encoding, just no comment... std::codecvt_utf8_utf16 is actually useful under windows and does what it is supposed to to with wchar_t... but under POSIX platform it is impossible to use std::codecvt_utf8_utf16 with wchar_t because wchar_t is UTF-32... So if you want to install utf8 to wchar_t codecvt facet that represents utf-16 or utf-32 according to platform you need to use if(sizeof(wchar_t) == 2) return new std::codecvt_utf8_utf16(); else // sizeof(wchar_t) == 4 return new std::codecvt_utf8(); So all <codecvt> was built wrong under strong Microsoft development policy influences and useless for any cross platform development. So... Boost community - please give yourself a favor Don't use <codecvt> unless you really understand what are you doing. [END: Long description regarding <codecvt> ] If you want to covert utf8 files properly to native wide character like for example for boost::filesystem, boost::serialization or std::fstream you need to use facet that converts to utf-16 or utf-32 according to what wchar_t holds and <codecvt> does not provide one (without platform specific tricks) So I'm not going to implement C++11 <codecvt> because IMHO it is broken by design in first place. Boost.Locale provides one but currently it is deep internal and complex part of library. The code I written for Boost.Nowide or one I suggest to put into Boost.Locale header-only part is codecvt that converts between utf8 and utf-16/32 according to size of character: boost::(nowide|or locale)::utf8_facet - utf-8 to utf-16 (windows) utf-32 (posix) boost::(nowide|or locale)::utf8_facet - utf-8 to utf-16 on any platform boost::(nowide|or locale)::utf8_facet - utf-8 to utf-32 on any platform That's it. It isn't <codecvt> because C++11 <codecvt> does not actually do the job needed. Artyom Beilis

Hmm - is there a reason that a better name wouldn't be Boost.Codecvt ?

On 10/8/15 12:14 PM, Peter Dimov wrote:

One could make the case for libs/utf8 which would contain utf8_facet and the "obvious"

bool is_valid_utf8( string const & s ); wstring utf8_decode( string const & s ); string utf8_encode( wstring const & s );

but this is already well into full review/bikeshed territory.

I would like to see libs/codecvt which included the above functions Robert Ramey

On 10/8/15 1:17 PM, Artyom Beilis wrote:

See, all this is already implemented in header only way in Boost.Locale - so no linking required.

good to hear.

https://github.com/boostorg/locale/blob/master/include/boost/locale/utf.hpp https://github.com/boostorg/locale/blob/master/include/boost/locale/encoding...

So just call boost::locale::conv::utf_to_utf("Hello World");

Full codecvt_facet for many encodings - inluding UTF-8, ISO-8859-*, Windows-125* are already there as well

excellent

However there is very useful specific codecvt - that converts between utf8 and wchar_t/char16_t/char32_t that can be implemented in header only without linking with big and complex Boost.Locale library.

also good.

Also I'm going to make it little bit more generic so you can implement wchar_t/char16_t/char32_t to any stateless encoding easily (I want to improve some stuff withing Boost.Locale as well)

So utf8 codecvt facet is INTEGRAL part of Boost.Locale already - it exists there.

so what are we talking about?

Just I think I'll make it more accessible to general libraries without requirement of linking and easiler to use by users without need of special locale generation.

LOL - So what I'm hearing here that there is already the codecvt facet created as part of Boost.Locale and all we need is to enhance the documenation in order to make it more obvious to those of us that need it. I've been hoping all along that we've have something like this. Some time ago I have Boost.Locale a cursory look in search of such a thing. It's quite possible it's always been there and I didn't see it, that's it's "implicit" and obvious to someone who has read and understood he rest of the documentation, or that the library has evolved since I did this or all of the above. In any case I'm very interested in this.

Ok... I decided what I'm going to do.

OK - do this and let us know, I'll look it over again and give you some feedback from a novice.

Next step is for other libraries to adopt this utf8_codecvt facet.

LOL - you do your part - and the rest will happen. Robert Ramey

Artyom Beilis -------------- CppCMS - C++ Web Framework: http://cppcms.com/ CppDB - C++ SQL Connectivity: http://cppcms.com/sql/cppdb/

_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

To be honest I don't know what guys who designed <codecvt> in first place

It was done in the early and mid 1990's, with primary input coming from Asian national bodies and the now long gone Unix vendors who had a big presence in that market. thought of - I feel string influence of broken MS Unicode policies

This was years before Microsoft folks started to participate in the LWG.

So I'm not going to implement C++11 <codecvt> because IMHO it is broken by design in first place.

Header <codecvt> isn't what we need, as you point out below.

Boost.Locale provides one but currently it is deep internal and complex part of library.

The code I written for Boost.Nowide or one I suggest to put into Boost.Locale header-only part is codecvt that converts between utf8 and utf-16/32 according to size of character:

boost::(nowide|or locale)::utf8_facet - utf-8 to utf-16 (windows) utf-32 (posix)

Don't forget utf-8 to utf-8 (some embedded systems). IMO, a critical aspect of all of those, including utf-8 to utf-8, is that they detect all utf-8 errors since ill-formed utf-8 is used as an attack vector. See Markus Kuhn's https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt I can contribute a Boost regression test friendly version of Kuhn's malformed tests. --Beman

On 09.10.2015 17:41, Peter Dimov wrote:

Beman Dawes wrote:

...

IMO, a critical aspect of all of those, including utf-8 to utf-8, is that they detect all utf-8 errors since ill-formed utf-8 is used as an attack vector.

That is what I alluded to earlier with my bikeshedding comment - I personally find this policy a bit too firm for my taste. Sure, sometimes I do want to reject any invalid UTF-8 with extreme prejudice, but at other times I do not. For instance, when I get a Windows file name, it can well be invalid UTF-16, which when converted will become invalid UTF-8 but which will roundtrip correctly back to its original invalid UTF-16 form and refer to the same file. That's why things like CESU-8 or WTF-8 exist.

So I like the "method" argument of locale::conv::utf_to_utf, except that I think that it doesn't offer enough control.

I think, UTF-8 is UTF-8 (i.e. the character encoding that is described by the standard), and the tool for working with it should adhere to the specification. This includes signalling about invalid code sequences instead of producing them. WTF-8 and CESU-8 are not UTF-8 but different encodings. Dealing with them should be the user's explicit choice (e.g. the user should write utf16_to_wtf8 instead of utf16_to_utf8).

On 09.10.2015 18:20, Peter Dimov wrote:

Andrey Semashev wrote:

...

WTF-8 and CESU-8 are not UTF-8 but different encodings. Dealing with them should be the user's explicit choice (e.g. the user should write utf16_to_wtf8 instead of utf16_to_utf8).

The user doesn't write such things in practice. He writes things like

string fn = get_file_name(); fopen( fn.c_str() );

and get_file_name and fopen must decide how to encode/decode UTF-8. So get_file_name gets some wchar_t[] sequence from Windows, which happens to be invalid UTF-16. But Windows doesn't care for UTF-16 validity and if you pass this same sequence to it, it will be able to open the file. So your choice is whether you make this work, or make this fail. I choose to make it work.

What I'm saying is that get_file_name implementation should not even spell UTF-8 anywhere, as the encoding it has to deal with is not UTF-8. Whatever the original encoding of the file name is (broken UTF-16, obtained from WinAPI, true UTF-8 obtained from network or a file), the target encoding has to match what fopen expects. As I remember, on Windows it is usually not UTF-8 anyway but something like CP1251. But even if you have a UTF-8 (in Windows terms) locale, Windows code conversion algorithm AFAIU actually implements something different so that it can handle invalif UTF-16. Your code should spell that 'something different' and not UTF-8. If it spells UTF-8 then it should fail on invalid code sequences.

The functions would of course never produce invalid UTF-8 when passed a valid input (and will deterministically produce the least-invalid UTF-8 for a given input)

There should be no such thing as 'least invalid' or 'almost valid' data. It's either valid or not. The tool should not produce invalid data, period. If you want to successfully convert invalid UTF-16 input to a multibyte encoding then choose that encoding and don't pretend it's UTF-8. Because UTF-8 cannot represent that input data.

but here again the definition of valid may change with time if, f.ex. more code points are added to Unicode beyond the current limit.

Unicode versioning is another issue. If it comes to this, we will decide what to do. We may well decide to go with utf8v2 in the naming, if the need for strict v1 conformance is strong enough in some cases.

You should also keep in mind that Unicode strings can have multiple representations even if using strict UTF-8. So one could argue that using strict UTF-8 provides a false sense of security.

There are normalization and string collation algorithms to deal with this. What's important is that the input to these and other algorithms is valid. Otherwise all bets are off.

On 09.10.2015 19:27, Peter Dimov wrote:

Andrey Semashev wrote:

...

...

string fn = get_file_name(); fopen( fn.c_str() );

What I'm saying is that get_file_name implementation should not even spell UTF-8 anywhere, as the encoding it has to deal with is not UTF-8. Whatever the original encoding of the file name is (broken UTF-16, obtained from WinAPI, true UTF-8 obtained from network or a file), the target encoding has to match what fopen expects.

'fopen' here is a function that decodes 'fn' and calls _wfopen, or CreateFileW, or whatever is appropriate.

get_file_name and fopen work in tandem to make it so that the file selected by the first function is opened by the latter. And to do that, they may need to put invalid UTF-8 in 'fn'.

Right. Just don't call it UTF-8 anymore.

...

There should be no such thing as 'least invalid' or 'almost valid' data.

There exists a legitimate notion of more valid or less valid UTF-8 because it can be invalid in different ways, some more basic than others.

Could you point me to a definition of these degrees of validity? In my understanding the string is valid if it can be decoded by a conforming parser. E.g. it should not contain invalid code points (i.e. those not allowed by the standard) or sequences thereof.

...

There are normalization and string collation algorithms to deal with this. What's important is that the input to these and other algorithms is valid.

This depends on the notion of valid. UTF-8 that encodes codepoints in more bytes than necessary corresponds to a valid codepoint sequence.

AFAIU, no, it is not a valid encoding. At least, not according to this: https://en.wikipedia.org/wiki/UTF-8#Overlong_encodings

Strict handling rejects it not because it's invalid Unicode, but because it's not the canonical representation of the codepoint sequence. But the codepoint sequence itself can be non-canonical, and hence code that assumes that "validated" UTF-8 is canonical is wrong.

Well, my Unicode kung fu is not very strong, but if the standard only allows minimal encoding then anything that doesn't follow is not conforming and should be rejected. For convenience we could provide a separate tool that would tolerate some deviations from the spec and produce valid UTF-8. But the user should have control over what exact deviations are allowed. Don't call the tool utf_to_utf though, as the name doesn't make sense to me.

The policy of strict UTF-8 is not a bad idea in general, but it's merely a first line of defense as far as security is concerned. Properly written code should not need it.

You mean all string-related code should be prepared for invalid input? Seems like too much overhead to me.

On 09.10.2015 22:15, Peter Dimov wrote:

Andrey Semashev wrote:

...

On 09.10.2015 19:27, Peter Dimov wrote:

...

...

...

string fn = get_file_name(); fopen( fn.c_str() );

... get_file_name and fopen work in tandem to make it so that the file selected by the first function is opened by the latter. And to do that, they may need to put invalid UTF-8 in 'fn'.

Right. Just don't call it UTF-8 anymore.

I don't know what this means.

I mean as a result you will have a string fn, whose encoding is not UTF-8. As a consequence algorithms that require UTF-8 input cannot be expected to work with this string.

...

...

There exists a legitimate notion of more valid or less valid UTF-8 > because it can be invalid in different ways, some more basic than > others.

Could you point me to a definition of these degrees of validity?

First, you can have invalid multibyte sequences in the input. Second, you can have overlong byte sequences. Third, the encoded codepoint sequence may be invalid, in various ways.

Ok, all these count as just invalid to me.

...

...

This depends on the notion of valid. UTF-8 that encodes codepoints in > more bytes than necessary corresponds to a valid codepoint sequence.

AFAIU, no, it is not a valid encoding.

It's an invalid UTF-8 encoding of a valid codepoint sequence.

Yes, but valid codepoint sequence is not enough to interpret the string.

...

You mean all string-related code should be prepared for invalid input?

I don't understand this, either.

You said that properly written code should not require string validity. Should such code be always prepared for invalid strings, at any point? If so, this looks like unnecessary overhead to me.

----- Original Message -----

From: Peter Dimov Andrey Semashev wrote:

...

WTF-8 and CESU-8 are not UTF-8 but different encodings. Dealing with them should be the user's explicit choice (e.g. the user should write utf16_to_wtf8 instead of utf16_to_utf8).

The user doesn't write such things in practice. He writes things like

string fn = get_file_name(); fopen( fn.c_str() );

and get_file_name and fopen must decide how to encode/decode UTF-8. So get_file_name gets some wchar_t[] sequence from Windows, which happens to be invalid UTF-16. But Windows doesn't care for UTF-16 validity and if you pass

Ok... that is interesting point relevant to Boost.Nowide however irrelevant to utf8_codecvt facets. The only way UTF-16 can be invalid is to have non-properly paired utf-16 surrogate units. They can technically be encoded to invalid UTF-8 prepresenting code points in closed range reserved to surrogate pairs. i.e. boost::nowide::narrow should generate invalid UTF-8 from invalid UTF-8 and invalid in very special way UTF-8 to invalid UTF-16. It looks horrifying for me but it maybe actually solution for such a problem But this should never-ever-ever be used outside Boost.Nowide And to be honest - IMHO if a program fails on files that encoded in invalid UTF-16 when Windows states that the encoding is UTF-16... than I think they should fail.

You should also keep in mind that Unicode strings can have multiple

representations even if using strict UTF-8. So one could argue that using strict UTF-8 provides a false sense of security.

This isn't correct - you are missing normalization forms and codepoint representation. Yes properly localized software should generally use normalized strings. However a sequence of valid codepoints has one and only one representation in both UTF-8 and UTF-16. There is no such things as strict UTF-8 - there is either UTF-8 or not. Interesting note: on Mac OS X there is a requirement that strings should be NFC normalized UTF-8 strings. Artyom

Artyom Beilis wrote:

Interesting note: on Mac OS X there is a requirement that strings should be NFC normalized UTF-8 strings.

I think that it uses form D, not form C, but I may be misremembering. I'm also not sure if it enforces this requirement on input, that is, whether it doesn't normalize as needed.

On 09.10.2015 18:41, Peter Dimov wrote:

Andrey Semashev wrote:

...

WTF-8 and CESU-8 are not UTF-8 but different encodings. Dealing with them should be the user's explicit choice (e.g. the user should write utf16_to_wtf8 instead of utf16_to_utf8).

In addition to what I wrote earlier, the choices here are not representable in a single U or W letter. When taking UTF-8, you need to decide whether to

- accept codepoints over 10FFFF - accept codepoints encoded with more bytes than necessary - accept surrogates - probably more because Unicode is hard

and then for each rejected byte sequence whether to

- throw - ignore and skip - replace with U+FFFD

As long as the code sequences are described by the spec, I consider them valid. We can provide a number of options to influence the conversion process, but the result should be something that can be decoded by a conforming Unicode parser.

Artyom Beilis wrote:

Codepoints above 10FFFF is like lets assume Pi=3.15..

No, sorry. This is not at all the same. The reason we're in this mess is precisely because codepoints above 0xFFFF were like pi=3.15. And then it turned out they weren't.

...

- probably more because Unicode is hard

Unicode isn't hard - it is just treated with ignorance by even big organization not talking about average programmers.

What I meant by that is for instance - is 0xCC 0x81 a valid UTF-8 string? - is 0x65 0xCC 0x81 0xCC 0x81 a valid UTF-8 string?

Artyom Beilis wrote:

...

What I meant by that is for instance

- is 0xCC 0x81 a valid UTF-8 string? - is 0x65 0xCC 0x81 0xCC 0x81 a valid UTF-8 string?

Both are valid strings.. and both are meaningless on their own i.e. accent without letter or two same accents.

Being illogical in human terms or representation does not make them UTF-8 illegal.

UTF-8 is simple, human language processing is complex.

My point here is that strictly valid UTF-8 is the valid multibyte encoding of a valid codepoint sequence, and that the definition of "valid codepoint sequence" may vary depending on context, such that the above sequences are considered invalid. Drawing a line at the place where codepoints over 10FFFF and single surrogates are invalid but the above sequences are valid is an arbitrary decision. Not that this decision is wrong, it isn't. But it may not be what the user needs. Saying "invalid UTF-8 is just invalid, period" doesn't always work very well, although it's a good default. There are cases in which you have to handle specific kinds of invalid UTF-8 (but not any invalid UTF-8) and having to write UTF-8 encoding/decoding functions for every such instance does not really contribute to either security or correctness. It's better - I posit - to have functions that can be configured to handle various invalid forms of UTF-8 (that is, to accept certain invalid UTF-8, not necessarily to produce it, of course).

Beman Dawes wrote:

Don't forget utf-8 to utf-8 (some embedded systems).

What does it do?

On Thu, Oct 8, 2015 at 10:03 AM, Peter Dimov wrote:

Artyom Beilis wrote:

We can create a "Separate" codecvt library with its own formal review and

...

it would be ready in best case in a year...

One option is to put it into utility;

+1

another is to use a mini-review if the new codecvt library is an implementation of the standard <codecvt> interface.

+1 It seems pretty useless to me if it isn't an implementation of the standard <codecvt> interface.

std::codecvt_utf8 is not quite the same as boost::utf8_codecvt_facet, but on the other hand, from your previous message it seems that your utf8_codecvt_facet is not std::codecvt_utf8 but std::codecvt_utf8_utf16, or perhaps it's the latter when wchar_t is 16 bit and the former when it's 32 bit.

I'm assuming Artyom is proposing implementing codecvt where externT is always UTF-8 encoding, and internT encoding is UTF-32 for 32-bit wchar_t (e.g. many POSIX-like systems), UTF-16 for 16-bit wchar_t (e.g. many Windows-like system), and UTF-8 for 8-bit wchar_t (e.g. some embedded systems). --Beman

On Thu, Oct 8, 2015 at 10:36 AM, Ion Gaztañaga wrote:

On 07/10/2015 16:49, Artyom Beilis wrote:

...

Some updated regarding boost.nowide 1. Library moved to github and its format is converted to modular boost layout: https://github.com/artyom-beilis/nowide2. Fixed unsupported std::ios::ate flag by boost::nowide::fstream3. Added some C++11 interfaces to boost::nowide::fstream4. Added integration functionality with boost::filesystem:

https://github.com/artyom-beilis/nowide/blob/master/include/boost/nowide/int...

Please don't make nowide dependent on Boost filesystem. It adds unneeded dependencies and disallows other "path" types (like std::experimental::path). Couldn't you template it on a "path" class? This will make the utility more general and avoids any unneeded dependency.

Right. As suggest by Peter Dimov, Utility would be a good place to put it. --Beman