Boost.Filesystem uses std::codecvt facet to convert between narrow and wide encodings. On windows the default is ANSI encoding. By starting integration it installs utf8/utf-16 codecvt facet forcing boost::filesystem to use utf-8 in its narrow strings on Windows.

2 questions about integration in filesystem: - is there a way to undo what boost::nowide::nowide_filesystem() does? - can do the same with std::filesystem that is coming soon? Frédéric

...

- is there a way to undo what boost::nowide::nowide_filesystem() does?

Good question. You need to restore the locale that was imbued to path. I don't see an API to get one without replacement.

So probably I need to return the previous locale

which is returned already by boost::filesystem::path::imbue, good?

...

So probably I need to return the previous locale

which is returned already by boost::filesystem::path::imbue, good?

but then we get: auto old_loc = boost::nowide::nowide_filesystem(); // by the way, why repeat nowide? ... boost::filesystem::imbue(old_loc); What about the following symmetrical approach: auto old_loc = boost::filesystem::imbue(boost::nowide::locale()); ... boost::filesystem::imbue(old_loc); Frédéric

...

It looks like std filesystem uses global locale... (Bad idea. ) Does it? They finally succeeded in breaking it after so many tries? Oh dear.

apparently, with std::filesystem, you can create a path from UTF-8 string with u8path and from a path, you can get its UTF-8 representation with u8string. You do not need a locale for this. I looked into the current draft standard about ill-formed UTF-16. The draft standard says in 30.10.8.4.6 path native format observers [fs.path.native.obs]: "Remarks: Conversion, if any, is performed as specified by 30.10.8.2. The encoding of the string returned by u8string() is always UTF-8." Here u8string() must return a perfectly valid UTF-8 string (but its name implies so). And in 30.10.8.2.1 path argument format conversions [fs.path.fmt.cvt], I read: "Pathnames are converted as needed between the generic and native formats in an operating-system-dependent manner. Let G(n) and N(g) in a mathematical sense be the implementation’s functions that convert native- to-generic and generic-to-native formats respectively. If g=G(n) for some n, then G(N(g))=g; if n=N(g) for some g, then N(G(n))=n. [ Note: Neither G nor N need be invertible. — end note ]" It is funny how it is said: it does not say the roundtrip must work, it says the roundtrip must work on any converted paths. This is the case for boost.nowide as once converted to UTF-8, the round trip narrow->wide->narrow works. I also read in 30.10.8.2.2 path type and encoding conversions [fs.path.type.cvt] "If the encoding being converted to has no representation for source characters, the resulting converted characters, if any, are unspecified. Implementations should not modify member function arguments if already of type path::value_type." This means that it is implementation defined how to convert an ill-formed path. Therefore, using the replacement character U+FFFD would match the std::filesystem policy. Note also that std::filesystem just uses narrow string as is on Posix as boost.nowide does. char: The encoding is the native narrow encoding (30.10.4.9). The method of conversion, if any, is "For POSIX-based operating systems path::value_type is char so no conversion from char value type arguments or to char value type return values is performed." Frédéric Frédéric

...

These aren't try functions as they succeed in general (unless you are out of memory after conversion policy update) and returing char */wchar_t* makes code shorter.

They will fail if the user-provided buffer isn't big enough.

I see. If some I'd rather expect to extend the boost.locale functionality rather boost.nowide - it seems to be more appropriate (since nowide functions finally call utf_to_utf functions from boost.locale library) Regarding iterator input/output interface. I must say that iterator based interface isn't right for encoding conversions. There was a long discussion regarding it during Boost.Locale review. The major reason is that you don't work on characters by themselves but rather character sequences and intermediate states. Since all conversion API gets some kind of "char *" and virtually every string class around has its "c_str()" variant I don't really see it necessary to provide arbitrary iterators (especially since under the hood I'll need to combine them into chunks of char * for processing) And if you want to process a stream you will be much better going with std::codecvt facet that works on chunks and stream.

...

What it does it uses native console Wide API to write/read to/from console:

http://cppcms.com/files/nowide/html/index.html#technical_cio

In case of redirection it just becomes alias of std::cin/cout (docs need to be updated a little)

OK. How about mixing use of the boost::nowide:: and the std:: versions? If that possible? Because that's something that I guess will be hard to avoid.

Shouldn't be a problem. Finally all goes to console. Artyom

On Fri, Jun 23, 2017 at 5:35 PM, Groke, Paul via Boost wrote:

I think it's convenient to have the function that does the conversion in the Nowide library itself, even if it just calls come other function. That way the user doesn't have to know or care what kind of exact conversion rules Nowide uses - they can just call nowide::widen/narrow.

I see

...

Regarding iterator input/output interface.

[snip]

I'm not suggesting an iterator-based interface. I'm just suggesting two changes. 1: change

stack_wstring wstr; wstr.convert(input);

to

stack_wstring wstr; widen(wstr, input);

I.e. move the conversion function out of the string class. And then 2: allow arbitrary (string) types instead of stack_wstring. Not via iterators, the widen/narrow functions could (should) simply use push_back(char) or append(char const*, char const*) to "build" the output string.

Now I understand what do you mean. I assume it can be done/

...

Shouldn't be a problem. Finally all goes to console.

Won't buffering cause trouble - especially for input?

It should be similar to behavior of std::cin as it uses tie call so apparently on the output side shoudn't be problem. (Also good point to test) Regarding input it of course will be much trickier. More testing would likely be needed and I see more potential problems. I'll need to check it as well and if limitation exist report them Artyom