Assuming you have the number 12345678909876543210, which is striding two buffers, with 1234567890 then 9876543210, the parser will call

uint64 value = 0;

value = value * pow10(size("1234567890")) + 1234567890; on_number_part("1234567890", ec)

value = value * pow10(size("9876543210")) + 9876543210; on_uint64(value, "9876543210", ec)

(the actual implementation works on character per character basis and is slower than this)

This will not really work once the value gets one digit larger, as the first argument will be garbage.

But if the value gets larger, the parser will not call on_uint64, it will call on_double.

The text to binary conversion shouldn't even be in the JSON parser,

OK, if you don't need the text to binary conversion, what's stopping you from ignoring it, and only using the text part? You do get all the info, it's just that the final part doesn't arrive via "on_number", but via the other three callbacks, to save one call.

...

What are the limitations, and what is `n` in your suggested replacement?

Same as for on_string_part/on_string, on_key_part/on_key, on_comment_part/on_comment etc.

I'm arguing it should be there for consistency, it's not strictly necessary.

Yeah, fair point.

Mathias Gaunard wrote:

...

OK, if you don't need the text to binary conversion, what's stopping you from ignoring it, and only using the text part? You do get all the info, it's just that the final part doesn't arrive via "on_number", but via the other three callbacks, to save one call.

Separation of concerns, wasted cycles, etc. Parsing numbers is the slowest part of all this parsing business, and the part that's most dependent on how you want to represent numbers into memory.

For example I typically represent numbers as decimals myself.

Right. So you acknowledge that the parser as-is would work for your use case, but you refuse to use it, and prefer Boost not to have it, because separation of concerns and wasted cycles. You intuitively feel that it would be too slow because it converts the numbers into int64/double, but haven't measured it against a parser that doesn't. (If you do, you may be surprised.) FWIW, there's a reason concerns aren't separated here. It's that the number can be arbitrarily long, so if you're going to separate the concerns, you'd need to buffer the entire number in string from before being able to convert it to int64/uint64/double. But since these have limited precision, if you do the conversion on the fly, you no longer need to buffer the entire number in string form. If the user wants the entire number, he can buffer it himself. If not, that part is unnecessary and isn't done. Since the majority of users are more interested in int64/uint64/double than in infinite precision, the parser chooses to favor this case at the expense of the other, and not vice versa.

On Mon, 21 Sep 2020 at 15:23, Vinnie Falco wrote:

"JSON-inspired binary formats" are not JSON. CBOR for example is "JSON-inspired", yet neither Node.js nor any web browser support it. They do however support standard JSON, and their respective DOMs have numerical limitations similar to the ones in Boost.JSON.

The main reason I suggested this is that you might also want to be able to parse your in-memory representation as some kind of visitation mechanism which you could push directly to the serializer. You cannot since conversion of numbers to binary occurs, so there is no string left. So I'd design the parser callbacks so that numbers can be either stored as text or binary in the source.

I do not think it is possible for a single parser API and single serializer API to address both of these use-cases well. Optimizing for one will necessarily come at the expense of the other. Perhaps there is a hypothetical not-yet-written library just waiting to be discovered which will prove me wrong, by doing everything well. I have my doubts.

Doing a pull parser that supports data being pushed incrementally is difficult as it requires the ability to suspend arbitrary code that is trying to pull data beyond what is available. That would be doable, maybe with a coroutine design, but adds a lot of complexity. To make people happy I think it's easier to provide a separate incremental push parser and a non-incremental pull parser. I don't think those should be in separate libraries, consistency is key.

...

The DOM does not support numbers that are neither 64-bit integral numbers nor double-precision floating-point. This is a common limitation, but still quite disappointing, since the proposal of it being a canonical type is quite tainted by the fact it can't represent JSON data losslessly. I think it would be better if another type was provided for numbers that don't fit in the previous two categories.

I disagree. Serialization of a canonical type should produce JSON that is interoperable with the majority of receivers. Neither Node.js nor browser implementations of JSON understand arbitrary precision integers out of the box. Unless the receiver is configured to process it, emitting an arbitrary precision number is certain to cause a compatibility problem.

You are writing the receiver, it is the JSON parser. Making it correctly deal with numbers is important so that you don't corrupt the work of whatever you are consuming. If you know you are writing data for a consumer with certain limitations, then up to you to transform your numbers into something else. Note that your code will still happily write 64-bit integers as numbers, even though JavaScript can't handle more than 53 bits, and will round those numbers and cause all sorts of problems. Another thing is that I don't buy the argument that web browsers, Node.JS and all the JavaScript ecosystem are the only thing that is relevant. The web is a domain that has a reputation of not being quite as rigorous as systems programming which is the main application domain as C++. It is ok if script kiddies want to do whatever with their webby stuff, but let's please do things properly when we're in C++-land.

...

Querying capabilities for that type are pretty limited and quite un-C++ like. Examples use a C-style switch/extractor paradigm instead of a much safer variant-like visitor approach.

visit() is a planned feature which is already implemented:

https://github.com/vinniefalco/json/tree/visit

Good.

JSON Path and JSON Pointer are planned features:

https://goessner.net/articles/JsonPath/

https://tools.ietf.org/html/rfc6901

Nice.

That is by design. Boost.JSON is not a serialization library. It addresses the use case for a JSON DOM with accompanying parser and serializer. This is not an insignificant use-case; witness the popularity of RapidJSON and nlohmann's JSON for Modern C++. In fact didn't you say you are already using RapidJSON?

I use RapidJSON because it's the established industry standard to do precisely what this library does. As I said I don't believe the value proposition of this library is sufficient to replace existing things when all it does is the same thing but with a Boost logo on it. There is no guarantee at this stage this library will get any traction or maintenance on the same level as one which is used by thousands of projects, so it's not worth the risk to me.

On Mon, 21 Sep 2020 at 16:48, Vinnie Falco wrote:

On Mon, Sep 21, 2020 at 8:26 AM Mathias Gaunard wrote:

...

To make people happy I think it's easier to provide a separate incremental push parser and a non-incremental pull parser. I don't think those should be in separate libraries, consistency is key.

I disagree. I don't have any special expertise to bring to the table when it comes to pull parsers. I don't need them for the applications and libraries that I am developing. However I am proficient at writing the kind of code needed for push parsing, and implementing containers (having already done so many times in the past). If I were to implement a JSON serialization library (which is what you are asking for) I would be doing so basically from a fresh start. What I am saying is that this type of implementation does not play to my strength. There have already been individuals who proclaim to be subject matter experts in this area - they should write this library.

"Consistency" is a vague goal. But here's what's not vague. Having JSON serialization in its own library means it gets its own Travis limits for individual forks. CI turnaround is faster. Tests run faster. The documentation is focused only on the serialization aspects. Users who need only JSON serialization, can use the particular library without also bringing in a JSON DOM library. And vice versa. To me, this separation of concerns has more value than "consistency."

I also have to wonder, to what consistency do you refer? The parser will not be consistent. One is pull, the other is push. The DOM will not be consistent. One has it, the other doesn't. Serialization is in a similar situation. In fact, when you compare a JSON DOM library to a JSON Serialization library, they are about as different as you can imagine, sharing only the nominal serialization format. Not much code would be shared between a DOM and a serialization library.

Maybe pull parser is not the right terminology, the parser in question would not have any DOM. It would be something like. template<class T> struct deserialize_impl; template T deserialize(Deserializer& s) { return deserialize_impl<T>::impl(s); } struct Bar { int value; }; struct Baz { }; struct Foo { Bar bar; string baz; }; template<> struct deserialize_impl<Bar> { template<class Deserializer> static Bar impl(Deserializer& s) { s.object_begin(); s.key("value"); int value = std::stoi(s.number()); s.object_end(); return Bar{std::move(value)}; } }; template<> struct deserialize_impl<Foo> { template<class Deserializer> static Foo impl(Deserializer& s) { s.object_begin(); s.key("bar"); Bar bar = deserialize<Bar>(s); s.key("baz"); string baz = s.string(); s.object_end(); return Foo{std::move(bar), std::move(baz)}; } }; (possibly with shorthand syntax to simplify redundant patterns, such as automatic object_begin/object_end and deserialize function variant with automatic key, which would allow constructing directly) You can see the name and sequence of the events is still perfectly in-line with the push parser. Serialize could also be written in a symmetric fashion.

I see nothing wrong with Boost.JSON being the library that users go to when they want a container that is suitable as an interchange type. And I see nothing wrong with someone proposing a new library that implements JSON serialization to and from user-defined types, with pull parsers, no DOM, and maybe even quotes from Aleister Crowley in the documentation. Having fine-grained libraries which target narrow user needs is something we should all agree is good for Boost (and C++ in general).

The problem with this is that you end up having to pick a million different things to solve very small problems all in slightly inconsistent ways. There is a cost with adopting any new library or API, both technical and mental. In 98% of cases, when I write a websocket interface (with boost beast, kudos to you), I define my messages as C++ types, and I parse incoming JSON into those types. I could parse the incoming JSON into json::value, and either use it directly, but then I lose static naming and typing, or I could convert it into my types, which is an unnecessary step which can even introduce loss of data. In the 2% remaining percent, I write arbitrary data to some log or record and I need to be able to visualize it or export it to a database. In that case, I don't know the schema of the information ahead of time, and json::value is therefore what I want to use. In both cases, I'd like to read/write my data from/to JSON with the same framework.

On Mon, 21 Sep 2020 at 17:49, Vinnie Falco wrote:

This looks very much like it depends on the order of appearance of keys in the JSON, which is non-standard.

You'd parse the object until you reach the desired key, building an index of where the various keys you meet on the way are (with possibly extra information of where the associated value starts/end etc.), then for the next one either it's in the index so you can use it then discard it, or it's ahead, so you can continue with the same algorithm. That would be optimized for keys being ordered as expected, ignoring extra keys, while still supporting out-of-order keys (albeit with some extra storage requirements that grows with your nesting level).

...

In both cases, I'd like to read/write my data from/to JSON with the same framework.

Why? What specifically, is the requirement here?

What I'd like is a way to describe how my C++ types map to a key-value structure with normalized types so that I can easily convert my objects back and forth through a structured self-describing and human-readable interchange format. Ideally I'd like to do so without repeating myself with all kinds of converters, serializers, etc. and reasonably efficiently, which is why I'd like these things to be reasonably unified. Boost.Serialization doesn't do this, since it takes the approach of designing a data format to serialize any C++ structure, rather than mapping a C++ structure to a simpler text-based structure.

On 21/09/2020 16:25, Mathias Gaunard via Boost wrote:

...

That is by design. Boost.JSON is not a serialization library. It addresses the use case for a JSON DOM with accompanying parser and serializer. This is not an insignificant use-case; witness the popularity of RapidJSON and nlohmann's JSON for Modern C++. In fact didn't you say you are already using RapidJSON?

I use RapidJSON because it's the established industry standard to do precisely what this library does.

For me RapidJSON is a legacy design useful only for legacy codebases. sajson and simdjson are the current state of the art for new C++ code. If you don't mind modifying the JSON being parsed (i.e. it's a mutable buffer and you will throw away the JSON afterwards in any case), refactoring around sajson's in-JSON AST generation delivers very sizeable performance gains.

As I said I don't believe the value proposition of this library is sufficient to replace existing things when all it does is the same thing but with a Boost logo on it. There is no guarantee at this stage this library will get any traction or maintenance on the same level as one which is used by thousands of projects, so it's not worth the risk to me.

Boost has a long history of duplicating something already widely used in the C++ ecosystem, but "Boostified" so it has STL naming and design patterns. And that's fine by me. People who like a standard C++ API design over other considerations will choose the Boost duplicate. Niall