On 2014-08-18 18:56, Robert Ramey wrote:

Roland,

Get documentation into your package so we can include it in "Library Writers Workshop"

Robert Ramey

Robert, I'll see what I can come up with until then. As written in another thread, I am not sure yet how to document the library in a more formal way. The statements' interface change while being used (not really, of course, but it kind of behaves that way). Here are a few examples (the first one will compile with the current development branch, the others will compile with any recent release): auto x = select(sqlpp::value(7).as(sqlpp::alias::a)); x is a statement that has a bunch of member functions like from(), where(), having(), etc, but none of those are required to be called. You can let this be executed by the database as is. auto y = select(all_of(t)); // t being a table y is a statement that has the same member functions, but two of them are required to execute the statement: from() and where(). I.e. db(y); // This will trigger two static asserts db(y.from(t).where(true)); // This will compile And auto z = select(all_of(t)).from(t).where(t.name == "XX"); z is a statement that can be executed by a database. It does not have the methods from() and where() anymore. You can call having(), group_by(), etc, though. This behaves really smoothly, but I don't know how to formally document it yet. Best, Roland

On 2014-08-18 19:58, Adam Wulkiewicz wrote:

Hi Roland,

Roland Bock wrote:

...

_NULL handling:_ Enabled by the restructured code and spurred by the library quince by Michael Shepanski, sqlpp11 can now calculate which result fields can or cannot be NULL.

Speaking of which, handling NULL for result values has been discussed a lot. The library now has compile-time configurable behavior, you can choose between an std::optional-like interface and mapping NULL to the trivial value of the result type, e.g. 0 for numbers or "" for strings, see also https://github.com/rbock/sqlpp11/wiki/NULL

You wrote in the docs:

<cite> One often discussed alternative would be boost::optional or (in the future) std::optional. There is one drawbacks (correct me if I am wrong, please):| optional| cannot be used for binding result values because it is unclear whether there already is a value to bind to. </cite>

What do you mean by that?

If I understand correctly, you have in mind returning boost::optional<> from a function. It's ok to do it, the value is stored in optional and deep copies are done if needed. http://www.boost.org/doc/libs/1_56_0/libs/optional/doc/html/boost_optional/t...

http://www.boost.org/doc/libs/1_56_0/libs/optional/doc/html/boost_optional/q...

Regards, Adam

Adam, I was referring to what some sql libraries do, e.g. Mysql's C interface: They take pointers to some memory and then write result fields to those memory blocks, see for instance http://dev.mysql.com/doc/refman/5.7/en/mysql-stmt-bind-result.html I don't know if that would be legal to do with the value in optional. Regards, Roland

On 2014-08-18 22:43, Adam Wulkiewicz wrote:

Roland Bock wrote:

...

I was referring to what some sql libraries do, e.g. Mysql's C interface: They take pointers to some memory and then write result fields to those memory blocks, see for instance http://dev.mysql.com/doc/refman/5.7/en/mysql-stmt-bind-result.html

I don't know if that would be legal to do with the value in optional. If some object containing value (corresponding to the data stored in DB) is created at some point, it may be stored in optional.

I'm guessing that the problem exists because select is lazily executed. C++ objects corresponding to the data aren't created (e.g. std::string, int, float, etc.). Instead some pointers to buffers are kept (in sqlpp::result_field_t?). And C++ objects are created and returned later from value() method or conversion operators each time one of them is called. Is that right? Not quite. Lets look at integral.h and the partial specialization of

template result_field_t {...}; Now, this contains an int64_t value. The address of this value is given to the backend in the method bind() when fetching each result row (no laziness here). It seems to me that I cannot replace int64_t by boost::optional. For instance, I cannot call get() to obtain the address of the value if the optional is not initialized (I would run into an assert).

But AFAIU if the C++ objects were created along with the representation of a row, the results could be stored as optionals. Or am I missing something? Yes the values are set when the row is fetched. The objects are re-used for each row. And the backend is given pointers to write the value into as explained above.

But while writing this, I just realized: I could do is offer a conversion operator for std::optional, of course. Thus, given a row with a column `a` which can be NULL, you would then have std::optional a1 = row.a; // OK int64_t a2 = row.a; // compile failure

I saw that there are optional-like wrappers of values, e.g. sqlpp::text. They're close to optionals since they're storing a value and some flag. What are they used for?

These are the result fields as described above. I guess the key for understanding their mechanics is the bind function. This gets called when a row is fetched: It hands over the address of the field's value to the database backend where it will be used in methods like mysql-stmt-bind-result for instance. I am offering the optional-like interface since I cannot use optional to actually store the field's value, if I understand the semantics of optional correctly. But as mentioned above, I probably could offer a converter to optional.

Are you using tabs? The code doesn't seem to look right on GitHub.

Yes, but actually, I think the worse problem is that vim indents template code a bit weirdly. Its not so bad with a tab width of two, but I'll have to take care of that sometime soon :-) Regards, Roland

On 2014-08-19 02:09, Adam Wulkiewicz wrote:

Roland Bock wrote:

...

On 2014-08-18 22:43, Adam Wulkiewicz wrote:

...

But AFAIU if the C++ objects were created along with the representation of a row, the results could be stored as optionals. Or am I missing something? Yes the values are set when the row is fetched. The objects are re-used for each row. And the backend is given pointers to write the value into as explained above.

But while writing this, I just realized: I could do is offer a conversion operator for std::optional, of course. Thus, given a row with a column `a` which can be NULL, you would then have

std::optional a1 = row.a; // OK int64_t a2 = row.a; // compile failure

Is it the same way with bigger objects, e.g. std::strings? Yes, but see below. Is the pointer to std::string passed to the backend (which means that some temporary buffer must be used) or must the pointer to the memory owned by std::string be passed, after resizeing a string? Afaik, I must not pass the memory owned by a string to anybody for modification, since writing to the return values of both std::string::data() and std::string:c_str() results in undefined behavior.

Anyway, with text results, the situation is special since the memory is actually owned by the backend. All the backends I've seen so far work this way. [...]

Even temporary buffers could be ommited. Hmm, it probably even wouldn't be required to pass an optional to the backend. An optional could be created with default-constructed value:

member = optional(int64_t());

or re-constructed in case there were no valid value stored, probably like this:

if ( !member ) member = optional(int64_t());

and then the address of this value already stored within optional could be passed deeper:

if ( backend.fill(member.get_ptr()) == false ) member = optional();// null

Or am I missing something?

Sure that's possible, but what's the benefit over using a plain value and a flag? For each row to be fetched, I would have to check and potentially reconstruct the optional value just to invalidate it again, if the value is NULL in the new row. In my eyes this just adds overhead for the internal representation.

Btw, I'm not saying you should change the interface. I was just surprised that it couldn't be done.

Got that :-) Regards, Roland

On 2014-08-19 04:17, Gavin Lambert wrote:

On 19/08/2014 09:55, Roland Bock wrote:

...

...

I'm guessing that the problem exists because select is lazily executed. C++ objects corresponding to the data aren't created (e.g. std::string, int, float, etc.). Instead some pointers to buffers are kept (in sqlpp::result_field_t?). And C++ objects are created and returned later from value() method or conversion operators each time one of them is called. Is that right? Not quite. Lets look at integral.h and the partial specialization of

template result_field_t {...};

Now, this contains an int64_t value. The address of this value is given to the backend in the method bind() when fetching each result row (no laziness here). It seems to me that I cannot replace int64_t by boost::optional. For instance, I cannot call get() to obtain the address of the value if the optional is not initialized (I would run into an assert).

Presumably it is the backend that knows whether the column/result is NULL or not. Correct. Therefore it should be the backend's responsibility to fill in the optional correctly If I had an optional in the result field, yes. -- ie. you should be passing an address/reference to the entire optional, not the internal integer. Some backends have functions like this (simplified):

void get_int_field(int index, int* retval); How would I interact with such an interface if I had an optional<int>? As suggested by Adam: optional<int> member; ... if (!member) member = 0 get_int_field(17, member.get());

Otherwise how does the backend return a NULL value?

The backend is called with two parameters, one pointer for the value, the other for the is_null information. Regards, Roland

I think we've had a terminology clash. By "backend" I thought you meant "the sqlpp11 class that knows how to talk to the native driver", not the native driver itself. Of course the native driver probably won't know how to drive an optional, nor should it be expected to.

There are several layers, I assume:

1. User code 2. sqlpp11 database-independent frontend 3. sqlpp11 database-specific connector 4. native database library

Nice definition of layers. AFAIK 3. is named "bindings" in UBlas where it binds to eg Atlas, Lapack The Multiprecision lib also has support for interfacing with various 3rd party libs like GMP but there I can't find a specific name.

On 2014-08-19 09:39, Gavin Lambert wrote:

On 19/08/2014 19:21, Roland Bock wrote:

...

On 2014-08-19 04:17, Gavin Lambert wrote:

...

...

Now, this contains an int64_t value. The address of this value is given to the backend in the method bind() when fetching each result row (no laziness here). It seems to me that I cannot replace int64_t by boost::optional. For instance, I cannot call get() to obtain the address of the value if the optional is not initialized (I would run into an assert). [...] Therefore it should be the backend's responsibility to fill in the

On 19/08/2014 09:55, Roland Bock wrote: optional correctly If I had an optional in the result field, yes. -- ie. you should be passing an address/reference to the entire optional, not the internal integer. Some backends have functions like this (simplified):

void get_int_field(int index, int* retval);

How would I interact with such an interface if I had an optional<int>?

I think we've had a terminology clash. By "backend" I thought you meant "the sqlpp11 class that knows how to talk to the native driver", not the native driver itself. Of course the native driver probably won't know how to drive an optional, nor should it be expected to.

There are several layers, I assume:

1. User code 2. sqlpp11 database-independent frontend 3. sqlpp11 database-specific connector 4. native database library Perfectly correct :-)

Between layers 3 & 4 obviously you have to use whatever the native library supports, which is unlikely to be boost::optional (but still possible in some cases). So you might have to provide a raw int64_t pointer to the database up front, and translate from a int64_t pointer and an "is this null" method call (or a bool*) to a boost::optional when it calls you back saying the complete row is ready. (I'm assuming this is asynchronous, otherwise it's easier.)

As of today, it is synchronous.

But between layers 1 & 2 and 2 & 3 you'd only have boost::optionals.

I can see the reasoning for 1&2. And I understand that it can be done with 2&3 of course, but I am not sure there is a benefit.

...

...

Otherwise how does the backend return a NULL value? The backend is called with two parameters, one pointer for the value, the other for the is_null information.

So once you know that those have been filled in, you can translate it into a boost::optional to be returned to the higher layer. It does mean the value has to be copied (unless boost::optional has had move assignment added since I last looked), but you'd be doing that anyway for std::string so this doesn't seem any worse than that.

I can see that for the 1&2 interface, but not necessarily for 2&3 since then I would have two copies of the string: once when obtaining the value from 4 and copying it into 2&3, one for copying it from 2->1. As of now, I create a temporary string when the value is requested in the 1&2 interface. Regards, Roland

On 2014-08-19 14:03, Adam Wulkiewicz wrote:

Roland Bock wrote:

...

On 2014-08-19 09:39, Gavin Lambert wrote:

...

I think we've had a terminology clash. By "backend" I thought you meant "the sqlpp11 class that knows how to talk to the native driver", not the native driver itself. Of course the native driver probably won't know how to drive an optional, nor should it be expected to.

There are several layers, I assume:

1. User code 2. sqlpp11 database-independent frontend 3. sqlpp11 database-specific connector 4. native database library

Perfectly correct :-)

...

Between layers 3 & 4 obviously you have to use whatever the native library supports, which is unlikely to be boost::optional (but still possible in some cases). So you might have to provide a raw int64_t pointer to the database up front, and translate from a int64_t pointer and an "is this null" method call (or a bool*) to a boost::optional when it calls you back saying the complete row is ready. (I'm assuming this is asynchronous, otherwise it's easier.) As of today, it is synchronous.

...

But between layers 1 & 2 and 2 & 3 you'd only have boost::optionals. I can see the reasoning for 1&2. And I understand that it can be done with 2&3 of course, but I am not sure there is a benefit.

As I see it, the benefit would be the use of the standard-approved way of handling values that could be invalid instead of using a library-specific handling.

But I think optionals wouldn't be safe. Correct me if I'm wrong. If optionals were used without a check for validity and the unexpected NULL value was set in the DB (maybe by mistake), it could result in segmentation fault. Of course assuming that the macro BOOST_ASSERT() wouldn't be expanded to some exception throw. This could lead to some security vulnerabilities in apps using this library. I thought the whole purpose of the optional-interface was to make people to check for validity first?

If I understand you correctly, you should be more happy with the current implementation then: As of today, depending on the compile-time configuration in the connector you have the following possible behaviors: You can always check for NULL by calling is_null. But if you ignore the outcome and try to obtain the value of a field which happens to be NULL Variant A: an exception is thrown Variant B: the "trivial" value is returned, 0 for numbers. The Variant is chosen at compile time. [...] Regards, Roland PS: I cut the rest since we are in agreement there and the number of copies, possible RVO etc depend on implementation details which might have to be adjusted if optional were to be used :-)

On 2014-08-19 16:01, Adam Wulkiewicz wrote:

Roland Bock wrote:

...

On 2014-08-19 14:03, Adam Wulkiewicz wrote:

...

But I think optionals wouldn't be safe. Correct me if I'm wrong. If optionals were used without a check for validity and the unexpected NULL value was set in the DB (maybe by mistake), it could result in segmentation fault. Of course assuming that the macro BOOST_ASSERT() wouldn't be expanded to some exception throw. This could lead to some security vulnerabilities in apps using this library. I thought the whole purpose of the optional-interface was to make people to check for validity first?

If I understand you correctly, you should be more happy with the current implementation

Yes, exceptions and default values seems to be better to handle the NULL/unexpected values.

I started the discussion because you wrote in the Wiki that the problem with optionals is related to the binding of values which IMO is just an implementation detail. Maybe it would be a good idea to explicitly state why do you think that optionals shouldn't be used. Define a rationale behind it. Thanks for the discussion, its been quite helpful :-)

Cheers, Roland

On 2014-08-20 01:54, Gavin Lambert wrote:

On 20/08/2014 00:49, Roland Bock wrote:

...

If I understand you correctly, you should be more happy with the current implementation then: As of today, depending on the compile-time configuration in the connector you have the following possible behaviors:

You can always check for NULL by calling is_null. But if you ignore the outcome and try to obtain the value of a field which happens to be NULL

Variant A: an exception is thrown Variant B: the "trivial" value is returned, 0 for numbers.

The Variant is chosen at compile time.

That seems like a reasonable justification for not using optional, then, as optional isn't configurable in this way.

If you made your type implicitly convertible to optional and back, that ought to make everybody happy. :) That's possible of course :-)

(Well, except maybe the folks who hate implicit conversions, but they're never happy.)

rofl! Cheers, Roland

On 20/08/2014 18:30, Michael Shepanski wrote:

On 20/08/2014 9:54 AM, Gavin Lambert wrote:

...

If you made your type implicitly convertible to optional and back, that ought to make everybody happy. :) (Well, except maybe the folks who hate implicit conversions, but they're never happy.)

I love implicit conversions. I love implicitly converting const char* to std::string. I love implicitly converting std::string to boost::optionalstd::string. So imagine my heartache when they told me I couldn't implicitly convert const char* to boost::optional<string>. What, you mean I can't really use const char * /wherever/ I can use std::string?

I feel that I'm in for more tears if we have an implicit conversion from boost::optional to something else.

Two implicit conversions is double-jeopardy, partly because that would bring an infinite cycle into scope. Yes, it does mean that you have to be a little more explicit sometimes, but I'm unconvinced that would be a real problem in practice. Methods in the sqlpp interface would accept a some_other_optional<T>, which means that you could pass them one of those directly, or a boost::optional<T> (via some_other_optional<T> implicit constructor), or a T (via some_other_optional<T> implicit constructor). Note that the latter cannot implicitly be constructed as a boost::optional<T> first because that would require two implicit conversions. So everything should just naturally work. (And you can't use const char* wherever you use std::string anyway -- "a" + "b" is a very different thing from std::string("a") + "b".)

On 08/19/2014 02:49 PM, Roland Bock wrote:

If I understand you correctly, you should be more happy with the current implementation then: As of today, depending on the compile-time configuration in the connector you have the following possible behaviors:

You can always check for NULL by calling is_null. But if you ignore the outcome and try to obtain the value of a field which happens to be NULL

Variant A: an exception is thrown Variant B: the "trivial" value is returned, 0 for numbers.

The Variant is chosen at compile time.

I find it a bit worrying that this behavior is changeable at compile-time, as it makes reviewing and reuse more challenging. With optional<T> you will get variant A. If you want variant B, then you use optional<T>::value_or(). This makes your intent clear in the code.

On 2014-08-21 12:32, Bjorn Reese wrote:

On 08/19/2014 02:49 PM, Roland Bock wrote:

...

If I understand you correctly, you should be more happy with the current implementation then: As of today, depending on the compile-time configuration in the connector you have the following possible behaviors:

You can always check for NULL by calling is_null. But if you ignore the outcome and try to obtain the value of a field which happens to be NULL

Variant A: an exception is thrown Variant B: the "trivial" value is returned, 0 for numbers.

The Variant is chosen at compile time.

I find it a bit worrying that this behavior is changeable at compile-time, as it makes reviewing and reuse more challenging.

With optional<T> you will get variant A.

That is not the case, is it? * If I call get_value() on a NULL field in variant A, an exception gets thrown. * If I call get() on an uninitialized boost::optional, I run into a BOOST_ASSERT. By default, this is equivalent to C-assert (ie not an exception), but you can also turn it off completely at compile time. Then, you will have basically undefined behavior. Or you can configure BOOST_ASSERT at compile time to do something entirely different. I fail to see how the optional-way would any better with respect to compile-time configuration and code re-use or code reviews. When get() is used, you have no idea what will happen tomorrow. You only know for sure when get_value_or() is used. That's why I prefer Variant B anyway: No matter what, at least you get a defined value in case you run into a NULL. And if you really need to know that something is NULL, you have to check that regardless of whether it is an optional or not. Best, Roland

On 2014-08-22 12:29, Bjorn Reese wrote:

On 08/21/2014 05:32 PM, Roland Bock wrote:

...

* If I call get() on an uninitialized boost::optional, I run into a

optional<T>::value() throws. Notice that get() is not part of the std::optional proposal. Oh, missed that. Sorry.

Thanks for the std::optional information :-) Ok, so I could have operator optional<int>; // only in variant A operator int() const; // only in variant B In addition: bool is_null() const; int value() const; // throws if field is null Best, Roland

Roland Bock wrote:

On 2014-08-19 16:43, Adam Wulkiewicz wrote:

...

I'm wondering, could it be possible to implement similar, compile-time interface in sqlpp, e.g. something similar to MPL? I'm asking because what the sqlpp library requires is very complicated:https://github.com/rbock/sqlpp11/blob/master/tests/Sample.h I'm aware that its purpose is to create a type reflecting required columns and to use them from the level of C++ but couldn't it be done simpler? The crucial part is the _member_t template. I need that one. The other stuff can be constructed whatever way you like. But if the name of the

column is `beta`, then I need a way to get this:

template<typename T> struct _member_t { T beta; T& operator()() { return beta; } const T& operator()() const { return beta; } };

This is the magic ingredient that makes tables, result rows and parameter sets of prepared statements to have members with a proper name.

The way it works is best to be observed in the table_t template, see https://github.com/rbock/sqlpp11/blob/master/include/sqlpp11/table.h

template struct table_t : public table_base_t, public ColumnSpec::_name_t ::template _member_t>...

Ok AFAIU a struct like _member_t should define some convenient member variable for the user and must define operator() for the library. But the rest could be automatically generated, couldn't it? Why not just pass a list of templates of classes adapted to MemberType concept (defined operator()) into the table/column/etc.? I'm thinking about something like the code below. I don't know exactly what's required so this is just an example of a technique rather than a solution ready-to-use in sqlpp. namespace sqlpp { template