On 2014-07-23 14:09, Michael Shepanski wrote:

On 23/07/2014 9:31 PM, Andrey Semashev wrote:

...

The library build scripts should perform auto-detection checks at build time. I have something like this in Boost.Log, although I test whether the compiler supports SSE/AVX and whether Message Compiler is available. But the approach should be similar - you attempt to compile a test application and set up build macros or select the actual sources to build from.

You can also take a look at Boost.IOStreams which detect if bzip2 and zlib are available. Boost.Thread also contains some auto-detection checks, I think.

Thanks Andrey.

It seems I'll have to lift my Boost.Build skills quite a bit. Fair enough.

Also I'm thinking that my approach of having separate backend libraries is not right. The idea had been that a user decides which backend library to download and build, but that is not going to be happening a boost context. Everything should be in the "quince" library, every user downloads it all, and its build script decides which components to build. Hi Michael,

There's also MySQL and MariaDb and Oracle and TransactSQL and Firebird and BerceleyDB and I don't know how many else, but a lot. To me it does not sound like a good idea to have all backends in one giant library. Just my 2ct. Regards, Roland

Michael Shepanski wrote:

On 23/07/2014, at 10:54 PM, Roland Bock wrote:

...

On 2014-07-23 14:09, Michael Shepanski wrote:

...

Also I'm thinking that my approach of having separate backend libraries is not right. The idea had been that a user decides which backend library to download and build, but that is not going to be happening a boost context. Everything should be in the "quince" library, every user downloads it all, and its build script decides which components to build.

Hi Michael,

There's also MySQL and MariaDb and Oracle and TransactSQL and Firebird and BerceleyDB and I don't know how many else, but a lot. To me it does not sound like a good idea to have all backends in one giant library.

I don't have any strong opinions about this, and I'm happy to go with the conventional approach -- as soon as I know what that is.

For a data point, the approach in sqlate is to generate strings from the edsl and delegate the query handling to third party drivers (sqlate doesn't depend on Qt, but the test does) https://github.com/KDAB/sqlate/blob/master/tests/selecttest.cpp If the interface of your library only creates strings, you can similarly allow users of it to use the Qt drivers, your drivers from somewhere separate, something else entirely, or whatever comes out of the ISO DB SG. Thanks, Steve.

On Wed, Jul 23, 2014 at 5:13 PM, Michael Shepanski wrote:

On 23/07/2014, at 10:54 PM, Roland Bock wrote:

...

On 2014-07-23 14:09, Michael Shepanski wrote:

...

Also I'm thinking that my approach of having separate backend libraries is not right. The idea had been that a user decides which backend library to download and build, but that is not going to be happening a boost context. Everything should be in the "quince" library, every user downloads it all, and its build script decides which components to build.

Hi Michael,

There's also MySQL and MariaDb and Oracle and TransactSQL and Firebird and BerceleyDB and I don't know how many else, but a lot. To me it does not sound like a good idea to have all backends in one giant library.

I don't have any strong opinions about this, and I'm happy to go with the conventional approach -- as soon as I know what that is.

Taking your lead and daring to imagine a future with quince backend code for a great many DBMSes, I ask: what do you propose? Would there be a separate boost/libs/quince_xyz for each DBMS xyz?

Also, what disadvantages do you see in the "one giant library" approach (assuming that its build script chooses which parts to compile, based on the presence of third-party libraries).

Monolithic library is not a good solution for distributed binaries. Once compiled library will require all the DBMSes it was compiled against and not support any other. E.g. on Linux such library package will pull all DBMS packages, even though the user might need just one. I think separate optionally compiled binaries (one backend per binary) is the preferred solution. That doesn't mean the library source code and docs must be split, though.

On 2014-07-23 15:13, Michael Shepanski wrote:

On 23/07/2014, at 10:54 PM, Roland Bock wrote:

...

On 2014-07-23 14:09, Michael Shepanski wrote:

...

Also I'm thinking that my approach of having separate backend libraries is not right. The idea had been that a user decides which backend library to download and build, but that is not going to be happening a boost context. Everything should be in the "quince" library, every user downloads it all, and its build script decides which components to build.

Hi Michael,

There's also MySQL and MariaDb and Oracle and TransactSQL and Firebird and BerceleyDB and I don't know how many else, but a lot. To me it does not sound like a good idea to have all backends in one giant library. I don't have any strong opinions about this, and I'm happy to go with the conventional approach -- as soon as I know what that is.

Taking your lead and daring to imagine a future with quince backend code for a great many DBMSes, I ask: what do you propose? Would there be a separate boost/libs/quince_xyz for each DBMS xyz?

Also, what disadvantages do you see in the "one giant library" approach (assuming that its build script chooses which parts to compile, based on the presence of third-party libraries).

I guess that you would not want to maintain the backends for the umpteen databases. I don't know what the best approach is. Maybe you provide backends for the most popular databases and let others provide backends for the rest? I don't think that there is a precedent for this situation. Personally, I would prefer the individual repositories. It makes it easier to distribute the work load and you would not have to decide which databases to include in the main repositories and which to keep out. It would therefore also emphasize the vendor neutrality of the main library. Regards, Roland

...

I guess that you would not want to maintain the backends for the umpteen databases.

+1

...

I don't know what the best approach is. Maybe you provide backends for the most popular databases and let others provide backends for the rest?

I'll definitely be doing that, but it leaves open the question of how to organise it.

For me I would have one lib (and repository): mylib/ include/ src/ backend1/ backend2/ build/ But really it's up to you - as long as the structure is clearly organised and folks can find their way around I don't believe anyone should beat you up too much about it. In short use the right organisation for the job in hand. HTH, John.

On 24/07/2014 01:13, Michael Shepanski wrote:

Taking your lead and daring to imagine a future with quince backend code for a great many DBMSes, I ask: what do you propose? Would there be a separate boost/libs/quince_xyz for each DBMS xyz?

I'm not sure exactly how it works but from the modularisation discussion it's evident that Boost's library structure supports "sub-libraries" (aka submodules) that exist in the same repository but are built separately. I suspect what you'll probably want to do would be to create a "core" submodule that each of the separate backend submodules depend on (each of those also depending on a single external library as needed), and then finally a "front-end" top level that doesn't directly depend on anything but the core, but allows any of the DB-specific backends to be used with it. Boost.Numeric seems to be an existing example of a library divided into submodules. Boost.Math seems to be an example of a library without submodules that nevertheless builds multiple output libraries. I'm not sure which style is "preferred".

On 2014-07-24 04:35, Michael Shepanski wrote:

On 24/07/2014 9:23 AM, Gavin Lambert wrote:

...

I'm not sure exactly how it works but from the modularisation discussion it's evident that Boost's library structure supports "sub-libraries" (aka submodules) that exist in the same repository but are built separately.

I suspect what you'll probably want to do would be to create a "core" submodule that each of the separate backend submodules depend on (each of those also depending on a single external library as needed), and then finally a "front-end" top level that doesn't directly depend on anything but the core, but allows any of the DB-specific backends to be used with it.

Currently the "quince" library contains both the "core" and "front-end" components. I figured that nobody would want one without the other. What advantage do you see in separating them?

...

Boost.Numeric seems to be an existing example of a library divided into submodules.

Boost.Math seems to be an example of a library without submodules that nevertheless builds multiple output libraries.

I'm not sure which style is "preferred".

Putting together your response and all the other responses, I'm starting to stabilize on the following set of views:

- I should put just one new directory into boost/libs, i.e. boost/libs/quince.

- The backend-specific source code should be in subdirs of boost/libs/quince.

- These subdirs should be separate git modules, in the manner of boost/libs/numeric/*/ .

- The output should consist of a core quince .lib file, and separate .lib files for each backend, laid out in the manner of boost/bin.v2/libs/math/build/*/*/*/*/*.lib.

- boost/libs/quince/Jamfile.v2 should somehow detect which third-party libraries are present, and build the corresponding backend libraries. If there are more than zero, it should build the core quince library.

Sounds good to me except for the last item. Assuming there is no backend for the database I am interested in, I might want to have quince core in order to develop that backend. I therefore think that quince core should always be built. Cheers, Roland

On 25/07/2014 18:53, Michael Shepanski wrote:

...

Assuming there is no backend for the database I am interested in, I might want to have quince core in order to develop that backend. I therefore think that quince core should always be built.

In that case, sooner or later you're going to have to tweak quince's jam file, to make the building of your backend contingent on the presence of some third-party library (fron MySQL or Oracle or whatever).

You're assuming that all backends must be built as part of the quince tree. What if I wanted to write a backend private to a particular application and never intended to be published (perhaps it's some proprietary data format rather than a "real" database)? If the core were always built, it could be used with a custom backend that quince's jam file knows nothing about. (Provided that the core itself doesn't have to be modified to add a new backend -- but I would hope that's true as it's part of standard coding principles.)

I'm coming around to Karsten Ahnert's idea that I should ship sqlite. (I wouldn't call it a "default backend", as Karsten does, because application code still has to make a choice -- but that's just a quibble.).

All of sqlite or just a connector to it? I don't have any particular opinion on this either way, but bear in mind that if you do the former, users may still wish to use a different version of it (perhaps they need a bugfix from a newer version than shipped with quince), and (especially if you don't provide a way to do that) users will expect you to keep it up to date, which increases maintenance costs slightly.

On 25/07/2014 5:49 PM, Gavin Lambert wrote:

You're assuming that all backends must be built as part of the quince tree.

Quite right: I assumed that, and I shouldn't have.

What if I wanted to write a backend private to a particular application and never intended to be published (perhaps it's some proprietary data format rather than a "real" database)?

If the core were always built, it could be used with a custom backend that quince's jam file knows nothing about.

Fair enough, and we have since heard from Roland and Rob, so I think we have consensus: quince should get built regardless of whether any of the third-party libraries that I had in mind are already installed, and regardless of what is decided about shipping sqlite.

(Provided that the core itself doesn't have to be modified to add a new backend -- but I would hope that's true as it's part of standard coding principles.)

I share your hope. :) When I look at quince+quince_postgresql+quince_sqlite, I can tell you that all the DBMS-specific code is in the hands of the backends. Quince has various classes with virtuals, and the backends have classes that override those virtuals to do backend-specific things. And I've tried to future-proof it, e.g. by putting "virtual" on all the functions that generate SQL phrases, whether they currently have backend-specific behaviour or not. Unfortunately, however, I don't have a formal specification of exactly which ways DBMSes are allowed vary, so I can't guarantee that that a new DBMS won't provoke some unanticipated, unimagined change to quince itself.

...

I'm coming around to Karsten Ahnert's idea that I should ship sqlite. (I wouldn't call it a "default backend", as Karsten does, because application code still has to make a choice -- but that's just a quibble.).

All of sqlite or just a connector to it?

I'll definitely be shipping a backend (aka connector) to sqlite. But to ship sqlite itself or not? That is the question I was soliloquizing about.

I don't have any particular opinion on this either way, but bear in mind that if you do the former, users may still wish to use a different version of it (perhaps they need a bugfix from a newer version than shipped with quince), and (especially if you don't provide a way to do that) users will expect you to keep it up to date, which increases maintenance costs slightly.

Hm. On one hand, users are free to use their own backend library, which might be a very slightly modified version of quince_sqlite, to go with their own slightly different sqlite3.[ch]. And application code always has to say which backend library it is going to use for each database (see http://quince-lib.com/getting_started/creating_the_database_object.html). The only problem I see is if quince forces my chosen sqlite3.o into the link, leading to doubly-defined symbol errors. IANL (I Am Not A Linkerologist) but I suspect there is some solution based on ordering arguments to the linker, so that symbols in the user's sqlite3.o trump any in my sqlite3.o. --- Michael

On 23/07/2014 9:31 PM, Andrey Semashev wrote:

The library build scripts should perform auto-detection checks at build time. I have something like this in Boost.Log, although I test whether the compiler supports SSE/AVX and whether Message Compiler is available. But the approach should be similar - you attempt to compile a test application and set up build macros or select the actual sources to build from.

So let's say I'm trying to decide whether libpq is installed. Your suggestion is that I "attempt to compile a test application". So I would have to tell the compiler the location of libpq's incude directory (and if I attempt to link then I have to tell the linker the location of the libpq binary). Now, at the moment, the only way I know these locations is from the constants "libpq_include_dir" and "libpq_lib_dir", defined in user-config.jam (see http://quince-lib.com/preparation/configuring_boost_build.html). So I might as well test those constants directly, no? Or should I be determining the "installed" locations of those things by some other means?

You can also take a look at Boost.IOStreams which detect if bzip2 and zlib are available.

Its Jamfile.v2 says: # The biggest trick in this Jamfile is to link to zlib and bzip2 when # needed. To configure that, a number of variables are used, which must # be set by user with 'path-constant' either in Boost's root Jamfile, or # in user-config.jam.

Boost.Thread also contains some auto-detection checks, I think.

Its Jamfile.v2 says: # You need to provide the include path and lib path in the variables # PTW32_INCLUDE and PTW32_LIB respectively. You can specify these # paths in site-config.jam, user-config.jam or in the environment. I'm thinking that the only auto-detection I need is inspection of bjam constants. --- Michael

On 24/07/2014 5:42 PM, Andrey Semashev wrote:

Having constants so that the user is able to specify the paths manually (either in user-config.jam or bjam command line) is required. However, on Linux (and on other UNIX-like systems as well) there are standard locations for headers and libs which need not be specified in the compiler command line. So if libpq is installed but the bjam constants are not set you might still be able to compile your backend. In order to check that you'll have to attempt to compile a test application. Then there is pkgconfig, which can be used to obtain these paths automatically. It is quite common in Linux and probably other UNIX-like systems, although I'm not sure. Of course, how far you are willing to go in inspecting the environment is up to you.

There is another aspect that may require test applications. I don't know if that's your case but you may have requirements on specific versions or features of the third party libraries. You can use test applications to verify that the third party library fulfills your requirements.

Thanks. Okay, I think I understand the issues now. Not being an expert in either Boost.Build or any modern form of Unix, I find it all rather daunting, so I'd appreciate any opinions on what counts as an adequate, as opposed to ideal, solution. Meanwhile I will proceed on the assumption that no corners are to be cut. Regards, --- Michael

Support for pkgconfig is nice to have but can be added later. Actually, it'd be great if support for pkgconfig was included into Boost.Build.

That would be nice to have. A long while ago Roger Leigh worked on creating the pkgconfig files by using the autolink feature in boost. See here: https://svn.boost.org/trac/boost/ticket/1094 -- View this message in context: http://boost.2283326.n4.nabble.com/Use-of-third-party-libraries-tp4665479p46... Sent from the Boost - Dev mailing list archive at Nabble.com.

On 23/07/2014 9:31 PM, Andrey Semashev wrote:

...

The library build scripts should perform auto-detection checks at build time. I have something like this in Boost.Log, although I test whether the compiler supports SSE/AVX and whether Message Compiler is available. But the approach should be similar - you attempt to compile a test application and set up build macros or select the actual sources to build from.

So let's say I'm trying to decide whether libpq is installed. Your suggestion is that I "attempt to compile a test application". So I would have to tell the compiler the location of libpq's incude directory (and if I attempt to link then I have to tell the linker the location of the libpq binary). Now, at the moment, the only way I know these locations is from the constants "libpq_include_dir" and "libpq_lib_dir", defined in user-config.jam (see http://quince-lib.com/preparation/configuring_boost_build.html). So I might as well test those constants directly, no? Or should I be determining the "installed" locations of those things by some other means?

While it can be convenient to have a backdoor that lets the user explicitly set the path to the lib, on most Unix-like systems those libraries will be installed in standard location which are searched by default. You might want to look at how the multiprecision lib handles locating gmp, mpfr etc for inspiration. HTH, John.