Library list generated from meta/libraries.json files

Peter Dimov

5 Jan 2015 5 Jan '15

2:10 p.m.

I've put a library list generated from the meta/libraries.json files, at: http://www.pdimov.com/tmp/boost/index.html Library authors may wish to take a look at their library entry and see if they're happy with it. What I've noticed so far is that many libraries are missing the boost-version (First Release) entry, and that Config is missing authors for some reason. The other thing is that the category names in the json files are the internal identifiers, which I've left as-is, as I didn't want to hardcode a mapping. We'll have to decide whether to use the proper names in the json files, or whether to have a separate categories.json with the mapping somewhere. I tend towards the first option.

Show replies by date

Daniel James

5 Jan 5 Jan

5:14 p.m.

On 5 January 2015 at 14:10, Peter Dimov wrote:

...

I've put a library list generated from the meta/libraries.json files, at:

http://www.pdimov.com/tmp/boost/index.html

We're already dealing with this on the website, this is how the documentation list is generated. The full data is at: http://www.boost.org/doc/libraries.xml Which was set up by Rene years ago. It includes libraries with no existing metadata file. Nowadays it's maintained by a mix of php scripts and manual editing. The way it stores historical data is awkward because I shoehorned it into existing code, so it'll probably be easier to handle if I write a little web service to get a release's details in json format.

...

Library authors may wish to take a look at their library entry and see if they're happy with it. What I've noticed so far is that many libraries are missing the boost-version (First Release) entry, and that Config is missing authors for some reason.

Historical data is handled centrally, so it would be redundant and error prone to store it in the library repos (e.g. if a library was pulled from a release, but wasn't updated accordingly). But some of the first metadata files still have it.

...

The other thing is that the category names in the json files are the internal identifiers, which I've left as-is, as I didn't want to hardcode a mapping. We'll have to decide whether to use the proper names in the json files, or whether to have a separate categories.json with the mapping somewhere. I tend towards the first option.

As you've probably noticed by now, the categories are in the xml file.

Peter Dimov

5:38 p.m.

Daniel James wrote:

...

On 5 January 2015 at 14:10, Peter Dimov wrote:

...
I've put a library list generated from the meta/libraries.json files, at:

http://www.pdimov.com/tmp/boost/index.html

We're already dealing with this on the website, this is how the documentation list is generated. The full data is at:

http://www.boost.org/doc/libraries.xml

Which was set up by Rene years ago.

Yes, I knew about this file. I'm not sure, however, what is the point of the pre-library meta/libraries.json files then, as they contain (a subset of) the same information. I had assumed that we'll be transitioning away from centrally stored files, and will be generating everything from the individual modules themselves. So, under that assumption, I expected the info in libraries.xml to be moved to the appropriate meta/libraries.json files, and then disappear. On second thought though, I now see that if someone requests the library list for boost_1_38_0, there would be no way to produce that from the json files.

...

Historical data is handled centrally, so it would be redundant and error prone to store it in the library repos (e.g. if a library was pulled from a release, but wasn't updated accordingly).

Right... but what is the point of the json files then? Wait, I think I figured it out... the release script generates (would generate) the release-specific info from the json files, then puts (would put) that into libraries.xml?

...

As you've probably noticed by now, the categories are in the xml file.

Right. This index.html is what "bpm index" generates, on the basis of what you have currently installed. It scans for meta/libraries.json files, and builds the list. So it would've been more convenient for me if the category names in the json are the full, human-readable names, rather than the identifiers, as I'd like to avoid downloading a separate category list. But I see how this might create a problem for the script that generates libraries.xml, and I also see why we can no longer change "workarounds" to "Workarounds" for consistency. A compromise might be to add category_name: [] to the json files and keep the current category: [] as is. Or, I could just hardcode the names and be done with it. :-)

Daniel James

6:22 p.m.

On 5 January 2015 at 17:38, Peter Dimov wrote:

...

Yes, I knew about this file. I'm not sure, however, what is the point of the pre-library meta/libraries.json files then, as they contain (a subset of) the same information.

It's so that this data can be managed by the library maintainers. Currently I have to do quite a lot of admin just to add a new library or change the details. And people don't realise they have to let me know when something has changed. Often the first time I would realise is when someone complains after the release. Eventually I want to also generate libs/libraries.htm and lib/maintainers.txt from it. I really should get round to doing that soon-ish, it's pretty awkward that it's part of the website.

...

I had assumed that we'll be transitioning away from centrally stored files, and will be generating everything from the individual modules themselves. So, under that assumption, I expected the info in libraries.xml to be moved to the appropriate meta/libraries.json files, and then disappear.

Maybe, but I never envisioned this as a full solution. If we have a full package system, it might want to use a different file format. When I set this up, it looked quite likely that it would be based on a third party system that would use something completely different, or that the implementer would want to design their own file format. I also didn't want to force it on anyone, anyone who didn't accept my pull-request would be very unlikely to keep the file up to date.

...

...
Historical data is handled centrally, so it would be redundant and error prone to store it in the library repos (e.g. if a library was pulled from a release, but wasn't updated accordingly).

Right... but what is the point of the json files then?

Answered earlier. I'm not sure why a package manager would need to know when a library was first released.

...

Wait, I think I figured it out... the release script generates (would generate) the release-specific info from the json files, then puts (would put) that into libraries.xml?

Sort of, it can also be updated from the beta and master branches.

...

But I see how this might create a problem for the script that generates libraries.xml, and I also see why we can no longer change "workarounds" to "Workarounds" for consistency.

Could make it case insensitive. It probably should be.

...

A compromise might be to add category_name: [] to the json files and keep the current category: [] as is. Or, I could just hardcode the names and be done with it. :-)

It shouldn't be too hard to automatically update the names, or add them as part of the release process.

Peter Dimov

6:42 p.m.

Daniel James wrote:

...

If we have a full package system, it might want to use a different file format.

I already have a full package system and it uses meta/libraries.json to generate the library list because those files already exist. :-)

...

I'm not sure why a package manager would need to know when a library was first released.

It doesn't. I just looked at http://www.boost.org/doc/libs/1_57_0/ and tried to replicate it. It has First Release, so I assumed that people are interested in it, so I included it. It didn't occur to me that this field is present because the script uses it to generate the historical lists, not because it's of particular interest for users. I was somewhat misled by the fact that functional/meta/libraries.json does have boost-version for all functional/ libraries. And since you're the author of that, I assumed that this is the golden reference for the format, and that therefore boost-version is a required field, and it's been omitted from all other files by mistake. But the mistake was, apparently, mine. :-) Incidentally, some libraries have "std": [ "tr1" ], and some have "std-proposal": false, "std-tr1": false. (Actually, now that I grepped, only Assert has the latter. I suppose "std" is correct and Assert is using an outdated schema?)

Daniel James

7:15 p.m.

On 5 January 2015 at 18:42, Peter Dimov wrote:

...

Daniel James wrote:

...
If we have a full package system, it might want to use a different file format.

I already have a full package system and it uses meta/libraries.json to generate the library list because those files already exist. :-)

You didn't when I designed this. At the time the existing efforts at creating a package manager were a moving target, so I felt it was best to ignore them, and then adapt to whatever they eventually settled on. Most of the people involved had very different ideas of how things should be done to me, so I didn't expect them to want to use anything I created. If it isn't clear, I'm happy for someone else to take charge of the metadata files, and will update the website scripts to use whatever they want.

...

...
I'm not sure why a package manager would need to know when a library was first released.

It doesn't. I just looked at http://www.boost.org/doc/libs/1_57_0/ and tried to replicate it. It has First Release, so I assumed that people are interested in it, so I included it. It didn't occur to me that this field is present because the script uses it to generate the historical lists, not because it's of particular interest for users.

It's there because Rene included it in the original version of the xml file. There were also some other fields which I removed because I couldn't keep the data up to date and it was confusing people (e.g. whether the library was header only). It's just that I'd rather let the update script automatically maintain data when it can. Less work that way.

...

I was somewhat misled by the fact that functional/meta/libraries.json does have boost-version for all functional/ libraries. And since you're the author of that, I assumed that this is the golden reference for the format, and that therefore boost-version is a required field, and it's been omitted from all other files by mistake. But the mistake was, apparently,

There's some documentation here: http://www.boost.org/development/library_metadata.html I'll add a link to the sidebar soon, and maybe update some of the development pages to link to it.

...

Incidentally, some libraries have "std": [ "tr1" ], and some have "std-proposal": false, "std-tr1": false. (Actually, now that I grepped, only Assert has the latter. I suppose "std" is correct and Assert is using an outdated schema?)

Yes, that was changed by request. My update script normalises it so that data in doc/libraries.xml is consistent.

Peter Dimov

9:03 p.m.

Daniel James wrote:

...

There's some documentation here:

http://www.boost.org/development/library_metadata.html

Thanks. We probably want to add "c++11", "c++14", and the various TSes to the allowed values of "std". TR1 isn't especially relevant nowadays, and most/all TR1 libraries are in C++11 (and in 14, so we also need to decide whether to only list c++11, or both).

Peter Dimov

9:13 p.m.

Daniel James wrote:

...

http://www.boost.org/development/library_metadata.html

I also think that we should change the "Correctness and testing" category to "Correctness, testing, error reporting, and diagnostics" and put Log, Exception, and throw_exception there. They're Miscellaneous now, which doesn't seem quite correct. Incidentally, Exception appears to be missing meta/libraries.json. We're probably waiting for Emil to merge the pull request, I'll ping him. :-)

Peter Dimov

11:33 p.m.

...

Incidentally, Exception appears to be missing meta/libraries.json.

Yeah, and 22 other libraries too, as it turns out. :-) Daniel, you should just merge the open metadata pull requests. IMO.

Daniel James

6 Jan 6 Jan

12:06 a.m.

On 5 January 2015 at 23:33, Peter Dimov wrote:

...

...
Incidentally, Exception appears to be missing meta/libraries.json.

Yeah, and 22 other libraries too, as it turns out. :-) Daniel, you should just merge the open metadata pull requests. IMO.

If a pull request isn't accepted, then that suggests I'd have to continue maintaining the metadata myself, in which case I'd rather keep it in the website. Although, if it's absolutely required, then I suppose that's not an option.

Peter Dimov

12:23 a.m.

Daniel James wrote:

...

If a pull request isn't accepted, then that suggests I'd have to continue maintaining the metadata myself, in which case I'd rather keep it in the website.

Well, yes, I suppose. I hadn't thought of that. On the other hand, unmaintained libraries aren't likely to change, so the metadata would probably not be that much of a maintenance burden; and if you're going to generate libraries.htm and maintainers.txt from the json files, the latter would need to be present.

Rene Rivera

12:29 a.m.

On Mon, Jan 5, 2015 at 6:23 PM, Peter Dimov wrote:

...

Daniel James wrote:

...
If a pull request isn't accepted, then that suggests I'd have to continue maintaining the metadata myself, in which case I'd rather keep it in the website.

Well, yes, I suppose. I hadn't thought of that.

On the other hand, unmaintained libraries aren't likely to change, so the metadata would probably not be that much of a maintenance burden; and if you're going to generate libraries.htm and maintainers.txt from the json files, the latter would need to be present.

I don't get it.. I don't see anything in the meta/libraries.json that would need frequent changes. For at least the Predef one I don't see it ever needing to change. Is there something I'm missing? -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

Daniel James

12:57 a.m.

On 6 January 2015 at 00:29, Rene Rivera wrote:

...

I don't get it.. I don't see anything in the meta/libraries.json that would need frequent changes. For at least the Predef one I don't see it ever needing to change. Is there something I'm missing?

Changes are infrequent, but they're still a pain. I was getting really get fed up dealing with unmaintained libraries. I'd rather not touch them at all.

Edward Diener

5 Jan 5 Jan

11:17 p.m.

On 1/5/2015 1:42 PM, Peter Dimov wrote:

...

Daniel James wrote:

...
If we have a full package system, it might want to use a different file format.

I already have a full package system and it uses meta/libraries.json to generate the library list because those files already exist. :-)

I am all for the work you are doing by creating a Boost package manager but a "full" package system would allow library XXX's aa.aa release to be packaged with library YYY's bb.bb release if they were compatible. In other words while I greatly applaud and thank you for bpm's ability to package a subset of Boost from a given Boost tree, a fully modularized Boost would allow a specific version of an individual library, however that is eventually specified, to be packaged with a different specific version of another library if some dependency information indicated that they will work correctly with each other. I realize that this is something for the future but if we ever go to a model where individual Boost libraries can be distributed separately we will need a system whereby version information and dependency information is specified somewhere as part of the library's metadata.

3612

Age (days ago)

3613

Last active (days ago)

List overview

Download

13 comments

4 participants

participants (4)

Daniel James
Edward Diener
Peter Dimov
Rene Rivera