On Fri, Jan 27, 2017 at 7:44 AM, Jonathan Wakely wrote:

On 27 January 2017 at 13:23, Jonathan Wakely wrote:

...

The link http://downloads.sourceforge.net/boost/boost_1_63_0.tar.bz2

It's insane that even works! Since there's no such path in the file release system.

...

redirects to http://downloads.sourceforge.net/project/boost/boost/ snapshots/master/boost_1_63_0.tar.bz2

Grrr.

which seems to be a snapshot from master, and VERY different to the

...

boost_1_63_0 release (which makes it a VERY misleading URL).

How often is that snapshot updated?

It's updates for each commit to master + CI build.

Shouldn't it have a different name if it's not boost_1_63_0.tar.bz2

...

but some completely different set of sources (which currently don't even build)?

The Fedora Linux packages have been using that URL, for several years by the look of things. So I have no idea what arbitrary collection of changes we've been shipping instead of a proper release.

OK, I've found https://sourceforge.net/projects/boost/files/boost/ snapshots/master/ which explains they're snapshots.

I still think having those tarballs called "boost_1_63_0" is insane, when it's not that release at all. Can't you append "-snapshot" or a date to the name or something?

I might be able to add "snapshot" to the name. But in a previous release it was decided that the names should be the same as the final package. As the step for doing the actual release is to just "cp" the files. But I can change the packaging to that the internal dir name is correct, but the archive file name is different. What I can't do is put a date or other varying tag. As it would make it impossible to use bintray. But I would hope this shadow download issue is not a problem in bintray. And I also think having

http://downloads.sourceforge.net/boost/boost_1_63_0.tar.bz2 redirect to a snapshot is rather sneaky, as there's no indication you're using a snapshot not a release. Has it always been like that, or did it change with http://lists.boost.org/Archives/boost/2016/08/230528.php ?

I have no idea why it does that. We explicit set the default files to download to be *only* the released files. The fact that SF is happily shoving over some random file for something that doesn't exists is insane. If it's at all possible the Everyone should be using links like this: < https://downloads.sourceforge.net/project/boost/boost/1.63.0/boost_1_63_0.ta...

That specify the full path to the file. If Linux distros are not doing that.. Shame on them. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

On 27 January 2017 at 20:44, Andrey Semashev wrote:

On 01/27/17 21:58, Rene Rivera wrote:

...

If it's at all possible the Everyone should be using links like this:

<

https://downloads.sourceforge.net/project/boost/boost/1.63.0/boost_1_63_0.ta...

...

That specify the full path to the file. If Linux distros are not doing that.. Shame on them.

The Fedora RPM spec file was changed to use the redirecting URL years ago, long before I took over maintenance of the package. It didn't occur to me to verify it (since it was definitely a sourceforge.net URL and for the boost project, and it seems that until the CI snapshots last summer it *was* getting the correct file). If we'd been using the snapshot URL it would have been obvious it was not what we wanted, so it seems the fault is with SourceForge for redirect an URL that doesn't imply "snapshot" to a snapshot file. I've updated the RPM spec file to use the full path now.

I'll add that the download links in the official Release notes [1] seem to be correct:

https://sourceforge.net/projects/boost/files/boost/1.63.0/boost_1_63_0.tar.b...

SHA checksums are given there as well to make sure you're dowloading what you want to download.

[1]: http://www.boost.org/users/history/version_1_63_0.html

Yes, that's how I noticed what I had wasn't right.

On 01/28/2017 10:26 AM, Olaf van der Spek wrote:

Doesn't the hash get verified, automatically, after downloading?

At least for SUSE, it would be if you guys actually signed your releases with well known keyring. As-is, Open Build Service will just re-download the entire file and compare it to what it has. - Adam

On Tue, Feb 7, 2017 at 11:12 PM, Jonathan Wakely wrote:

On 28 January 2017 at 09:26, Olaf van der Spek wrote:

...

On Sat, Jan 28, 2017 at 3:40 AM, Jonathan Wakely wrote:

...

The Fedora RPM spec file was changed to use the redirecting URL years ago, long before I took over maintenance of the package. It didn't occur to me to verify it (since it was definitely a sourceforge.net URL and for the boost project, and it seems that until the CI snapshots last summer it *was* getting the correct file).

Doesn't the hash get verified, automatically, after downloading?

No, because you don't download the file every time you build the RPM. That would be a problem if the upstream went offline, for example.

Instead the source tarball is downloaded once when updating the package to a new version (which I did using the problematic URL in the Subject) and then stored on Fedora's servers, and in future is pulled from there when building an SRPM (at least using the standard packaging workflow).

Even if you trust Fedora infrastructure (and thus don't check the hash when the archive is downloaded from there), the hash should still have been verified when the archive was first downloaded from SourceForge. At that point updating the Fedora servers should have failed.

On 9 Feb 2017 11:54, "Jonathan Wakely via Boost" wrote: Checking the hash is a manual process that should be done by the maintainer, it can't cause updating the Fedora servers to fail (the infrastructure can't check the hash because it doesn't know what to compare it to). I screwed that up for the first cycle of rebuilds I did for Boost 1.63.0. If you want the download info in a machine readable format, let me know. For example, I wrote a little script to generate a csv file: http://beta.boost.org/doc/downloads.csv.php

On 9 February 2017 at 15:14, Neil Groves via Boost wrote:

The actions required from this, as I see them are: 1. eliminate the forwarding of the URL to snapshots. This may require altering the filename of the archive based on earlier discussions about the source forge forwarding logic. Perhaps include snapshot and date in the archive name?

Someone (Rene IIRC) has previously stated that putting a date in the filename would be problematic, as the CI scripts would need to keep looking for a different filename (and the date the CI jobs run might not be the same day as the last snapshot was made). Just putting "snapshot" in the name would be sufficient to show it's not the release, and maintain a stable name for the CI processes to look for.

2. produce a well-known download location for the hash, so that our release archives are able to be validated by scripts.

While that might be useful in general, it wouldn't be for me, so don't do it on my behalf.

On 9 February 2017 at 15:31, Rene Rivera via Boost wrote:

It could be helpful for the release managers to have the hashes generated automatically as part of the CI build (less manual work is good). If someone wants to give that a try the script above is where it would go :-)

FWIW you can get the snapshot hashes from the bintray API. They're checked when updating the website.

On Thu, Feb 9, 2017 at 9:31 AM, Rene Rivera wrote:

On Thu, Feb 9, 2017 at 7:21 AM, Jonathan Wakely via Boost < boost@lists.boost.org> wrote:

...

On 9 February 2017 at 15:14, Neil Groves via Boost wrote:

...

The actions required from this, as I see them are: 1. eliminate the forwarding of the URL to snapshots. This may require altering the filename of the archive based on earlier discussions about the source forge forwarding logic. Perhaps include snapshot and date in the archive name?

Someone (Rene IIRC) has previously stated that putting a date in the filename would be problematic, as the CI scripts would need to keep looking for a different filename (and the date the CI jobs run might not be the same day as the last snapshot was made). Just putting "snapshot" in the name would be sufficient to show it's not the release, and maintain a stable name for the CI processes to look for.

I'm traveling, and busy, for work this week so I haven't had a chance to look at making those changes. Earliest I can get to it is this weekend. If someone else wants to give it a shot the CI release build script is here: https://github.com/boostorg/release-tools/blob/develop/ci_boost_release.py

That's now done. Ci generated archives now have "-snapshot" as a tag in the file name. -- -- Rene Rivera -- Grafik - Don't Assume Anything -- Robot Dreams - http://robot-dreams.net -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail

On 9 February 2017 at 14:21, Daniel James via Boost wrote:

On 9 Feb 2017 11:54, "Jonathan Wakely via Boost" wrote:

Checking the hash is a manual process that should be done by the maintainer, it can't cause updating the Fedora servers to fail (the infrastructure can't check the hash because it doesn't know what to compare it to). I screwed that up for the first cycle of rebuilds I did for Boost 1.63.0.

If you want the download info in a machine readable format, let me know. For example, I wrote a little script to generate a csv file:

http://beta.boost.org/doc/downloads.csv.php

Thanks, but that's not necessary for my purposes. http://www.boost.org/users/history/version_1_63_0.html has the info I need (I have to look there anyway to see which new libraries there are, and for any breaking changes to existing libs). The problem was simply that I wasn't using the URL and hash on that page, because I was mistakenly using the redirecting URL we had in the RPM spec file, and I didn't check the hash until later. I don't need help verifying the hash (that will always be at least a semi-manual process, and if I forget to do it then I forget to do it). And after wasting my own time so badly (and updating the URL in our spec file) I'm unlikely to make this mistake again. I just think having snapshots with the same filename is the release is bonkers, and that's the only thing I'd suggest changing.

On Thu, Feb 09, 2017 at 11:53:30AM +0000, Jonathan Wakely via Boost wrote:

But this is not the main point. Having archives called boost_1_63_0.tar.bz2 that are something completely different to boost 1.63.0 is just wrong. That should be self-evident. Putting "snapshot" in the name would avoid any confusion (and would not require generating a new name every day, which Rene has said would be difficult).

I'd like to echo this request to name CI builds differently. I help maintain Boost for Debian and recently I made the same mistake of downloading the wrong file from a sourceforge URL reflector because it was the same filename. I very nearly released a CI build into Debian. -Steve