[testing][regressions] All my regressions testers failing due to connection errors to GitHub
Hi, All my 4 testers (I have 4 machines running several Mingw targets) are always failing pulling new versions from GitHub. This problem started some days ago. I've tried to reset all machines, network equipment, disabling antivirus, firewall, modified http attributes, modified the regression script to use git protocol instead of https, etc. with no luck. I'm using Windows machines with official Git. Any clue on what's going on? Maybe Github recently does not like so many submodules? Example of failure (several libraries can fail during "git pull "--recurse-submodules""): ------------------- ------------------- ------------------- Fetching submodule libs/io fatal: unable to access 'https://github.com/boostorg/io.git/': OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054 Then: # git_source_checkout failed with message "GIT command "git pull "--recurse-submodules"" failed with code 1" # Retrying (5 more attempts). # Executing GIT command: c:\boost\develop\boost_root> git submodule foreach "git submodule deinit ." ------------------- ------------------- All retries fail and the script fails with error: ------------------- ------------------- ------------------- # git_source_checkout failed with message "GIT command "git pull "--recurse-submodules"" failed with code 1" # Giving up. Traceback (most recent call last): File "run.py", line 71, in <module> runner(root) File "c:\boost\develop\boost_regression_src\regression.py", line 286, in __init__ self.main() File "c:\boost\develop\boost_regression_src\regression.py", line 623, in main getattr(self,action_m)() File "c:\boost\develop\boost_regression_src\regression.py", line 578, in command_regression self.command_get_source() File "c:\boost\develop\boost_regression_src\regression.py", line 330, in command_get_source self.retry(self.git_source_checkout) File "c:\boost\develop\boost_regression_src\regression.py", line 662, in retry return f() File "c:\boost\develop\boost_regression_src\regression.py", line 875, in git_source_checkout self.git_checkout(git_info['boost'], self.git_branch(), clean) File "c:\boost\develop\boost_regression_src\regression.py", line 849, in git_checkout self.git_command( 'pull', '--recurse-submodules' ) File "c:\boost\develop\boost_regression_src\regression.py", line 834, in git_command raise Exception( 'GIT command "%s" failed with code %d' % (git_cli, rc) ) Exception: GIT command "git pull "--recurse-submodules"" failed with code 1 END 24/05/2018 15:28:37,46 ------------------- ------------------- ------------------- Errors are never in the same libraries, very few times the checkout succeeds and a regression is run. Anyone else having these problems? I've attached the log of one failure in case someone can see any clue. I really don't see how to solve this, so I won't be able to run regressions. Best, Ion
On Sat, May 26, 2018 at 10:49 AM, Ion Gaztañaga via Boost < boost@lists.boost.org> wrote:
Hi,
All my 4 testers (I have 4 machines running several Mingw targets) are always failing pulling new versions from GitHub. This problem started some days ago. I've tried to reset all machines, network equipment, disabling antivirus, firewall, modified http attributes, modified the regression script to use git protocol instead of https, etc. with no luck.
I'm using Windows machines with official Git. Any clue on what's going on? Maybe Github recently does not like so many submodules?
Example of failure (several libraries can fail during "git pull "--recurse-submodules""):
------------------- ------------------- -------------------
Fetching submodule libs/io fatal: unable to access 'https://github.com/boostorg/io.git/': OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 10054
Then:
# git_source_checkout failed with message "GIT command "git pull "--recurse-submodules"" failed with code 1" # Retrying (5 more attempts). # Executing GIT command: c:\boost\develop\boost_root> git submodule foreach "git submodule deinit ."
------------------- -------------------
All retries fail and the script fails with error:
------------------- ------------------- -------------------
# git_source_checkout failed with message "GIT command "git pull "--recurse-submodules"" failed with code 1" # Giving up. Traceback (most recent call last): File "run.py", line 71, in <module> runner(root) File "c:\boost\develop\boost_regression_src\regression.py", line 286, in __init__ self.main() File "c:\boost\develop\boost_regression_src\regression.py", line 623, in main getattr(self,action_m)() File "c:\boost\develop\boost_regression_src\regression.py", line 578, in command_regression self.command_get_source() File "c:\boost\develop\boost_regression_src\regression.py", line 330, in command_get_source self.retry(self.git_source_checkout) File "c:\boost\develop\boost_regression_src\regression.py", line 662, in retry return f() File "c:\boost\develop\boost_regression_src\regression.py", line 875, in git_source_checkout self.git_checkout(git_info['boost'], self.git_branch(), clean) File "c:\boost\develop\boost_regression_src\regression.py", line 849, in git_checkout self.git_command( 'pull', '--recurse-submodules' ) File "c:\boost\develop\boost_regression_src\regression.py", line 834, in git_command raise Exception( 'GIT command "%s" failed with code %d' % (git_cli, rc) ) Exception: GIT command "git pull "--recurse-submodules"" failed with code 1 END 24/05/2018 15:28:37,46
------------------- ------------------- -------------------
Errors are never in the same libraries, very few times the checkout succeeds and a regression is run. Anyone else having these problems?
I've attached the log of one failure in case someone can see any clue. I really don't see how to solve this, so I won't be able to run regressions.
Best,
Ion
I've been having horrible problems getting all the submodules from github, that look similar to what you've got going. I'm not sure where the issue is, but I see it across linux and windows boxes at my house, but not when I run in a cloud data center. I'm thinking it has something to do with my connection (AT&T), but I'm not sure. I ended up mirroring all the repos to a local machine, with a cron job to update them every 15 min) and then cloning from that for the regression runs. The script I use for that can be found here: https://github.com/teeks99/boost-build/blob/master/Regression/mirror_boostor... For more backgroun, I was seeing the same failed connections over https or ssh. I got the script failure when even one of the submoduels failed to update...which I guess is necessary. Now with my script almost all runs one of the repos fails to fetch, but since they are all just standalone repos, that failures doesn't cause an overall failure. Tom
On 26/05/2018 19:08, Tom Kent via Boost wrote:
On Sat, May 26, 2018 at 10:49 AM, Ion Gaztañaga via Boost <
I've been having horrible problems getting all the submodules from github, that look similar to what you've got going. I'm not sure where the issue is, but I see it across linux and windows boxes at my house, but not when I run in a cloud data center. I'm thinking it has something to do with my connection (AT&T), but I'm not sure.
Looks like a GitHub problem, which is the worst case ;-) Maybe it thinks it's a DoS attack or similar with so many connections.
I ended up mirroring all the repos to a local machine, with a cron job to update them every 15 min) and then cloning from that for the regression runs. The script I use for that can be found here: https://github.com/teeks99/boost-build/blob/master/Regression/mirror_boostor...
Looks like I should do something similar. Many thanks for the script.
For more backgroun, I was seeing the same failed connections over https or ssh. I got the script failure when even one of the submoduels failed to update...which I guess is necessary. Now with my script almost all runs one of the repos fails to fetch, but since they are all just standalone repos, that failures doesn't cause an overall failure.
I wonder if other GitHub projects have similar problems, but perhaps not many projects have so many submodules. I'll have a look at your workaround and try to do something, or else I won't be able to run regression tests. Many thanks! Ion
On 05/27/18 12:22, Ion Gaztañaga via Boost wrote:
On 26/05/2018 19:08, Tom Kent via Boost wrote:
On Sat, May 26, 2018 at 10:49 AM, Ion Gaztañaga via Boost <
I've been having horrible problems getting all the submodules from github, that look similar to what you've got going. I'm not sure where the issue is, but I see it across linux and windows boxes at my house, but not when I run in a cloud data center. I'm thinking it has something to do with my connection (AT&T), but I'm not sure.
Looks like a GitHub problem, which is the worst case ;-) Maybe it thinks it's a DoS attack or similar with so many connections.
I ended up mirroring all the repos to a local machine, with a cron job to update them every 15 min) and then cloning from that for the regression runs. The script I use for that can be found here: https://github.com/teeks99/boost-build/blob/master/Regression/mirror_boostor...
Looks like I should do something similar. Many thanks for the script.
For more backgroun, I was seeing the same failed connections over https or ssh. I got the script failure when even one of the submoduels failed to update...which I guess is necessary. Now with my script almost all runs one of the repos fails to fetch, but since they are all just standalone repos, that failures doesn't cause an overall failure.
I wonder if other GitHub projects have similar problems, but perhaps not many projects have so many submodules. I'll have a look at your workaround and try to do something, or else I won't be able to run regression tests.
Maybe try writing to GitHub support?
On 27/05/2018 12:37, Andrey Semashev via Boost wrote:
I wonder if other GitHub projects have similar problems, but perhaps not many projects have so many submodules. I'll have a look at your workaround and try to do something, or else I won't be able to run regression tests.
Maybe try writing to GitHub support?
Working on it. Best, Ion
participants (3)
-
Andrey Semashev -
Ion Gaztañaga -
Tom Kent