[gimp-web/wip/Jehan/fix-ci] tools: waits a few secs between download loops to prevent IP banning.
- From: Jehan <jehanp src gnome org>
- To: commits-list gnome org
- Cc:
- Subject: [gimp-web/wip/Jehan/fix-ci] tools: waits a few secs between download loops to prevent IP banning.
- Date: Mon, 2 May 2022 14:44:52 +0000 (UTC)
commit 9f7bfbcbe2410d691ec8fc07fa4a9c6d3f0a1736
Author: Jehan <jehan girinstud io>
Date: Mon May 2 16:41:46 2022 +0200
tools: waits a few secs between download loops to prevent IP banning.
I am not sure at all if this is what we are experiencing (seemingly
randomly) on some servers, but I wonder if it might be. Basically we try
to load the tarball, the Windows installer and the macOS DMG in threads,
so nearly at the same time (if there are enough slots in the thread
pool). So why do some server seem to take forever? Could it be a
fail2ban-like service which bans us temporarily when several files are
requested in a too-short interval?
tools/downloads/gimp-check-mirrors.py | 11 +++++++++++
1 file changed, 11 insertions(+)
---
diff --git a/tools/downloads/gimp-check-mirrors.py b/tools/downloads/gimp-check-mirrors.py
index 1559536c..8c3766a2 100755
--- a/tools/downloads/gimp-check-mirrors.py
+++ b/tools/downloads/gimp-check-mirrors.py
@@ -8,6 +8,7 @@ import hashlib
import re
import requests
import sys
+import time
# argparse for mirrorsfile and uri
parser = argparse.ArgumentParser(description='Check if GIMP download mirrors have a file from
download.gimp.org.')
@@ -168,6 +169,7 @@ if args.verify_checksum:
# which is fine by us.
with concurrent.futures.ThreadPoolExecutor(max_workers=None) as executor:
test_results = {}
+ first = True
for local_uri in local_uris:
test_results[local_uri] = []
@@ -175,6 +177,15 @@ with concurrent.futures.ThreadPoolExecutor(max_workers=None) as executor:
if args.verify_checksum:
origin_checksum = origin_checksums[local_uri]
+ # Some downloads seem to block forever. I wonder if maybe there is a
+ # problem of trying to download immediately several files on a same
+ # mirror and from the same IP. Maybe there is some fail2ban or other
+ # similar service which temporarily bans the test IP?
+ # Let's try to sleep on it a bit and see if it improves things.
+ if not first:
+ time.sleep(5)
+ first = False
+
with fileinput.input(files=(args.mirrorsfile), mode='r') as f:
for line in f:
if args.verify_checksum and line.strip() == 'https://download.gimp.org/pub/gimp/':
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]