Skip to content

Increase the HTTP requests timeout to 30 in fetch pipes #1144 #1567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tdruez merged 1 commit into from
Jan 27, 2025
Merged

Increase the HTTP requests timeout to 30 in fetch pipes #1144 #1567

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions scanpipe/pipes/fetch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,17 @@

Download = namedtuple("Download", "uri directory filename path size sha1 md5")

# Time (in seconds) to wait for the server to send data before giving up.
# The ``REQUEST_CONNECTION_TIMEOUT`` defines:
# - Connect timeout: The maximum time to wait for the client to establish a connection
# to the server.
# - Read timeout: The maximum time to wait for a server response once the connection
# is established.
# Notes: Use caution when lowering this value, as some servers
# (e.g., https://cdn.kernel.org/) may take longer to respond to HTTP requests under
# certain conditions.
HTTP_REQUEST_TIMEOUT = 30


def run_command_safely(command_args):
"""
Expand Down Expand Up @@ -107,7 +118,7 @@ def fetch_http(uri, to=None):
path.
"""
request_session = get_request_session(uri)
response = request_session.get(uri, timeout=5)
response = request_session.get(uri, timeout=HTTP_REQUEST_TIMEOUT)

if response.status_code != 200:
raise requests.RequestException
Expand Down Expand Up @@ -416,7 +427,7 @@ def check_urls_availability(urls):

request_session = get_request_session(url)
try:
response = request_session.head(url, timeout=5)
response = request_session.head(url, timeout=HTTP_REQUEST_TIMEOUT)
response.raise_for_status()
except requests.exceptions.RequestException:
errors.append(url)
Expand Down