Skip to content

urllib.request.getproxies() misparses Windows registry proxy settings #86793

Closed
Closed
urllib.request.getproxies() misparses Windows registry proxy settings#86793
Labels
3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifeOS-windowsstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@benrg
benrg
mannequin
opened on Dec 12, 2020
BPO 42627
Nosy @pfmoore, @tjguk, @zware, @zooba, @corona10, @chrisxiao, @NateScarlet, @CrazyBoyFeng, @keuin
PRs
  • bpo-42627: Fix wrong parsing of Windows registry proxy settings #26307
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-12-12.20:27:52.875>
labels = ['type-bug', '3.9', '3.10', '3.11', 'library', 'OS-windows']
title = 'urllib.request.getproxies() misparses Windows registry proxy settings'
updated_at = <Date 2021-11-20.14:38:47.145>
user = 'https://bugs.python.org/benrg'

    bugs.python.org fields:

    activity = <Date 2021-11-20.14:38:47.145>
actor = 'CrazyBoyFeng'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'Windows']
creation = <Date 2020-12-12.20:27:52.875>
creator = 'benrg'
dependencies = []
files = []
hgrepos = []
issue_num = 42627
keywords = ['patch']
message_count = 10.0
messages = ['382921', '387468', '388149', '388152', '393927', '394668', '395276', '401215', '401995', '406658']
nosy_count = 11.0
nosy_names = ['paul.moore', 'tim.golden', 'benrg', 'zach.ware', 'steve.dower', 'corona10', 'chrisxiao', 'kotori', 'NateScarlet', 'CrazyBoyFeng', 'keuin']
pr_nums = ['26307']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42627'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

    Activity

    1. benrg commented on Dec 12, 2020

      @benrg
      benrgmannequin
      on Dec 12, 2020
      MannequinAuthor

      If HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyServer contains the string http=host:123;https=host:456;ftp=host:789, then getproxies_registry() should return

      {'http': 'http://host:123', 'https': 'http://host:456', 'ftp': 'http://host:789'}

      for consistency with WinInet and Chromium, but it actually returns

      {'http': 'http://host:123', 'https': 'https://host:456', 'ftp': 'ftp://host:789'}

      This bug has existed for a very long time (since Python 2.0.1 if not earlier), but it was exposed recently when urllib3 added support for HTTPS-in-HTTPS proxies in version 1.26. Before that, an https prefix on the HTTPS proxy url was silently treated as http, accidentally resulting in the correct behavior.

      There are additional bugs in the treatment of single-proxy strings (the case when the string contains no = character).

      The Chromium code for parsing the ProxyServer string can be found here: https://source.chromium.org/chromium/chromium/src/+/refs/tags/89.0.4353.1:net/proxy_resolution/proxy_config.cc;l=86

      Below is my attempt at modifying the code from getproxies_registry to approximately match Chromium's behavior. I could turn this into a patch, but I'd like feedback on the corner cases first.

          if '=' not in proxyServer and ';' not in proxyServer:
        # Use one setting for all protocols.
        # Chromium treats this as a separate category, and some software
        # uses the ALL_PROXY environment variable for a similar purpose,
        # so arguably this should be 'all={}'.format(proxyServer),
        # but this is more backward compatible.
        proxyServer = 'http={0};https={0};ftp={0}'.format(proxyServer)

    for p in proxyServer.split(';'):
        # Chromium and WinInet are inconsistent in their treatment of
        # invalid strings with the wrong number of = characters. It
        # probably doesn't matter.
        protocol, addresses = p.split('=', 1)
        protocol = protocol.strip()

        # Chromium supports more than one proxy per protocol. I don't
        # know how many clients support the same, but handling it is at
        # least no worse than leaving the commas uninterpreted.
        for address in addresses.split(','):
            if protocol in {'http', 'https', 'ftp', 'socks'}:
                # See if address has a type:// prefix
                if not re.match('(?:[^/:]+)://', address):
                    if protocol == 'socks':
                        # Chromium notes that the correct protocol here
                        # is SOCKS4, but "socks://" is interpreted
                        # as SOCKS5 elsewhere. I don't know whether
                        # prepending socks4:// here would break code.
                        address = 'socks://' + address
                    else:
                        address = 'http://' + address

            # A string like 'http=foo;http=bar' will produce a
            # comma-separated list, while previously 'bar' would
            # override 'foo'. That could potentially break something.
            if protocol not in proxies:
                proxies[protocol] = address
            else:
                proxies[protocol] += ',' + address
    2. benrg
      added
      OS-windows
      stdlibStandard Library Python modules in the Lib/ directory
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      3.10 (EOL)end of life
      3.9 (EOL)end of life
      type-bugAn unexpected behavior, bug, or error
      on Dec 12, 2020

    3. kotori commented on Feb 21, 2021

      @kotori
      kotorimannequin
      on Feb 21, 2021
      Mannequin

      I came across this issue as well.
      I checked Microsoft documentations and it seems InternetGetProxyInfo in WinInet is deprecated, while WinHttpGetIEProxyConfigForCurrentUser in WinHTTP will return the exact same string as what it stored registery.

      https://docs.microsoft.com/en-US/troubleshoot/windows-client/networking/configure-client-proxy-server-settings-by-registry-file
      Also from this documentation, a proxy server could have "http://" prefix, so I guess it could also support "https://" prefix if a user set a https proxy.

    4. kotori
      added
      OS-windows
      stdlibStandard Library Python modules in the Lib/ directory
      and removed
      OS-windows
      stdlibStandard Library Python modules in the Lib/ directory
      on Feb 21, 2021

    5. CrazyBoyFeng commented on Mar 5, 2021

      @CrazyBoyFeng
      CrazyBoyFengmannequin
      on Mar 5, 2021
      Mannequin

      We know Windows reslove:
      http=host:port;https=host:port;ftp=host:port;socks=host:port
      as:
      http=http://host:port;https=http://host:port;ftp=http://host:port;socks=socks://host:port
      means:
      Using HTTP type proxy for HTTP, HTTPS and FTP requests, but Socks4/4a type proxy for the other TCP requests.
      We notice that socks are different from the others.

      Now I want to know what if Windows slove this:
      http=socks://host:port;ftp=https://host:port;socks=https://host:port
      Does it mean using Socks4/4a type proxy for HTTP requests, HTTPS type proxy for FTP requests, and HTTP type proxy for the other TCP requests?
      Or just invalid settings?

    6. CrazyBoyFeng commented on Mar 5, 2021

      @CrazyBoyFeng
      CrazyBoyFengmannequin
      on Mar 5, 2021
      Mannequin

      I make some black box tests with the HTTPS proxy.
      Set system proxy http=https://host:port and start a HTTPS proxy, then IE, Edge (Chromium) and benrg's code (using urllib3) work fine while fetching http://website. Then I shutdown the HTTPS proxy and start a HTTP proxy on the same port, I get SSLError. That's it.

      benrg should add your patch as soon as possible.
      Too few people actually use HTTPS proxy. Even if there is a bug with HTTPS proxy, it will not be too late to fix it when it is discovered.

    7. 4 remaining items

    8. zooba commented on Sep 16, 2021

      @zooba
      zooba
      on Sep 16, 2021
      Member

      I think the PR is basically ready, unfortunately it's stuck behind a CI issue we only just fixed, and needs to merge from main before it'll clear.

      Posting here once CI is green will get attention faster than on GitHub.

    9. zooba
      added
      3.11only security fixes
      and removed
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      on Sep 16, 2021

    10. CrazyBoyFeng commented on Nov 20, 2021

      @CrazyBoyFeng
      CrazyBoyFengmannequin
      on Nov 20, 2021
      Mannequin

      Sorry I didn't see this comment before.

      Can it be merged now?

    11. ezio-melotti
      transferred this issue fromon Apr 10, 2022

    12. SekiBetu commented on Apr 28, 2022

      @SekiBetu
      SekiBetu
      on Apr 28, 2022

      I think the PR is basically ready, unfortunately it's stuck behind a CI issue we only just fixed, and needs to merge from main before it'll clear.

      Posting here once CI is green will get attention faster than on GitHub.

      any updates?

    13. zooba commented on Apr 28, 2022

      @zooba
      zooba
      on Apr 28, 2022
      Member

      I retriggered the CI to see if it passes now.

    14. TTimo commented on May 10, 2022

      @TTimo
      TTimo
      on May 10, 2022

      Hello,

      I am curious if this will also address https://stackoverflow.com/questions/65931275/python-requests-module-does-it-use-system-level-on-windows-proxy-settings or if it's yet another problem?

    15. zooba
      closed this as completedon May 12, 2022

    16. zooba commented on May 12, 2022

      @zooba
      zooba
      on May 12, 2022
      Member

      @TTimo From a quick glance, I think it'll deal with part of the second half of that question. It should be able to pick up manually customised proxies, but I don't think it'll get automatically configured ones. And anything that's bypassing urllib here won't get any benefit, naturally.

    17. coletdjnz
      added a commit that references this issue on Jul 1, 2022

      Implement workaround for python/cpython#86793

      5d13088
    18. coletdjnz
      added a commit that references this issue on May 27, 2023

      [core] Workaround erroneous urllib Windows proxy parsing (#7092)

      3f66b6f
    19. aalsuwaidi
      added a commit that references this issue on Apr 21, 2024

      [core] Workaround erroneous urllib Windows proxy parsing (yt-dlp#7092)

      bf07509
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.11only security fixes3.9 (EOL)end of lifeOS-windowsstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions