If Content-Length header is missing, look for a custom header named X-Download-Size. This is to allow servers that send chunked transfer encoding to specify download size, as Content-Length is prohibited for such transfers per RFC2616 section 4.4. An example of a server that sends downloads in such a way is App Engine Blobstore. So this change fixes the progress bar for Simian clients, as well as anyone else that may wish to use chunked transfer encoding for large downloads.

2026-04-23 04:59:17 -05:00 · 2011-12-15 10:40:03 -05:00
parent cb435998be
commit 19b4a0bbc7
1 changed files with 9 additions and 1 deletions
@@ -2468,7 +2468,15 @@ def curl(url, destinationpath, onlyifnewer=False, etag=None, resume=False,
                # we got an empty line; end of headers (or curl exited)
                donewithheaders = True
                try:
-                    targetsize = int(header.get('content-length'))
+                    # Prefer Content-Length header to determine download size,
+                    # otherwise fall back to a custom X-Download-Size header.
+                    # This is primary for servers that use chunked transfer
+                    # encoding, when Content-Length is forbidden by RFC2616 4.4.
+                    # An example of such a server is App Engine Blobstore.
+                    targetsize = (
+                        header.get('content-length') or
+                        header.get('x-download-size'))
+                    targetsize = int(targetsize)
                except (ValueError, TypeError):
                    targetsize = 0
                if header.get('http_result_code') == '206':