upsies.utils.http

HTTP methods with caching

Module Attributes

upsies.utils.http.cache_directory = None

Where to store cached responses

If this is set to a falsy value, default to DEFAULT_CACHE_DIRECTORY.

Functions

upsies.utils.http.clear_session_cookies(domain=None)[source]

Delete all session cookies for domain

If domain is None, delete all session cookies.

async upsies.utils.http.download(url, filepath, *args, cache=False, **kwargs)[source]

Write downloaded data to file

Parameters:

  • url – Where to download the data from

  • filepath – Where to save the downloaded data

Any other arguments are passed to get().

If filepath exists, no request is made.

Raises:

RequestError – if anything goes wrong

Returns:

filepath

async upsies.utils.http.get(url, *, headers={}, params={}, auth=None, cache=False, max_cache_age=inf, user_agent=False, follow_redirects=True, raise_on_error_status=True, verify=True, timeout=180, cookies=None, debug_file=None)[source]

Perform HTTP GET request

Parameters:

  • url (str) – URL to request

  • headers (dict) – Custom headers (added to default headers)

  • params (dict) – Key value pairs passed in the URL, e.g. {"k": "v"} → http://host/get?k=v

  • auth – Basic access authentication; sequence of <username> and <password> or None

  • cache (bool) – Whether to use cached response if available

  • max_cache_age (int,float) – Maximum age of cache in seconds

  • user_agent – Custom User-Agent header (str), True to send our User-Agent (“upsies/<VERSION>”), the literal string BROWSER to send a browser User-Agent, or False to not send any User-Agent

  • follow_redirects (bool) – Whether to follow redirects

  • raise_on_error_status (bool) – Whether to raise RequestError if the HTTP status code indicates an error (e.g. 404)

  • verify (bool) – Whether to verify TLS connection or ignore any errors like expired certificate

  • timeout (int,float) – Maximum number of seconds the request may take

  • cookies –

    Cookies to include in the request (merged with existing cookies in the global client session)

    • If cookies is a dict, forget them after the request.

    • If cookies is a str, load cookies from that path (if it exists), perform the request, and save the new set of cookies to the same path. If this is a relative path, cache_directory is prepended.

    Note

    Session cookies are handled separately and automatically.

Returns:

Response text

Return type:

Response

Raises:

RequestError – if the request fails for any expected reason

Return commonly used user agent

The most recent user agent is cached for USER_AGENT_CACHE_MAX_AGE seconds.

If no user agent can be acquired, return DEFAULT_USER_AGENT.

async upsies.utils.http.post(url, *, headers={}, params={}, data={}, files={}, auth=None, cache=False, max_cache_age=inf, user_agent=False, follow_redirects=True, raise_on_error_status=True, verify=True, timeout=180, cookies=None, debug_file=None)[source]

Perform HTTP POST request

Parameters:

  • url (str) – URL to request

  • headers (dict) – Custom headers (added to default headers)

  • params (dict) – Key value pairs passed in the URL, e.g. {"k": "v"} → http://host/get?k=v

  • data (dict) –

    Data to send as application/x-www-form-urlencoded

    Note

    None values are not included in the POST request.

  • files (dict) –

    Files to send as multipart/form-data as a dictionary that maps field names to file paths or dictionaries. For dictionaries, these keys are used:

    file

    File path or file-like object, e.g. return value of open() or BytesIO.

    filename (optional)

    Name of the file that is reported to the server. Defaults to basename() of file if file is a string, None otherwise.

    mimetype (optional)

    MIME type of the file content. If not provided and file is a string, guess based on extension of file. If None, do not send any MIME type.

    Examples:

    >>> files = {
>>>     "image": "path/to/foo.jpg",
>>>     "document": {
>>>         "file": "path/to/bar",
>>>         "filename": "something.txt",
>>>         "mimetype": "text/plain",
>>>     },
>>>     "data": {
>>>         "file": io.BytesIO(b"binary data"),
>>>         "mimetype": "application/octet-stream",
>>>      },
>>>     "more_data": io.BytesIO(b"more binary data"),
>>> }

  • auth – Basic access authentication; sequence of <username> and <password> or None

  • cache (bool) – Whether to use cached response if available

  • max_cache_age (int,float) – Maximum age of cache in seconds

  • user_agent – Custom User-Agent header (str), True to send our User-Agent (“upsies/<VERSION>”), the literal string BROWSER to send a browser User-Agent, or False to not send any User-Agent

  • follow_redirects (bool) – Whether to follow redirects

  • raise_on_error_status (bool) – Whether to raise RequestError if the HTTP status code indicates an error (e.g. 404)

  • verify (bool) – Whether to verify TLS connection or ignore any errors like expired certificate

  • timeout (int,float) – Maximum number of seconds the request may take

  • cookies – Cookies to include in the request (merged with existing cookies in the global client session); see get()

Returns:

Response text

Return type:

Response

Raises:

RequestError – if the request fails for any expected reason

Classes

class upsies.utils.http.Response(text, bytes, url=None, headers=None, status_code=None)[source]

Bases: str

Response to an HTTP request

This is a subclass of str with additional attributes and methods.

property url

URL the response came from

property headers

HTTP headers

property status_code

HTTP status code

property bytes

Response data as bytes

json()[source]

Parse the response text as JSON

Raises:

RequestError – if parsing fails