gws.lib.net

URL utilities and an HTTP client.

URLs:

parse_url splits a URL into a Url object (scheme, host, port, path, query parameters and so on), make_url builds a URL string from such an object or a dict. make_qs, add_params, extract_params and the quote_* functions handle query strings and URL quoting.

HTTP:

http_request sends a request with requests, using one session per process, so that connections to the same host are reused. It does not raise on failures; instead, it returns an HTTPResponse with ok set to False. Connection errors, timeouts and other request errors get the pseudo status codes 900, 901 and 999. HTTPResponse.raise_if_failed turns a failed response into an HTTPError subclass. The response text is decoded with the charset from the content type header, or as UTF-8 or ISO-8859-1 if there is none.

Example:

u = gws.lib.net.parse_url('https://example.com/wms?SERVICE=WMS')
url = gws.lib.net.add_params(u.url, REQUEST='GetCapabilities')

res = gws.lib.net.http_request(url, timeout=10)
res.raise_if_failed()
xml = res.text

Source code: gws.lib.net

Package Contents

gws.lib.net.add_params(url: str, params: dict = None, **kwargs) → str

Add query parameters to a URL.

Existing parameters with the same names are replaced.

Parameters:

  • url – A URL.

  • params – Parameters to add.

  • **kwargs – More parameters to add.

Returns:

The new URL.

exception gws.lib.net.ConnectionError

Bases: HTTPError

HTTP connection failed.

exception gws.lib.net.Error

Bases: gws.Error

Network error.

gws.lib.net.extract_params(url: str) → tuple[str, dict]

Split the query parameters off a URL.

Parameters:

url – A URL.

Returns:

The URL without the query string, and the query parameters.

exception gws.lib.net.GenericError

Bases: HTTPError

HTTP request failed for another reason.

gws.lib.net.http_request(url, **kwargs) → HTTPResponse

Send an HTTP request.

Failures are logged and returned as a response with ok set to False, they do not raise. By default, the connect and read timeouts are 60 seconds, certificates are verified with certifi, and a GBD WebSuite User-Agent header is sent.

Parameters:

  • url – Request URL.

  • **kwargs – Options for requests.Session.request. method is the HTTP method (GET by default), params are added to the URL, a numeric timeout applies to both connect and read.

Returns:

The response.

exception gws.lib.net.HTTPError

Bases: Error

HTTP request failed.

class gws.lib.net.HTTPResponse(ok: bool, url: str, res: requests.Response = None, text: str = None, status_code=0)

Response of http_request.

ok

True if the request succeeded with a 2xx status.

url

Request URL.

content

Response body.

content_type

Content type, without parameters.

content_encoding

Charset from the content type header, or None.

status_code

HTTP status code, or a pseudo status code for requests that failed without a response.

property text: str

Response body as text, decoded on first access.

Returns:

The decoded body.

ok
url
raise_if_failed()

Raise an error if the request failed.

Raises:
gws.lib.net.is_abs_url(url)

Check if a URL has a host part, i.e. starts with // or a scheme and //.

Parameters:

url – A URL.

Returns:

A truthy match object if the URL is absolute, None otherwise.

gws.lib.net.make_qs(x) → str

Convert a dict or a list of pairs to a query string.

Values are encoded as UTF-8 and quoted. Booleans become true and false, lists and other iterables are joined with commas, other values are converted with str.

Parameters:

x – A dict, an object that can be converted to a dict, or a list of (name, value) pairs.

Returns:

The query string, without ?.

gws.lib.net.make_relative_url(path: str, params: dict = None, **kwargs) → str

Build a URL from a path and query parameters, without a scheme and host.

Parameters:

  • path – URL path; a leading slash is added if needed.

  • params – Query parameters.

  • **kwargs – More query parameters.

Returns:

The URL.

gws.lib.net.make_url(u: Url | dict | None = None, **kwargs) → str

Build a URL string.

Uses scheme, hostname, port, username, password, path, params and fragment; other keys, like query or url, are ignored. The port is omitted if it is the default port of the scheme.

Parameters:

  • u – URL components, a Url object or a dict.

  • **kwargs – Components that override the values from u.

Returns:

The URL.

gws.lib.net.parse_qs(x) → dict

Parse a query string.

Parameters:

x – Query string.

Returns:

A dict that maps each parameter name to a list of its values.

gws.lib.net.parse_url(url: str, **kwargs) → Url

Parse a URL.

A URL without // is treated as starting with a host name.

Parameters:

  • url – URL string.

  • **kwargs – Values to set on the Url object after parsing.

Returns:

A Url object.

gws.lib.net.quote_param(s: str) → str

Quote a string for use in a URL, including slashes.

Parameters:

s – A string.

Returns:

The quoted string.

gws.lib.net.quote_path(s: str) → str

Quote a URL path, leaving slashes as they are.

Parameters:

s – A path.

Returns:

The quoted path.

exception gws.lib.net.Timeout

Bases: HTTPError

HTTP request timed out.

gws.lib.net.unquote(s: str) → str

Unquote a URL-quoted string.

Parameters:

s – A quoted string.

Returns:

The unquoted string.

class gws.lib.net.Url(*args, **kwargs)

Bases: gws.Data

Components of a URL, as returned by parse_url.

fragment: str

Fragment, without #.

hostname: str

Host name.

netloc: str

Network location, including user name, password and port.

params: dict

Query parameters. For repeated parameters, the first value is used.

password: str

Password, unquoted.

path: str

Path.

pathparts: gws.lib.osx.ParsePathResult

Components of the path.

port: int

Port, 0 if not given.

qsl: list

Query parameters as a list of (name, value) pairs.

query: str

Query string, without ?.

scheme: str

Scheme, e.g. https.

url: str

The parsed URL; with a // prefix if the original URL had no scheme.

username: str

User name, unquoted.