Fetching URLs

Table of contents

1. Introduction
2. Fetching a file
3. Choosing the hosts
4. What is refused
5. Following redirects
6. Reference
7. Advanced
8. See also

---

Introduction

When your feature downloads a URL that someone typed, such as a model to import or an image for a gallery, the server makes the request. Without care, anyone who can type a URL can make your server read its own network, the database or a cloud’s metadata service. fetch reads one public address and refuses the rest.

from splent_framework.fetch import FetchError, fetch


@models_bp.route("/models/import", methods=["POST"])
def import_from_url():
    try:
        body = fetch(
            request.form.get("url", ""),
            allowed_hosts=["raw.githubusercontent.com"],
            max_bytes=5 * 1024 * 1024,
            timeout=10,
        )
    except FetchError as error:
        flash(str(error), "danger")
        return redirect(url_for("models.import_form"))
    model = models_service.save(body, content_type=body.content_type)
    return redirect(url_for("models.show", model_id=model.id))

---

Fetching a file

fetch answers the body as bytes. The answer also carries content_type, without its parameters, and url, the address it finally came from.

body = fetch(url, allowed_hosts=None, max_bytes=2_000_000, timeout=10)
body.content_type     # "image/png"
body.url              # the address after any redirect

max_bytes and timeout are required, so every fetch has a limit.

Choosing the hosts

Pass the host names you trust in allowed_hosts. A host is matched by its exact name, so raw.githubusercontent.com.evil.example does not pass.

Pass allowed_hosts=None to allow any host. The address must still be public.

What is refused

Every refusal raises FetchError with a message you can show to the person who typed the URL.

• A scheme other than http or https.
• A user name or password in the address.
• A host off the list you allowed.
• A host that resolves to an address that is not public, such as private, loopback, link-local, multicast or reserved, an IPv4 address written inside an IPv6 one included.
• An answer other than 200.
• A body larger than max_bytes, or one that takes longer than timeout.

The connection goes to the address that was checked, not to a second lookup, so a name that resolves somewhere else the next time reaches nothing new. With https the certificate is still checked against the host.

Following redirects

body = fetch(url, allowed_hosts=HOSTS, max_bytes=LIMIT, timeout=10, max_redirects=3)

By default a redirect is refused. With max_redirects, each new address passes every check again, and one redirect too many raises FetchError.

---

Reference

fetch(url, *, allowed_hosts, max_bytes, timeout, user_agent="SPLENT safe fetch", max_redirects=0)

Name	What it is
Fetched	The answer, bytes with content_type and url
FetchError	A ValueError whose message a user can read
check(url, allowed_hosts)	Runs the checks alone, without fetching, and raises FetchError

Advanced

fetch makes its connection with http.client, which reads no proxy from the environment and never follows a redirect on its own.

See also

• Background tasks, for a fetch too slow to make in a request.
• Cache, to keep what you fetched for a while.