Storage

Introduction
How the storage is laid out
Storing a private file
Serving a public file
Sending a private file after a check
Working in a scratch folder
Running one process at a time
Choosing where the storage lives
Moving files from an older layout
Reference
Advanced
See also

Introduction

Every feature gets a folder of its own for the files it keeps, such as uploads, generated images or exports. You ask for it with feature_storage(__name__) and never build a path by hand. In production the whole storage is one volume, so your files survive a redeploy.

import os

from werkzeug.utils import secure_filename

from splent_framework.storage import feature_storage

files = feature_storage(__name__)


def save_attachment(upload):
    name = secure_filename(upload.filename)
    upload.save(os.path.join(files.ensure("attachments"), name))
    return name

The file lands in storage/notes/private/attachments/, a folder only your feature writes to.

How the storage is laid out

storage/                  SPLENT_STORAGE_DIR, else the instance folder's storage/
├── notes/
│   ├── public/           served at /files/notes/...
│   ├── private/          never served, you send a file after your own check
│   ├── state/            locks, and anything the feature keeps for itself
│   └── tmp/              scratch folders, removed after use
└── reports/
    └── private/

feature_storage(__name__) reads the feature’s name from the module, so splent_io.splent_feature_reports.services gets reports. You can pass the short name too, feature_storage("reports").

Nothing is created until you write. A volume that is not mounted then never passes for an empty folder.

Storing a private file

path = files.path("attachments", name)     # storage/notes/private/attachments/<name>
folder = files.ensure("attachments")       # the same folder, created if missing

path gives a path in the private folder, and ensure creates the folder first. Both raise ValueError for a path that leaves the feature’s folder, such as ../../secret, so a name taken from a request cannot escape.

Serving a public file

with open(os.path.join(files.ensure(public=True), "logo.png"), "wb") as out:
    out.write(image_bytes)

files.url("logo.png")    # "/files/notes/logo.png"

Pass public=True to path or ensure to work in the public folder. url gives the address the framework serves the file at, /files/<feature>/<path>. In a command or a task, with no request around, it gives that same path from the site’s root.

The framework answers /files/... with the file, X-Content-Type-Options: nosniff and an hour of caching. Anything outside the public folder answers 404, private files included.

Sending a private file after a check

from flask import abort
from flask_login import current_user, login_required

from splent_framework.services.file_access import send_protected_file


@notes_bp.route("/notes/<int:note_id>/attachment")
@login_required
def attachment(note_id):
    note = notes_service.get_or_404(note_id)
    if note.owner_id != current_user.id:
        abort(403)
    return send_protected_file(
        files.path("attachments", note.attachment_name),
        files.private_dir,
        mimetype=note.attachment_type,
        download_name=note.attachment_original_name,
    )

You decide who may read the file, then send_protected_file sends it. It answers 404 for a path outside the folder you give it, marks the answer private, no-store, and shows only PDFs and images inline. Protected files has the details.

Working in a scratch folder

with files.temp("import-") as folder:
    archive.extractall(folder)
    build_from(folder)
# the folder is gone here, whatever happened inside

temp makes a folder under tmp/ and removes it when the block ends, even after an error. It sits on the same volume as your files, so moving a finished file from it into private is a rename.

A process that dies inside the block leaves its folder behind. Sweep those now and then, from a task for example.

removed = files.sweep_temporaries(older_than=24 * 3600)

Running one process at a time

with files.lock("reindex"):
    rebuild_index()

lock holds a file lock in state/ while the block runs. Every worker and container that mounts the same storage waits for it.

To give up at once instead of waiting, pass blocking=False. You get a BlockingIOError when someone else holds the lock.

try:
    with files.lock("reindex", blocking=False):
        rebuild_index()
except BlockingIOError:
    flash(_("The index is already being rebuilt."), "info")

A lock name is lowercase letters, digits and underscores.

Choosing where the storage lives

The storage is SPLENT_STORAGE_DIR, read from the app’s config or from the environment. Without it, it is the storage folder inside the app’s instance folder.

The production files that product:create writes set SPLENT_STORAGE_DIR=/workspace/<product>/storage and mount one named volume there.

The app’s own Flask command line can tell you where it is and which features have a folder.

flask storage where

Moving files from an older layout

A feature that kept its files somewhere else, such as a folder several features shared, says where they were from init_feature.

import os

from splent_framework.storage import register_storage_move


def init_feature(app):
    old = os.path.join(app.instance_path, "protected_uploads", "notes")
    register_storage_move("notes", old)

The product’s start runs splent storage:migrate after the database migrations. It moves every entry of the old folder into the feature’s private folder, once. What moved is gone from the old folder, so the next start finds nothing to do.

A file that is already in the target stays where it was and is reported. Nothing is ever overwritten.

You can shape the move with a few options.

register_storage_move(
    "notes",
    lambda app: os.path.join(app.static_folder, "uploads"),
    public=True,
    into="covers",
    only=lambda name: name.startswith("note-"),
)

Option	Means
`source`	A path, or a function `(app) -> path`
`public`	Move into the public folder instead of the private one
`into`	A subfolder of that folder
`only`	`(name) -> bool`, the entries that are yours when several features shared the source

To see what would move without moving anything, run splent storage:migrate --dry-run.

Reference

Call	Gives
`feature_storage(name_or_module)`	The `FeatureStorage` of a feature
`files.path(*parts, public=False)`	A path in the private or public folder
`files.ensure(*parts, public=False)`	That folder, created if missing
`files.url(*parts)`	The address of a public file
`files.temp(prefix="")`	A scratch folder for a `with` block
`files.sweep_temporaries(older_than)`	Removes scratch entries older than that many seconds, and answers how many
`files.lock(name, blocking=True)`	A lock for a `with` block
`files.root`, `files.private_dir`, `files.public_dir`, `files.state_dir`	The feature’s folders
`storage_root(app=None)`	The storage itself
`register_storage_move(feature, source, *, public=False, into="", only=None)`	Where a feature’s files were
`migrate_storage(app, dry_run=False)`	Moves them, and answers one report per move

Setting	Default
`SPLENT_STORAGE_DIR`	The `storage` folder of the app’s instance folder

Advanced

With PROTECTED_FILES_ACCEL_PREFIX set, send_protected_file hands a file of the storage to nginx as <prefix>/<feature>/private/.... Point that internal location at the storage’s root.

storage_root() raises RuntimeError outside any app when SPLENT_STORAGE_DIR is not set, since there is no instance folder to fall back to.