Continuous delivery

A product of a line is a configuration of released features, and SPLENT delivers it as one. Every change is checked against the line, a new feature release reaches each product as a pull request that proves the new combination valid, and a product release publishes an image to a channel. Production follows that channel through the autodeploy feature, which backs up the database, swaps the container and rolls back a release that does not start.

Table of contents

• The loop
• Continuous integration
  • The features’ tests
  • The servers the features need
  • Keeping the workflows in step
• The release channel
• Automatic deployment
  • Settings
  • Turning it on
• Who decides what
• See also

---

The loop

Everything around the release is automatic. The release itself, merging the pull request and running product:release, stays a person's decision.

SPLENT and DevOps serve each other in two ways.

• DevOps for the line. On every push, continuous integration resolves the releases the product pins, checks the selection against the line’s UVL model, checks that the features’ contracts, refiners and dependencies agree, lints the product’s code and runs the features’ tests in that configuration. A new feature release arrives as a pull request with the newer combination resolved by pip, checked against the model, and the migrations it brings listed.
• The line for DevOps. Deploying itself is a feature of the line. A product that selects autodeploy gets the agent, its settings and a Deployments page in its admin panel. A product that does not is deployed by hand. The UVL decides, as it does for everything else a product has.

Continuous integration

product:create gives a product two Forgejo workflows, and product:sync-template --ci adds them to a product that lacks them. The commands that change the features keep them in step, see below.

Workflow	When	What it does
splent-validate.yml	Every push to main, every pull request	Three jobs. validate runs product:resolve --prod --strict, then product:validate, then product:build --no-image, and fails when the build changes the versioned deploy/, docker/requirements.lock, splent.contracts.json or docker/spl/. A product that fails is not a valid product of its line. lint runs splent lint --target app on the product’s own code with the Ruff the CLI pins. test runs the features’ tests in this configuration once validate passes, see below
splent-upgrade.yml	On a schedule, Mondays at 06:00 UTC unless upgrade_schedule under [tool.splent.ci] says otherwise	feature:upgrade --yes. When the pins change it pushes them to the branch splent/feature-upgrades and opens a pull request with the upgrade plan

Neither workflow releases or deploys. The features are public, so the runner clones them anonymously and over HTTPS (SPLENT_CLONE_TRANSPORT=https).

The features’ tests

A feature’s tests passed when it was released, in whatever product it was released from. The test job runs them again in this product, against the releases it pins, the refiners it selects and its own configuration, which is where a combination of features breaks.

The job stands in for the product’s web container. SPLENT_CONTAINER=product makes feature:test run in the job instead of delegating to <product>_web, and a MariaDB 11.4 service, db, holds the test database <product>_db_test. The servers the features run next to the product run here too, see below.

1. Install what the product runs. The framework and CLI versions [tool.splent.runtime] pins, as the production image installs them, with the framework’s dev extra for pytest. Third-party packages follow docker/requirements.lock through PIP_CONSTRAINT. A product with no runtime pins gets the newest releases.
2. Resolve the features, product:resolve --dev --strict, the selection feature:test reads with SPLENT_ENV=dev.
3. Install the features and their dependencies, feature:pip-install, the same path the production image takes.
4. Run the tests, feature:test at its default levels, unit, integration and functional.

A product whose features have no tests passes, since feature:test exits 0 when nothing ran. tests = false leaves the job out.

[tool.splent.ci]
tests = false

Run splent product:sync-template <product> --ci after changing it.

The servers the features need

A feature that brings a server, such as redis or elasticsearch, declares it in its docker compose file, and the code that uses it reaches it by the compose service’s name, through a URL in the feature’s docker/.env.example (REDIS_URL=redis://splent_feature_redis:6379). When the workflow is written, the CLI reads the development compose file (docker-compose.dev.yml, else docker-compose.yml) of every feature in features and features_dev, and decides for each service.

The service runs	In the test job
redis, elasticsearch or mailhog (by the image name)	A service with the compose service’s name and image, the literal values of its compose environment and a health check. The variables of the feature’s docker/.env.example that name the service go into the job’s environment, REDIS_URL and ELASTICSEARCH_URL when the file has none. For mailhog the job also gets MAIL_HOST (the service), MAIL_PORT=1025 and MAIL_USE_TLS=false, which splent_feature_mail sends through
nginx or phpmyadmin	Nothing, since no test reaches a reverse proxy or a database UI
Anything else	Nothing, and the test job is left out, see below

A feature whose compose files are all for production, such as autodeploy, runs nothing in the job. So diversolab_app and innosoft_app get splent_feature_redis with REDIS_URL, egc_wiki and isia_wiki get splent_feature_elasticsearch with ELASTICSEARCH_URL, and uvlhub_splent gets both and splent_feature_mailhog, with the mail settings that point splent_feature_mail at it.

      splent_feature_redis:
        image: redis:7.4
        options: >-
          --health-cmd "redis-cli ping" --health-interval 5s
          --health-timeout 5s --health-retries 30
    env:
      ...
      REDIS_URL: redis://splent_feature_redis:6379

A service the CLI cannot run leaves the test job out, because without the server the job would be red for no fault of the product. The workflow says so where the job would be, and product:sync-template --ci prints the same, with a line for each such service.

  The test job is left out of splent-validate.yml. splent_feature_ldap runs splent_feature_ldap (image osixia/openldap:1.5.0), a service the CI does not know how to run.
  To run the tests anyway, set tests = true under [tool.splent.ci].
  If they need splent_feature_ldap, add [tool.splent.ci.services.splent_feature_ldap] with image = "osixia/openldap:1.5.0".

A feature that is not in the workspace yet is not such a case. Its compose files cannot be read until it is there, fetched by product:resolve when it is pinned or cloned at the workspace root when it is editable, so no workflow is written for it, see below.

tests = true puts the job back, without the service. A service declared under [tool.splent.ci.services.<service>] runs in the job under that name, with its image, and optionally env (its container’s environment) and options (docker create options, such as a health check). It is provided, so it no longer leaves the job out, and it replaces what the CLI would run for a compose service of that name. The job’s environment still gets the variables of docker/.env.example that name it. No service may be called db.

[tool.splent.ci]
tests = true

[tool.splent.ci.services.splent_feature_ldap]
image = "osixia/openldap:1.5.0"
env = { LDAP_ORGANISATION = "Test" }
options = "--health-cmd \"ldapsearch -x -H ldap://localhost\" --health-retries 30"

Keeping the workflows in step

The workflows depend on the product’s features, so the commands that change the selection or the pins write again, once they succeed, each workflow whose rendering they changed, exactly as product:sync-template --ci would. These are product:configure, product:resolve, product:derive, product:auto-require, and feature:install, feature:add, feature:remove, feature:attach, feature:detach, feature:upgrade, feature:pin, feature:unlock and feature:rename. A file that changes gets one line, and it is committed with the change that brought it. When the test job is left out, or runs without a server, the lines product:sync-template --ci prints about it follow.

  Updated .forgejo/workflows/splent-validate.yml for the product's features.
  The test job is left out of splent-validate.yml. splent_feature_ldap runs ldap (image osixia/openldap:1.5.0), a service the CI does not know how to run.
  To run the tests anyway, set tests = true under [tool.splent.ci].
  If they need ldap, add [tool.splent.ci.services.ldap] with image = "osixia/openldap:1.5.0".

A command that runs another one says each line once. Inside product:derive the line comes from product:resolve, which writes the file, and product:derive does not repeat it. feature:install runs feature:add or feature:attach in a process of its own with the output captured, so that command writes the file without a word and feature:install prints the line when it is done.

The file is written whole, with the templates of the CLI that runs the command. A command that changes nothing there, such as a dry run or a feature without servers, writes nothing, even when that CLI’s templates differ from the file. A command that does change the rendering brings the CLI’s templates along with the servers. On a fresh clone that is the first product:resolve, because the features were not there to read before it. When splent-validate.yml is as SPLENT wrote it, that resolve writes it again with the templates of the CLI on that machine. With the CLI that wrote it nothing changes, and with a newer or an older one more than the servers does. Look at the diff before committing it.

Only a file SPLENT wrote and nobody edited since is rewritten. Its first line, which product:create and product:sync-template --ci write, holds the sha256 of the rest of the file.

# splent-generated sha256 <hash>
# Written by splent (product:create, product:sync-template --ci). While

An edited workflow, or one an older CLI wrote without that line, is left as it is, and the command says so.

  .forgejo/workflows/splent-validate.yml was edited by hand, so it is left as it is. splent product:drift myapp -v shows what SPLENT would write.
  .forgejo/workflows/splent-validate.yml was written by an older CLI or by hand, so it is left as it is. splent product:sync-template myapp --ci rewrites it.

A pinned feature that is not in the workspace yet, as after product:configure in a new workspace, has compose files nobody can read, so the workflow cannot be computed. Nothing is written, one line says so, and product:resolve writes the workflow once it has fetched the features.

  .forgejo/workflows/splent-validate.yml is updated once splent product:resolve fetches splent_feature_auth, splent_feature_profile and 5 more.

An editable feature that is not at the workspace root is not something product:resolve fetches, since it only links editable features from there. The line names the command that clones it, feature:install <namespace>/<feature> --editable, which writes the workflow once the copy is there.

  .forgejo/workflows/splent-validate.yml is updated once splent feature:install splent-io/splent_feature_notes --editable clones splent_feature_notes at the workspace root.

product:sync-template --ci stops there with exit code 1, writes nothing and names the same commands. Without a group flag the other groups are synced and .forgejo/workflows/ is left as it is, with the same message and exit code 1. product:drift does not compare splent-validate.yml until then, and says so instead of reporting every file up to date. The settings under [tool.splent.ci] are not part of the features, so after editing them run splent product:sync-template <product> --ci.

The pull request of splent-upgrade.yml commits pyproject.toml alone. A new pin that brings a server reaches splent-validate.yml when product:resolve has to fetch that release. When the release is already in the machine’s cache, the rendering does not change and nothing is written, so product:drift shows the difference and product:sync-template <product> --ci writes it.

The release channel

product:release publishes the image to the registries the product lists, the forge’s container registry by default, with two tags. The version tag never moves. The stable tag moves to every new release, and only a release moves it, so a host that runs <image>:stable runs exactly what somebody released.

A production host follows the channel through SPLENT_IMAGE in deploy/.env. When a product’s releases publish an image, the deploy bundle that product:build writes and every release commits runs the channel, and its .env.example sets the variable.

SPLENT_IMAGE=git.splent.io/splent-io/my_product:stable

With it, docker compose up on the host, or product:deploy, pulls the image instead of building one, and the autodeploy feature knows which channel to watch.

Automatic deployment

splent_feature_autodeploy adds an agent container to the production stack. It holds the Docker socket, publishes no port, and answers only on the deployment’s own network, to requests that carry the shared secret. Every few minutes it checks the channel, and when stable points to a new image it deploys it.

1. Pull the channel image. When it is the image the web container already runs there is nothing to do.
2. Skip an image that failed before, until the channel points elsewhere. Honour the pause switch and the update window.
3. Back up the database with mariadb-dump, inside the database container, with the credentials that container holds. No complete dump, no deploy.
4. Swap the web container for one on the new image, with the same name, environment, labels, mounts, ports, networks, restart policy and healthcheck. The old one is renamed and stopped, not removed.
5. Wait for the new container to be healthy. It applies every new migration of every feature as it starts, so the healthcheck covers the migrations too.
6. Keep it when it is healthy, and record which features changed version.
7. Roll back when it is not. The new container goes, the database is dropped and restored from the backup, the previous container is renamed back and started, and the image is held so it is never tried again.

The agent records each step, so an agent stopped in the middle of a deploy finishes or undoes it when it starts again.

The admin panel’s Deployments page shows the channel, the running image and product version, the agent’s state, the update window, and the latest deploys with the feature versions each one moved. A rolled back deploy shows the last log lines of the version that failed. From the page an administrator can check now, which deploys at once even outside the update window, and pause or resume automatic updates. The same actions are commands.

splent feature:autodeploy status
splent feature:autodeploy history
splent feature:autodeploy check
splent feature:autodeploy pause
splent feature:autodeploy resume

Settings

The feature declares its settings in [tool.splent.config], so they reach deploy/.env.example, and from there deploy/.env, with the rest.

Setting	Default	Meaning
AUTODEPLOY_IMAGE	empty, SPLENT_IMAGE is used	The channel to follow
AUTODEPLOY_INTERVAL	300	Seconds between checks, 30 at least
AUTODEPLOY_WINDOW	empty, any time	HH:MM-HH:MM in UTC. A window may wrap past midnight
AUTODEPLOY_HEALTH_TIMEOUT	600	Seconds a new version has to become healthy, migrations included
AUTODEPLOY_KEEP_BACKUPS	5	Database backups kept
AUTODEPLOY_REGISTRY_USER, AUTODEPLOY_REGISTRY_TOKEN	empty	For a private registry
AUTODEPLOY_SECRET	empty, required	Shared by the web container and the agent. docker compose up refuses to start until it is set. product:deploy offers a random one, and --ci generates it

Turning it on

1. Add autodeploy to the product line’s model as an optional feature (it requires admin) and publish the model with spl:publish.
2. Attach the feature to the product and release the product with product:release.
3. On the server, run the bundle the release committed. Clone the product’s repository (or pull it), copy deploy/.env.example to deploy/.env, fill in the required values, AUTODEPLOY_SECRET among them, and run docker compose up -d in deploy/. The bundle already runs the release’s stable image. From a workspace, product:deploy does the same.

From then on, publishing a release is deploying it.

Whoever holds the Docker socket is root on the host. The agent runs no code it is sent, publishes no port, and accepts nothing without the shared secret, but it should run only on a host that serves this product.

Who decides what

Step	Who
Checking every change against the line	The forge, on every push
Proposing newer feature releases	The forge, on a schedule, as a pull request
Accepting them	A person merges the pull request
Publishing a version	A person runs product:release
Deploying it	The agent, inside the update window, or at once when somebody asks
Undoing a version that does not start	The agent, database included

See also

• Build, deploy and release
• product:release
• product:deploy
• feature:upgrade