product:test-profile

Table of contents

  1. Introduction
  2. What is measured
  3. What it reports
  4. Reference
  5. See also

Introduction

product:test-profile runs the product’s tests and shows where their time goes: per feature, per test and per fixture, and how much of it is spent on things that make tests slow without testing anything. It changes no code; it tells you what to change.

splent product:test-profile
  By feature (6579 tests, 80m01s)
    Feature      Tests  Time    Share  Per test
    dataset      862    14m00s  17%    0.97s
    auth         559    8m51s   11%    0.95s
    ...

  Where the time goes
    Cause             Time    Share  Tests  Most in
    database          56m27s  71%    2956   dataset 9m26s, auth 5m11s, dataset_api 4m26s
    password hashing  10m23s  13%    1942   auth 2m18s, dataset 2m13s, apikeys 1m18s
    outbound HTTP     30.8s   1%     221    search 11.4s, dataset_api 6.3s, dataset 5.0s

  The 15 slowest tests
    Time   Feature  Test                                                    Mostly
    35.7s  dataset  test_authors_migration.py::test_the_downgrade_leaves_…  database 35.1s
    ...

  The fixtures that cost the most
    Fixture      Scope     Set up  Time    Each
    test_client  function  2139×   22m14s  0.62s
    test_app     session   42×     13m28s  19.26s
    ...

  What to do
    database (71% of the time)
      Much of the time is in database statements. ...

It profiles the active product, which must be running, as product:derive --dev leaves it. The tests run in its web container, as with product:test.


What is measured

A pytest plugin, splent_cli.testing.profile_plugin, goes into every test run through PYTEST_ADDOPTS, so no feature has to change. It measures at the lowest level every library goes through, so it counts whatever a feature uses.

Cause Measured at
password hashing hashlib.scrypt, hashlib.pbkdf2_hmac, bcrypt
sleep time.sleep
outbound HTTP http.client, which requests and urllib use. Flask’s test client does not, so a request to the app is not counted
database Every statement SQLAlchemy sends, with the number of them

Every test is timed in its setup, call and teardown, and every fixture each time it is set up. A test belongs to the feature whose src/<namespace>/<package>/tests/ holds it, whatever the folders are called.


What it reports

Section What it shows
By feature Tests, time, share of the whole and time per test
Where the time goes Each cause: its time, its share, the tests it touched, the features it weighs most in
The slowest tests With the cause that took most of each
The fixtures that cost the most How often each was set up and what it cost. A function-scoped fixture set up for many tests is pointed out
Tests with more than 200 queries Candidates for fixtures built in bulk, or for one query per row in the code under test
What to do For each cause that takes 5% of the time or more, how to remove it

The time of every test file is kept in .splent_cache/test_timings/<product>.json, merged with earlier runs, for planning the next ones.


Reference

Option What it does
--feature NAME Only this feature’s tests. Repeatable. Without it, every feature’s and the product’s own
--unit, --integration, --functional Only these levels
-j, --jobs N Runs at a time, as in feature:test
--top N How many tests and fixtures each list shows, 15 by default
--json Print the profile as JSON

The measurements of each run are in .splent_cache/test_profile/<product>/<time>/, one file per pytest process. It exits 1 when nothing could be measured, for example because the product’s container runs an older splent_cli without the plugin. Failed tests are counted like the others, and the report says how many there were.


See also


Back to top

splent. Distributed by an LGPL license v3. Contact us: drorganvidez@us.es