product:test-profile
Table of contents
Introduction
product:test-profile runs the product’s tests and shows where their time goes: per feature, per test and per fixture, and how much of it is spent on things that make tests slow without testing anything. It changes no code; it tells you what to change.
splent product:test-profile
By feature (6579 tests, 80m01s)
Feature Tests Time Share Per test
dataset 862 14m00s 17% 0.97s
auth 559 8m51s 11% 0.95s
...
Where the time goes
Cause Time Share Tests Most in
database 56m27s 71% 2956 dataset 9m26s, auth 5m11s, dataset_api 4m26s
password hashing 10m23s 13% 1942 auth 2m18s, dataset 2m13s, apikeys 1m18s
outbound HTTP 30.8s 1% 221 search 11.4s, dataset_api 6.3s, dataset 5.0s
The 15 slowest tests
Time Feature Test Mostly
35.7s dataset test_authors_migration.py::test_the_downgrade_leaves_… database 35.1s
...
The fixtures that cost the most
Fixture Scope Set up Time Each
test_client function 2139× 22m14s 0.62s
test_app session 42× 13m28s 19.26s
...
What to do
database (71% of the time)
Much of the time is in database statements. ...
It profiles the active product, which must be running, as product:derive --dev leaves it. The tests run in its web container, as with product:test.
What is measured
A pytest plugin, splent_cli.testing.profile_plugin, goes into every test run through PYTEST_ADDOPTS, so no feature has to change. It measures at the lowest level every library goes through, so it counts whatever a feature uses.
| Cause | Measured at |
|---|---|
password hashing |
hashlib.scrypt, hashlib.pbkdf2_hmac, bcrypt |
sleep |
time.sleep |
outbound HTTP |
http.client, which requests and urllib use. Flask’s test client does not, so a request to the app is not counted |
database |
Every statement SQLAlchemy sends, with the number of them |
Every test is timed in its setup, call and teardown, and every fixture each time it is set up. A test belongs to the feature whose src/<namespace>/<package>/tests/ holds it, whatever the folders are called.
What it reports
| Section | What it shows |
|---|---|
| By feature | Tests, time, share of the whole and time per test |
| Where the time goes | Each cause: its time, its share, the tests it touched, the features it weighs most in |
| The slowest tests | With the cause that took most of each |
| The fixtures that cost the most | How often each was set up and what it cost. A function-scoped fixture set up for many tests is pointed out |
| Tests with more than 200 queries | Candidates for fixtures built in bulk, or for one query per row in the code under test |
| What to do | For each cause that takes 5% of the time or more, how to remove it |
The time of every test file is kept in .splent_cache/test_timings/<product>.json, merged with earlier runs, for planning the next ones.
Reference
| Option | What it does |
|---|---|
--feature NAME |
Only this feature’s tests. Repeatable. Without it, every feature’s and the product’s own |
--unit, --integration, --functional |
Only these levels |
-j, --jobs N |
Runs at a time, as in feature:test |
--top N |
How many tests and fixtures each list shows, 15 by default |
--json |
Print the profile as JSON |
The measurements of each run are in .splent_cache/test_profile/<product>/<time>/, one file per pytest process. It exits 1 when nothing could be measured, for example because the product’s container runs an older splent_cli without the plugin. Failed tests are counted like the others, and the report says how many there were.
See also
product:testruns the tests.- Test environment for the tests’ own database server.