Non-functional requirements you can actually test
A non-functional requirement without a number cannot be violated — which means it cannot be met either. Of the four parts of a testable requirement, specs usually contain one.
“The system must be fast and reliable” appears in almost every spec. The sentence cannot be checked, so nobody checks it at acceptance. A month after release the catalogue has tripled and an incident arrives: the report takes four minutes to build. The requirement existed — but in a form that made it impossible to violate.
Non-functional requirements fail to reach testing not because they were forgotten. They are written as wishes: no number, no conditions, no way of measuring.
What a testable requirement is made of
Every testable non-functional requirement has four parts. If even one is missing, the argument at acceptance will be about the meaning of words rather than about the system.
- Property — what exactly is measured: response time, share of successful requests, recovery time.
- Value — the boundary beyond which the requirement is violated.
- Conditions — data volume, number of concurrent users, time of day, type of operation.
- Method — where and with what it is measured, who checks and when.
Before
Catalogue search must be fast.
After
NFR-03 Catalogue search response time
value 95% of requests within 1 s, 99% within 3 s
conditions catalogue of 500k items, 200 concurrent users
method load test on staging before every release;
in production — from the server log over 24 hoursA percentile, not the average
Average response time hides the tail. If nine requests out of ten return in 0.2 seconds and the tenth takes eight, the average is about a second and looks decent. The user who lands on the tenth sees eight seconds. So time is specified as a percentile — “95% of requests within…” — and for important operations the 99th as well.
What to ask even when the customer is silent
Customers rarely name non-functional requirements on their own. They have to be drawn out, and a list helps. ISO/IEC 25010 provides a ready catalogue of quality characteristics; in practice these are the ones needed most often:
| Property | Question for the customer | What people usually don’t know |
|---|---|---|
| Performance | How long can the most frequent operation take? | The difference between “usually” and “at peak” |
| Volume and growth | How many records now, and how many in a year? | Growth, especially after a new channel is connected |
| Availability | How much downtime per month is acceptable, and when can maintenance happen? | The cost of an hour of downtime |
| Recovery | How much data can be lost in a failure, and how fast must we be back? | That these are two different numbers, set separately |
| Access and audit | Who sees what, and do we need to know who did what? | That an audit log cannot be added retroactively |
| Retention | How long is data kept, and what happens to it after? | Legal and contractual obligations |
| Compatibility | Which devices and browsers must it work on? | The share of users on old versions |
Where to get a number when nobody knows one
“I don’t know, just make it fast” is a normal answer. A number can still be obtained, and not out of thin air:
- From the current state. Measure how the system behaves today and agree that the new version is no worse.
- From money. If an hour of downtime costs an amount the business can name, acceptable downtime is calculated, not guessed.
- From comparables. How similar products behave — the ones users compare yours with.
A number obtained this way remains an assumption, and that should be written next to it: “taken from a measurement of the current version, review after the first month”. Such a number is better than a blank — it gives something to argue about.
A requirement with no way to check it is a wish. If nobody can say who will measure it, where and with what, the requirement will not make it into the acceptance criteria and will not be checked. A load test remembered the day before release does not get run.
Who checks, and when
A functional requirement is checked on the finished feature. A non-functional one often only on the assembled system with realistic data, and such an environment has to be prepared in advance. So for each non-functional requirement it pays to state in the spec not only the method but the moment of checking: on staging before release, in production from logs, once a quarter as a recovery drill.
The sign there are no non-functional requirements: performance is first discussed in an incident.
A ten-minute check
Search the current spec for “fast”, “convenient”, “reliable”, “secure”, “if necessary”, “in reasonable time”. Each is an unfinished requirement. For each, try to name the four parts: property, value, conditions and method.
A ready-made form and three examples — the non-functional requirement template.
What next
Many non-functional requirements run into the neighbours rather than the system itself. If search must answer within a second and the external pricing service takes five, the requirement is broken before any code is written. That is the subject of describing an integration.