Test data: who describes it, and why “it works on staging”

The feature was checked on staging — everything passed. On its first day in production it fails on orders created before the release: they lack a field the new version treats as mandatory. Staging had not a single such order.

“Works on staging” is the phrase after which an incident review usually begins. The code on staging and in production is the same. The data is not. On staging there is little of it, and it is clean and fresh. In production there is a lot, it has history, and some of it was created five years ago under rules nobody remembers any more.

Test data is usually considered the tester’s concern. But deciding which data states a check needs can only be done by whoever knows the requirements — that is, the analyst.

Why everything works on staging

How staging differs from production	What goes uncaught because of it
Hundreds of records, not millions	Slow queries, unreadable lists, export timeouts
Data created by the new version	Old records missing fields that are now mandatory
Everything filled in neatly	Empty values, stray spaces, long names, unusual characters
No history	Orders in intermediate statuses, cancelled and partly refunded
External systems replaced by stubs	Slow and failed responses, duplicates, out-of-order messages
Time is “now”	Expired deadlines, month and year rollover, switching to another plan

Each row is not a reason to blame staging but a list of what has to be put there deliberately.

The analyst describes states, not records

The analyst does not need to prepare data by hand. What is needed is to say which states must exist on staging so that every acceptance criterion can be checked. The sources for that list already exist:

the “Given” part of every acceptance criterion — that is the state needed;
the exceptions of the scenario — each needs data under which it occurs;
the entity lifecycle from the data model — one object in each status, including the rare and final ones.

TEST DATA FOR TASK-2140 (item return)
D-1  order delivered 14 days ago            deadline boundary, criterion 1
D-2  order delivered 15 days ago            deadline boundary, criterion 2
D-3  order in transit                       negative case, criterion 3
D-4  order with a BR-12 bundle              refund amount, scenario 3a
D-5  order already partly refunded          refund amount, remainder
D-6  2021 order with no time zone           old data, criterion 4
D-7  customer with 1,200 orders             long list, speed

From such a list the tester or developer prepares the data themselves. And the analyst sees which criteria have nothing to be checked with, before testing even begins.

Old data always gets its own line. Every new mandatory field, new status or new calculation rule meets records created before it existed. What happens to them is a requirement, and if the spec does not state it, the code will decide on the first day in production.

Time as data

Half the rules in a product depend on dates: 14 days to return, a trial period, month-end closing. Nobody will wait two weeks for a check, so staging needs a way to “move time”: create a record dated in the past, or set the system clock for a test. Without that, criteria involving deadlines get checked only in production — by users.

Calendar boundaries are worth checking separately: the last day of the month, 29 February, the new year, daylight-saving changes where they exist. That is exactly where “within 14 days” quietly turns into 13 or 15.

A production copy is not the answer

The temptation is understandable: take a copy of the live database, and staging will look like real life. But a copy has three problems:

Personal data. Customers’ names, phone numbers and addresses on an environment the whole team and contractors can access break both the promises made to users and the law. A copy is acceptable only anonymised.
The case you need may not be there. Production has a million orders, but not one delivered exactly 14 days ago, today.
It cannot be repeated. A test that passed on Tuesday’s copy may fail on Friday — the data has changed. A good data set is restored to the same state with a single command.

A working setup is usually mixed: an anonymised copy supplies volume and “dirt”, and deliberately created records supply the specific states from the list.

The sign there is no test data: before every check the tester spends half an hour creating an order by hand — slightly differently each time.

A fifteen-minute check

Take the acceptance criteria of the next task and, for each “Given”, ask: does such a record exist on staging right now? If not, can it be created in a minute? Criteria where the answer is “no” twice will be checked superficially or not at all.

What next

Everything not caught on staging will meet real data on release day. The last check happens in production — and it too has to be prepared in advance.