The problem with “it feels better”
After a change such as consolidating a team, adopting a new system or standardizing a process, someone will inevitably ask whether it worked. If nobody measured how things stood beforehand, the honest answer is “we do not know”. People then fall back on impressions, which are shaped by recent events, personal experience and how much effort went into the project.
A baseline is a record of current performance, taken before the change, using the same definitions you will use afterwards. It is the reference point that makes a later comparison meaningful.
What a baseline is and is not
A baseline is not a target and not a judgement. It simply says “this is where we are”. It may show the current situation is better or worse than expected, and either is useful. Sometimes the process of collecting it reveals that the data cannot be collected at all, which is itself a finding worth having before a large investment.
Choose what to baseline
Baseline the measures you expect the change to influence, plus a few that might be harmed by it. If a centralization project is meant to speed up request handling, record handling time, but also record quality indicators, such as how often requests are reopened, and the experience of the people served. Otherwise a gain in one area could hide a loss in another.
Each baselined measure needs a clear written definition, because a baseline calculated one way cannot be compared with a later figure calculated another way. The article on metric definitions covers how to write them.
Collect over a long enough period
A single week or month can be unrepresentative. Work often varies by day, month or season: a finance team is busier around period end, a service desk may be busier after a software release, and a retail operation may have obvious peaks.
Some practical suggestions:
- Gather several periods rather than one, so that you can see how much the number normally moves.
- Where work is seasonal, compare against the same period in a previous year if the data exists, in addition to the most recent months.
- Note unusual events in the baseline period, such as an outage, a holiday or a staffing gap, so they are not mistaken for the normal level.
How long is enough depends on the volume of work and how variable it is. For low-volume work, a long period may be needed before the figure is stable; for high-volume work, a shorter period can be enough. When in doubt, collect more and decide later what to use.
Record the spread, not just the average
An average alone hides a lot. Two services can both average ten hours, one with consistent results and one with a mix of very quick and very slow cases. Recording the range, or a middle value such as the median, along with the average gives a better picture of normal variation.
That variation matters when judging a change. If monthly values have typically moved between, say, 8 and 12 hours, then a month at 11 hours after the change says very little. If they have usually stayed within a narrow band, the same movement could be more meaningful. These figures are illustrations only.
Capture the context
Numbers without context are easy to misread later. Alongside the baseline, jot down:
- how the work is currently organized and who does it
- the systems and tools in use
- known data limitations and workarounds
- other changes underway or planned in the same period
- the date range and exact sources used
When a project team later wonders why a result looks odd, this note often explains it.
Make the comparison as fair as possible
Even with a baseline, before-and-after comparisons are imperfect. Many things change over time besides the project: staffing, demand, seasonal patterns, other initiatives. If volumes double at the same time as a new system goes live, the effect of the system is hard to isolate.
Ways to improve fairness include:
- comparing like with like periods, for example the same month in successive years
- looking at ratios or rates rather than raw totals when volume changes
- keeping a log of other changes that could influence the result
- where practical, introducing the change in one area first and comparing it with an area that has not yet changed
Even then, it is wise to describe results cautiously. Showing “the figure moved from X to Y, alongside these other changes” is more honest than claiming the project caused all of the movement.
Expect a dip
Performance often worsens briefly after a change, as people learn new tools and processes. Knowing the baseline lets you tell the difference between a temporary dip and a lasting decline. It is a good idea to agree in advance how long a settling-in period is expected to last and when the first formal comparison will take place. Where the change involves new systems, adoption measurement complements performance figures.
Link to planning
A baseline fits naturally into a current state assessment before a migration, and it informs an honest change case. If a business case is built on expected improvements, the baseline shows how large the starting gap really is, which helps keep the case from being oversold. Alongside go-live testing, the baseline gives later reviews something concrete to compare with.
When no baseline exists
Sometimes the change is already under way and there is nothing recorded. Options include reconstructing figures from historical records, using a comparable site that has not changed, or surveying staff and customers about how they remember things, while acknowledging that memory is less reliable. Treat any reconstructed baseline as approximate and say so. Then start recording properly from now, so the next comparison is on firmer ground.