From spotting problems to managing quality
Finding an individual error is one task. Data quality management is the wider discipline: managing quality means having a continuing system for preventing errors, detecting them and fixing their causes. Centralizing that system means one set of rules and measures applied across shared data, instead of each team inventing its own checks.
The case for a central approach grows when data is shared widely. If a single customer table feeds billing, marketing and reporting, an error in it is no longer one team’s problem.
Dimensions of quality
Quality is not a single property. Commonly used dimensions include:
- Accuracy: does the value match reality? A correct-looking address that is no longer where the customer lives is inaccurate.
- Completeness: are the required values present?
- Uniqueness: is each real-world thing recorded once?
- Validity: does the value follow the allowed format or range?
- Consistency: does the same fact agree across systems?
- Timeliness: is the data recent enough for its purpose?
Not every dimension matters equally for every data set. A marketing list may tolerate some staleness, while a payment run cannot tolerate wrong bank details. Setting priorities by use is a central task, because only a wider view shows which data many processes rely on.
Core controls
Validation rules
Validation checks data against stated rules as it is entered or loaded. Examples are that a date must be a real date, that a postcode matches the expected pattern, that quantity must be a positive number, or that a country must come from an approved list. Preventing a bad value at entry is generally cheaper than hunting it down afterwards.
Standardization
Agreeing formats and reference lists, such as one way of writing dates, units and country names, removes many mismatches before they happen.
Deduplication
Matching rules identify records that probably refer to the same entity so they can be reviewed and merged or linked. This is closely tied to master data management, where the shared record is maintained.
Cleansing
Existing errors are corrected, standardized or removed. It is important to log what was changed, since automated cleansing can introduce its own mistakes.
Monitoring
Regular measures show whether quality is improving: for instance the share of records missing a required field, the number of suspected duplicates, or how many records have not been updated within an expected window. Trends matter more than a single reading. Thresholds can trigger alerts when a measure worsens, which often points to a changed process or broken feed.
Issue management
A clear route for reporting, prioritizing and resolving problems, with a named owner for each data area, keeps the work from stalling. This is where data stewardship carries much of the day-to-day load.
How bad data spreads
Central systems reduce some errors but can carry others further. Three illustrations, all invented to show the mechanism:
A mistyped price. Suppose a product’s price is entered as 19.00 when it should be 91.00 in the product master. The order system uses it for quotes, the invoicing system bills it, and the sales report totals it. Each downstream system works exactly as designed, and every one of them reproduces the error. If it is caught only at month-end, several days of orders may need correcting.
A duplicate customer. Imagine one customer recorded twice with slightly different names. Credit limits are applied to each record separately, so the combined exposure is higher than either record shows. Marketing counts two customers, and the customer receives two copies of every mailing. Fixing it involves merging two histories without losing either.
A changed definition. If a central team changes how “active customer” is calculated, say from a 12-month window to a 6-month one, reports based on it will show a fall that reflects only the new rule. Without a note on the change, managers may react to a decline that is not real.
In each case the central design amplified the effect of one error by sharing it. That is a reason to put quality controls at the point where data enters the shared environment, and not just downstream.
The role of the central function
A central quality function typically:
- defines the dimensions, rules and thresholds for shared data;
- provides tools and templates so teams can apply checks;
- publishes quality measures and reports on them;
- coordinates fixes where an issue crosses systems or teams;
- advises on quality requirements for new projects, such as data migration, where mapping and validation decide how much bad data is carried over.
What it should not do is take over every correction. People who create data are best placed to get it right the first time. A central team can set rules, but if the entry process makes mistakes easy, quality will keep slipping.
Limits and trade-offs
- Cost and effort. Rule-writing, tooling and monitoring take sustained time. The return is hard to calculate precisely, so be wary of confident savings claims.
- Over-strict rules. Validation that rejects legitimate but unusual cases pushes people to enter dummy values, which is worse than an imperfect real one.
- False assurance. Passing automated checks does not guarantee data is true. A well-formed but wrong value passes validation.
- Central bottleneck. If only the central team can change a rule or correct a record, fixes may be slow and local users may stop reporting problems.
- Diminishing returns. Effort is best focused on data that drives important processes, not on perfecting every field.
A practical start
Choose a data area where errors hurt, such as customer contact details or supplier bank information. List its required fields and the rules each should obey. Measure a baseline for completeness and duplicates. Add validation at the entry point, name a steward, and re-measure after a few months. Then decide, on the evidence, whether to extend the approach. This article is general information and not a substitute for advice on a specific system.