Delivery Approach Guide
Knowledge Guides /How do you check project data quality, and when should training or simulation stop?
Last updated: · New Horizon Digital Outsourcing Team
Before modeling, training, or simulation, check data completeness, units, time, labels, and usage permissions. Data problems propagate and grow through subsequent workflows. If key fields are missing or sources are unknown, assess the impact first rather than silently filling gaps with defaults.
Build a data dictionary first
- Define each field's meaning, unit, permitted range, null values, and anomalies. Identical equipment names do not establish identical field semantics. Record collection methods and processing history, distinguishing original values, manual corrections, and calculated values.
Perform targeted spot checks
- Check for duplicates, abrupt changes, reversed time order, and contradictions between sources. Spot-check labels for image tasks, sampling intervals for time-series tasks, and physical ranges for model parameters. General statistical reports do not replace checks against domain rules.
How to handle missing data
- Depending on the intended use, collect additional data, exclude records, or make explicit estimates, documenting the handling rules. Estimated data must not enter acceptance sets as measured data. If key conclusions are sensitive to missing items, stop the affected analysis or downgrade the conclusions to comparisons of assumptions.
Deliver data-quality records
- Retain issue lists, cleaning rules, versions, and changes in sample counts. Tell the recipient which data can be used for training, validation, or display. Results after processing must be traceable to original inputs; do not deliver only an opaque final file.