Before You Use Collected Data for AI: A Quality Checklist

Write down what a usable batch must contain

Say what the system should do and what a record needs to support that task. Suppose a support assistant must answer return-policy questions for US and UK stores using a retrieval-augmented generation (RAG) index. A usable record contains the policy text, its market, a link to the approved source and enough version or date information to tell whether it applies today.

Write the conditions before inspecting the export. The current-policy index might require a verified active policy for every supported market. A separate archive might retain superseded policies for questions about past orders. The same document can pass review for the archive and fail for the current index. If the source gives no effective date, mark it as unknown and verify which version applies through the approved source or its owner.

Start with the records behind your most important questions. A missing return window can block a useful answer. An absent promotional image usually has no bearing on that task. Mark which fields are required, which are optional, and which may legitimately be empty.

Compare a small sample with the source

Open records from each market, source and page template you expect to use. Include an older record and any row that looks suspicious. Compare the extracted values with the source itself. A populated number can still be wrong. For example, a page might display a 30-day return window next to a 14-day exchange window, and a parser could save 14 in the returns field.

Run checks you can express as rules across the whole batch. Count required blanks, group rows by their expected unique key and flag missing source references. Reserve manual review for questions that need context, such as whether a return window applies to all items or only to unopened ones. If you check three rows from a template, report that you reviewed three. A clean sample cannot tell you that every row was extracted correctly.

In the hypothetical batch below, the source register calls for current return policies in two markets. The export contains five rows. Every row describes the same type of policy, so the review depends on the market, version and source for each row.

The reviewed batch currently covers only the US. The UK counts toward current coverage after the repaired record passes review.

Run the five data quality checks

1. 1
   Duplicates

   Decide what makes two rows the same for this batch. For policies, compare the policy ID, market and version, then inspect the text if a key repeats. Two copies of the same US policy can collapse into one. Separate US and UK policies may share wording today and change independently tomorrow. Keep both. If rows share a key but disagree on the return window, check the approved source before choosing a value.

2. 2
   Missing fields

   Count blanks in each required field and check whether the failures cluster by source or template. In the example, the UK page contains the return window, so the fix belongs in extraction. If a source truly omits the window, record it as unknown and hold the row from uses that need that value. Copying the US value would hide the gap.

3. 3
   Freshness

   Record when the page was collected and, where available, when the policy took effect. A newly downloaded page can describe an expired policy. Choose a refresh interval based on how often the source changes and what a stale answer would cost. Keep historical versions in a separate archive and route historical questions there when the system can identify the relevant period.

4. 4
   Traceability

   Make sure a reviewer can follow a row back to a specific source. Keep the source URL, collection timestamp with time zone, and a version or snapshot reference where available. If a transformation changes a value, keep enough detail to reproduce it. For the orphaned UK export, find the approved policy and confirm the value before releasing the row.

5. 5
   Coverage

   List the markets, languages, document types or time periods the system needs. Compare the list with records that passed review. In the example, counting five exported rows would hide the missing verified UK policy. If a needed group has zero accepted records, collect it through an approved source or narrow the supported scope until it is ready.

Record the gaps someone else needs to know about

Save a short review note with the dataset version, the expected groups, the counts you observed and the action for each unresolved issue. State the denominator with a count. One missing return window in one UK record is a different handoff from one missing value across hundreds of checked UK records. If you inspected a sample, give its size and how you chose it.

For this example, the note could say that one of two required markets is ready for the current index. The US policy is verified, the UK extraction needs repair, and the untraceable copy is on hold. Include the relevant record IDs so the next reviewer can recheck those rows without rebuilding the whole batch.

When a record is missing, compare the export with the list of sources you expected to read. An absent UK row could mean the page was skipped, the job failed to read it or extraction discarded it. Record which case you can verify. Repairing a parser helps when the saved page contains the missing value. A page the job never saved needs another permitted collection attempt or a narrower release.

For a training dataset, define coverage around the intended tasks and examples the system will encounter. Look at how many accepted examples each group contributes. Where you keep separate training and evaluation sets, check that copies or versions of the same example do not leak across the split when that would make the evaluation misleading.

Accept, repair, collect again, or hold

Accept a batch for the scope it passed. Repair a field when the approved source already contains the correct value, then recheck other rows that use the same template. Collect again when a source has changed or a required group was missed. Hold records when the source or applicable version remains unclear.

If you release with a narrower scope, make that limit visible to whoever runs the index. In the example, release for US questions only and route UK questions to the approved support page until the UK policy passes.

After repairing the UK extraction, compare the new value with the approved UK page and rerun the market count. Once both markets pass, test actual assistant questions for each one. Ask a question whose answer depends on the market and check both the answer and the source retrieved. Batch review gives you an input you can explain. The system test shows how that input is used for the questions you tried.

References

• UK Government Data Quality Framework, Data quality dimensions. Covers completeness, uniqueness and timeliness in relation to intended use.
• W3C PROV Overview. Describes how provenance records help people assess the origin and reliability of information.
• NIST AI Risk Management Framework Playbook, Map and Measure. Discusses documenting data suitability and checking representation in the intended setting.
• Google Machine Learning Crash Course, Dividing the original dataset. Explains why duplicate examples can distort evaluation.