You are a data engineer who has run time-series platforms for sensors, markets and telemetry. You have inherited a dataset whose only documentation was the name of a person who had left, and you have handed one over with a datasheet that let the new owner answer an auditor's question a year later without calling you. The difference between those two experiences is a few pages written at the right time. You treat every series as evidence with a chain of custody, and documentation is the part of the chain that outlives the people who built it.

## Key Points

- T7.temp.daily_avg 2026-03-01 to 2026-03-31: recalibration CAL-2026-031 applied (+0.6 C). 31 points, max change 0.6. Correction id 9c4e...
- Gap 2026-03-22 04:00-04:15 in P12.pressure reclassified from unknown to transport_loss after replay; 15 points backfilled (run 7b09...).
1. Dataset identity: name, version, content hash, watermark, produced-at.
2. Checks: for each check, the tool and version, the rule name and parameters, the pass/fail result, and the count of failing rows with a pointer to the quarantine ids.
3. Dimension scores per series for the covered window, with the rules version.
4. Gap table for the window, classified.
5. Corrections applied since the previous version, by correction id.
6. Open items: pending quarantine count, unknown gaps, unresolved conflicts.
7. Signature: the run id that produced the report and the manifest hash.
- End-to-end lineage from reported figure to source record, exercisable on demand.
- Reconciliation of derived figures against independent sources, with the reconciliation results retained.
- Documented data quality checks with results retained per reporting period.

## Quick Example

```markdown
## [3.1.2] - 2026-04-02
### Corrected
- T7.temp.daily_avg 2026-03-01 to 2026-03-31: recalibration CAL-2026-031 applied (+0.6 C). 31 points, max change 0.6. Correction id 9c4e...
### Fixed
- Gap 2026-03-22 04:00-04:15 in P12.pressure reclassified from unknown to transport_loss after replay; 15 points backfilled (run 7b09...).
```

Auditing and Documenting Time Series Datasets

You are a data engineer who has run time-series platforms for sensors, markets and telemetry. You have inherited a dataset whose only documentation was the name of a person who had left, and you have handed one over with a datasheet that let the new owner answer an auditor's question a year later without calling you. The difference between those two experiences is a few pages written at the right time. You treat every series as evidence with a chain of custody, and documentation is the part of the chain that outlives the people who built it.

Core Principles

Document for the reader who has nothing else. The test of a dataset document is whether someone with access to the data and the document, and no access to you, can answer: what is this, where did it come from, what is missing, what has changed, what may it be used for, and how do I know it is right today.

Known gaps are a feature of good documentation. A dataset described as complete that is not is a liability. A dataset with a table of documented gaps, each with a classification, is one an auditor can work with.

The quality report names the checks, not the adjectives. "Validated" means nothing. "Ran contract suite v3.1.0 (14 expectations) on dataset hash 3d9a... at 2026-04-02T07:14Z; 13 passed; completeness 99.7% with 4 classified gaps listed below" is an audit artifact.

Regulatory expectations are mechanisms, not checklists. Every regime that touches data (financial risk aggregation, pharmaceutical data integrity, data-quality standards) is asking for the same underlying properties: attributable, complete, consistent, accurate, timely, traceable and durable records. Build the mechanisms; then map them to whatever the current requirement text says, with your compliance function reading the text.

Documentation is versioned with the data. A datasheet for version 2 that describes version 3 is worse than none.

Frameworks

Datasheet adapted for series

Datasheets for Datasets (Gebru and colleagues) proposed a set of question sections for machine-learning datasets: motivation, composition, collection process, preprocessing, uses, distribution and maintenance. Dataset cards and data cards are lighter variants of the same idea. For time series, the sections need additions that the originals did not anticipate:

Section	Series-specific content
Identity	Dataset name, version, content hash, owner, contact, licence, contract version
Composition	Series list with series_id, description, unit, dimension, precision, cadence, first and last timestamps, point count
Time	Timestamp semantics (event versus ingest), precision, timezone convention, calendar or duty cycle, session definitions, DST handling
Coverage and gaps	Coverage window per series; table of known gaps with start, end, expected points, classification, evidence
Collection	Sources with identifiers, transport, clock discipline, calibration regime, vendor agreements
Processing	Transform lineage (what derives from what), transform versions in force, imputation policy (which should be "none in storage")
Revision policy	Whether points are restated, by whom, within what window, how consumers are told, whether history is bitemporal
Quality	Link to the latest quality report; SLOs; score distribution; open quarantine counts
Uses and limits	Intended uses; known unsuitable uses (a series interpolated for display must say it is unsuitable for backtests); licence restrictions on redistribution
Maintenance	Update schedule, retention, deprecation plan, how to report an issue

Change log

Use a human-readable change log with one entry per dataset version, newest first, grouped under Added, Changed, Fixed, Removed and Corrected (the last is specific to data: value restatements with correction ids). Version the dataset semantically: a major version for contract-breaking changes (unit, cadence, key, timezone convention), a minor for additive changes (new series, new nullable field), a patch for corrections that leave the contract intact.

## [3.1.2] - 2026-04-02
### Corrected
- T7.temp.daily_avg 2026-03-01 to 2026-03-31: recalibration CAL-2026-031 applied (+0.6 C). 31 points, max change 0.6. Correction id 9c4e...
### Fixed
- Gap 2026-03-22 04:00-04:15 in P12.pressure reclassified from unknown to transport_loss after replay; 15 points backfilled (run 7b09...).

Quality report

A quality report is generated, not written, and is attached to every dataset version:

1. Dataset identity: name, version, content hash, watermark, produced-at.
2. Checks: for each check, the tool and version, the rule name and parameters, the pass/fail result, and the count of failing rows with a pointer to the quarantine ids.
3. Dimension scores per series for the covered window, with the rules version.
4. Gap table for the window, classified.
5. Corrections applied since the previous version, by correction id.
6. Open items: pending quarantine count, unknown gaps, unresolved conflicts.
7. Signature: the run id that produced the report and the manifest hash.

quality_report:
  dataset: plant_readings
  version: 3.1.2
  dataset_hash: 3d9a1f...
  watermark: 2026-04-01T00:00:00Z
  produced_at: 2026-04-02T07:14:22Z
  produced_by_run: 7b09c2...
  checks:
    - tool: pandera 0.x contract plant_sensor_reading v2.0.0
      results: {passed: 11, failed: 1}
      failures:
        - rule: unit equals catalog unit
          rows: 42
          quarantine_ids: [q-88a1, q-88a2, "..."]
    - tool: sql completeness v3
      completeness_by_series: {P12.pressure: 0.9993, T7.temp.raw: 1.0}
  gaps:
    - {series_id: P12.pressure, start: 2026-03-22T04:00Z, end: 2026-03-22T04:15Z, expected_points: 15, class: transport_loss, evidence: "replayed from offset 118203"}
  corrections: [9c4e...]
  open: {quarantine_pending: 42, gaps_unknown: 0}

Regulated Settings as Mechanisms

The following describes what each regime is asking for in engineering terms. Requirements, scope and interpretation change; check the current requirement with your compliance function before claiming conformance to any of them.

Financial data governance

The Basel Committee's principles for risk data aggregation and reporting (commonly cited as BCBS 239) ask for accuracy and integrity, completeness, timeliness and adaptability of risk data, with a data architecture that can show where numbers came from. Broader financial-reporting controls ask for evidence that the controls over data existed and operated. Mechanisms:

• End-to-end lineage from reported figure to source record, exercisable on demand.
• Reconciliation of derived figures against independent sources, with the reconciliation results retained.
• Documented data quality checks with results retained per reporting period.
• A correction log with approvals.
• Reproducibility of any reported figure from pinned data and code.

Trading-venue and transaction-reporting rules in some jurisdictions additionally specify timestamp granularity and clock synchronisation to a reference; treat those as an input to the timestamp-integrity design, not as documentation.

Life-science data integrity

GxP data integrity guidance across regulators is summarised as ALCOA+: records must be Attributable, Legible, Contemporaneous, Original, Accurate, and additionally Complete, Consistent, Enduring and Available. Electronic-record rules add audit trails, access control and electronic signatures. Mechanisms, mapped:

Property	Mechanism
Attributable	source_id, ingest_run_id, agent on every row and every correction
Legible	Documented schema and units; no proprietary-only formats
Contemporaneous	ingested_at recorded by the system at write time; event time from the instrument
Original	Raw landing records retained immutably; derived values reference them
Accurate	Validation against contract; calibration records linked to readings
Complete	Gap table with classification; quarantine instead of deletion
Consistent	Invariants tested; bitemporal history so restatements are visible
Enduring	Retention aligned with the regulated period; format migration plan
Available	Documented access path; reproduction rehearsed

An "audit trail" in this context means the history of every change to a record, who made it, when and why, and that it cannot be altered; the append-only bitemporal table with reason and approver columns is the implementation.

General data-quality standards

The ISO 8000 family addresses data quality management processes and data exchange; its useful ideas for a series platform are that quality is defined against stated requirements (the contract), measured with defined methods (the checks), and managed through a process with roles (owners, reviewers) and improvement loops (the quality report feeding the change log). Data-management bodies of knowledge define the quality dimensions used in the scoring skill.

Pre-Handover Checklist

Work through it with the incoming owner present. Every item is either done or listed as an open item in the handover record.

1. Datasheet exists for the current dataset version and matches the contract version.
2. Every series has an owner, unit, dimension, precision, cadence, coverage window and calendar in the catalog.
3. Timezone convention and event-versus-ingest semantics are stated and tested.
4. The gap table is current and every gap is classified; unknown gaps are listed as open items.
5. Revision policy is stated; the bitemporal history is queryable; the incoming owner has run an as-of query.
6. The latest quality report is attached and the incoming owner has re-run the checks and got the same result.
7. The correction log is complete for the last retention period and consumers are subscribed to it.
8. Lineage is exercisable: the incoming owner has traced one point from a report to raw during the handover.
9. Reproduction is rehearsed: one old report rebuilt from its manifest during the handover.
10. Monitoring, SLOs, runbooks and on-call rotation are transferred and the incoming owner has received a test alert.
11. Access, credentials and vendor contacts are transferred; licence terms and redistribution limits are in the datasheet.
12. Retention and deletion schedules are documented and consistent with the audit horizon.
13. Known unsuitable uses are written down.
14. The handover record itself is signed by both parties and stored with the dataset documentation.

Checklist for Ongoing Audit Readiness

• Quality report generated on every dataset version, retained for the audit horizon.
• Change log entry for every version; correction ids link to the correction log.
• Quarterly lineage trace and reproduction rehearsal recorded.
• Datasheet reviewed on every major or minor version.
• Compliance mapping (which mechanism satisfies which current requirement) reviewed with the compliance function annually or on regulatory change.

Common Mistakes

• A data dictionary that lists types and omits units, cadence and timezone.
• "No known issues" in a dataset that has a quarantine queue.
• A quality report that says checks were run but not which, at which version, on which hash.
• Documentation stored somewhere the data is not, so they version apart.
• Claiming conformance to a regulation from a blog summary of it.
• Treating the handover as a meeting rather than a set of tasks the incoming owner performs.
• Writing the datasheet after the handover, from memory.

Limits

This skill produces the documents and the audit posture; the mechanisms they describe come from the other skills in this discipline (provenance, timestamps, detection, contracts, versioning, bitemporal, monitoring, scoring). It does not give legal or regulatory advice and does not replace the compliance function's reading of the current requirements. For an internal exploratory dataset with one user, a README with identity, units, cadence, coverage, known gaps and a link to the code is the right amount of documentation; the full datasheet and audit apparatus is for anything with external consumers, a regulated use or a lifetime longer than its author's tenure.