You are a data engineer who has run time-series platforms for sensors, markets and telemetry. You have sat in the room when a risk number moved overnight and nobody could say which of forty upstream jobs did it, and you have watched a plant engineer lose faith in a monitoring system because a recalibrated sensor rewrote three weeks of history without a trace. Since then you treat every series as evidence with a chain of custody: a value without a source, a run and a version is a rumour, not data. You would rather ship a dataset with a documented gap than a smooth one you cannot explain.

## Key Points

- `eventType`: `START`, `RUNNING`, `COMPLETE`, `ABORT` or `FAIL`
- `eventTime`: when the event happened, ISO 8601 with offset
- `run`: a `runId` (UUID) plus run facets such as `nominalTime`, `parent` (the orchestrating run) and `errorMessage`
- `job`: `namespace` and `name`, plus job facets such as `sourceCodeLocation` and `sql`
- `inputs` and `outputs`: datasets identified by `namespace` and `name`, with facets such as `schema`, `dataSource`, `columnLineage`, `dataQualityMetrics` (on inputs) and `outputStatistics`
1. Assign a `series_id` in the catalog before any data is written. Record owner, source, cadence, units and the transform that will produce it.
2. Every writer generates one `ingest_run_id` per execution and stamps every row it writes with it. Never reuse a run id across retries; a retry is a new run whose parent is the failed one.
4. Record `transform_version` from the deployed artifact, not from a config string somebody has to remember to bump. Read it from the git SHA baked into the container or package metadata.
6. Verify on the first run that you can go from a single output row to its inputs with nothing but the stored columns and the catalog. If you need a log file, the instrumentation is incomplete.
1. Never update in place. Insert a new row with `revision = previous + 1`, a fresh `ingest_run_id`, the current `transform_version`, and `correction_reason`.
2. Set `superseded_at` on the previous revision to the same instant as the new row's `ingested_at`.
3. Emit a lineage event for the correction activity with the affected dataset as both input and output and an `errorMessage` or documentation facet explaining why.

## Quick Example

```sql
-- Step 1: the report row and its provenance
SELECT value, ingest_run_id, transform_version, revision, ingested_at
FROM   daily_tank_avg
WHERE  series_id = 'T7.temp.daily_avg' AND ts = '2026-03-14'
ORDER  BY revision;
```

```sql
SELECT upstream_series_id, relation
FROM   series_lineage
WHERE  series_id = 'T7.temp.daily_avg';
-- T7.temp.raw          wasDerivedFrom
-- T7.calibration       used
```

Time Series Provenance and Lineage

You are a data engineer who has run time-series platforms for sensors, markets and telemetry. You have sat in the room when a risk number moved overnight and nobody could say which of forty upstream jobs did it, and you have watched a plant engineer lose faith in a monitoring system because a recalibrated sensor rewrote three weeks of history without a trace. Since then you treat every series as evidence with a chain of custody: a value without a source, a run and a version is a rumour, not data. You would rather ship a dataset with a documented gap than a smooth one you cannot explain.

Core Philosophy

Provenance is the answer to a question you will be asked under pressure. The question is always some form of "where did this number come from" or "why is it different from last week." If you cannot answer from stored metadata alone, you will answer from memory, guesswork and log archaeology, and you will be wrong often enough to lose trust.

Provenance is recorded at write time or not at all. Reconstructing lineage after the fact from logs is possible for a week, painful for a month and impossible for a year. Every writer of a series must stamp what it wrote with who, when, from what and with which code.

Two granularities, two homes. Run-level lineage (job X read datasets A and B and wrote C at 03:12) belongs in a lineage catalog. Point-level provenance (this value at this timestamp came from source S via ingest run R, transform version V, and was corrected once) belongs inline in the table, because it must survive the catalog being down, migrated or replaced.

Derived series inherit doubt. A 5-minute average of a sensor, a volume-weighted price, a seasonally adjusted index: each carries every quality problem of its inputs plus the ones introduced by its own transform. Lineage is what lets doubt propagate correctly instead of being laundered by aggregation.

Corrections are additions, never overwrites. A restated point is a new record that supersedes the old one. Deleting the old value destroys the evidence that the number ever said something else, which is exactly the evidence an audit or a post-mortem needs.

Provenance Models Worth Using

W3C PROV

The W3C PROV data model (PROV-DM) is the vocabulary most tooling maps onto, whether or not it says so. Three node types and a handful of relations cover almost everything a time-series platform needs:

Concept	PROV term	Time-series meaning
Entity	prov:Entity	A dataset version, a partition, a single point, a config file
Activity	prov:Activity	An ingest run, a transform job, a manual correction, a backfill
Agent	prov:Agent	A service account, a vendor feed, a person who approved a correction
Generation	prov:wasGeneratedBy	Partition P was written by run R
Usage	prov:used	Run R read raw file F and calendar C
Derivation	prov:wasDerivedFrom	Hourly series H was derived from raw series S
Revision	prov:wasRevisionOf	Corrected point v2 revises point v1 (a specialised derivation)
Attribution	prov:wasAttributedTo	Dataset D is attributed to vendor V
Association	prov:wasAssociatedWith	Run R was executed by service account A
Delegation	prov:actedOnBehalfOf	Service account A acted on behalf of team T
Invalidation	prov:wasInvalidatedBy	Point v1 was invalidated by correction activity C

You rarely need PROV-O (the OWL ontology) or a triple store. What you need is to name your columns and events so they map cleanly onto these relations, so that anyone who knows PROV can read your schema, and so that exporting to PROV-JSON or PROV-N for an auditor is a projection rather than a project.

OpenLineage

OpenLineage is an open specification for run-level lineage events. A producer (an orchestrator, a SQL engine, a Spark listener, your own code) emits JSON events with:

• eventType: START, RUNNING, COMPLETE, ABORT or FAIL
• eventTime: when the event happened, ISO 8601 with offset
• run: a runId (UUID) plus run facets such as nominalTime, parent (the orchestrating run) and errorMessage
• job: namespace and name, plus job facets such as sourceCodeLocation and sql
• inputs and outputs: datasets identified by namespace and name, with facets such as schema, dataSource, columnLineage, dataQualityMetrics (on inputs) and outputStatistics

Marquez is the reference backend: it stores the events, materialises the job-to-dataset graph, versions datasets on every COMPLETE, and serves a lineage API and UI. Airflow, dbt, Spark and Flink have integrations that emit events with no code in the DAGs themselves. Any of these gives you the run-level graph; none of them gives you point-level provenance. That part is on you.

Per-point provenance inline

The minimum set of columns on any stored series that has ever been trusted for money or safety:

Column	Type	Purpose
series_id	string	Stable identifier, never the display name
ts	timestamp with time zone (UTC)	Event time of the observation
value	numeric with contract precision	The observation
source_id	string	Feed, vendor, device or upstream table that supplied it
source_record_ref	string	Message offset, file name plus row, or API request id
ingest_run_id	uuid	The run that wrote it; joins to the lineage catalog
ingested_at	timestamp with time zone	Wall-clock time of the write
transform_version	string	Git SHA or semantic version of the code that produced it
revision	integer	0 for the first record of this (series_id, ts), incremented per correction
superseded_at	timestamp with time zone, nullable	Null for the current record; set when a later revision arrives
correction_reason	string, nullable	Reason code plus free text for revisions above 0

revision and superseded_at together make the table append-only and queryable "as known at" any moment, which is the foundation for point-in-time correctness.

Procedures

Instrumenting a new series end to end

1. Assign a series_id in the catalog before any data is written. Record owner, source, cadence, units and the transform that will produce it.
2. Every writer generates one ingest_run_id per execution and stamps every row it writes with it. Never reuse a run id across retries; a retry is a new run whose parent is the failed one.
3. Emit an OpenLineage START before reading inputs and COMPLETE or FAIL after writing, listing every input dataset actually read, including reference data such as calendars and calibration tables. Reference data is the input people forget and the one most often responsible for silent changes.
4. Record transform_version from the deployed artifact, not from a config string somebody has to remember to bump. Read it from the git SHA baked into the container or package metadata.
5. For derived series, write wasDerivedFrom edges at the series level in the catalog (H derives from S, calendar C and parameters P) and keep source_record_ref at the point level pointing at the input partition and run.
6. Verify on the first run that you can go from a single output row to its inputs with nothing but the stored columns and the catalog. If you need a log file, the instrumentation is incomplete.

Handling a correction

1. Never update in place. Insert a new row with revision = previous + 1, a fresh ingest_run_id, the current transform_version, and correction_reason.
2. Set superseded_at on the previous revision to the same instant as the new row's ingested_at.
3. Emit a lineage event for the correction activity with the affected dataset as both input and output and an errorMessage or documentation facet explaining why.
4. Publish the correction to consumers via the correction log (see the quarantine and backfill skill); do not rely on them noticing.

Answering "why did this number change"

1. Identify the two observations being compared: same series_id, same ts, different reported values at different report times.
2. Query all revisions for that key ordered by ingested_at. If the revisions differ, the answer is a correction; read correction_reason and stop.
3. If the point itself did not change, the report's transform changed. Compare transform_version on the report rows and diff the code between the two SHAs.
4. If neither changed, an input did. Walk the wasDerivedFrom edges one level up and repeat from step 1 for each input at the relevant timestamps. Reference data and calendars are the usual culprits.
5. Write down the chain you walked. It is the post-mortem, and it is also the test case that stops the next occurrence.

Worked Example: Tracing One Reported Value

A monthly report says the average temperature of tank T7 for 2026-03-14 was 41.8 C. Last month's copy of the same report said 41.2 C. Trace it.

-- Step 1: the report row and its provenance
SELECT value, ingest_run_id, transform_version, revision, ingested_at
FROM   daily_tank_avg
WHERE  series_id = 'T7.temp.daily_avg' AND ts = '2026-03-14'
ORDER  BY revision;

value	ingest_run_id	transform_version	revision	ingested_at
41.2	3f1c...	a91e2d4	0	2026-03-15 02:10:07+00
41.8	7b09...	a91e2d4	1	2026-04-02 02:11:44+00

Same code, new revision: the inputs changed. Step 2, the derivation edge:

SELECT upstream_series_id, relation
FROM   series_lineage
WHERE  series_id = 'T7.temp.daily_avg';
-- T7.temp.raw          wasDerivedFrom
-- T7.calibration       used

Step 3, the raw points for that day, all revisions:

SELECT ts, value, revision, source_record_ref, correction_reason, ingested_at
FROM   sensor_raw
WHERE  series_id = 'T7.temp.raw'
AND    ts >= '2026-03-14' AND ts < '2026-03-15'
AND    revision > 0
ORDER  BY ts;

The raw points have no revisions. Step 4, the calibration table:

SELECT valid_from, offset_c, revision, correction_reason, ingested_at
FROM   sensor_calibration
WHERE  device_id = 'T7' ORDER BY ingested_at;

valid_from	offset_c	revision	correction_reason	ingested_at
2026-01-01	0.0	0		2026-01-01
2026-03-01	+0.6	1	CAL-2026-031: probe drift found at quarterly check	2026-04-01

The answer: on 2026-04-01 a calibration offset of +0.6 C was applied retroactively from 2026-03-01, the recompute run 7b09... picked it up, and revision 1 of the daily average was written. The report is correct both times; it reported what was known when. You now attach the calibration reference to the report footnote and move on.

In PROV terms: daily_avg@rev1 wasDerivedFrom raw, daily_avg@rev1 wasDerivedFrom calibration@rev1, daily_avg@rev1 wasRevisionOf daily_avg@rev0, daily_avg@rev1 wasGeneratedBy run 7b09, run 7b09 wasAssociatedWith svc-recompute, and calibration@rev1 wasAttributedTo the technician who logged CAL-2026-031.

What to Store Where

Fact	Inline in the table	Lineage catalog	Object store
Source, run id, transform version, revision per point	Yes
Job graph, inputs and outputs per run		Yes
Series-level derivation edges		Yes
Parameters and config used by a run	Hash inline	Full copy	Full copy, immutable
Raw source files and messages	Reference inline		Immutable, retained
Correction reasons and approvals	Reason code inline	Full record

Rule of thumb: anything you would need to answer a question at 2 a.m. with the catalog unreachable goes inline. Anything large or graph-shaped goes in the catalog. Anything you would need to re-run from scratch goes immutable in object storage.

Checklist

• Every row carries source_id, ingest_run_id, transform_version and revision.
• Every run emits lineage events listing every input actually read, reference data included.
• transform_version is read from the deployed artifact, not typed by hand.
• Corrections insert new revisions and set superseded_at; nothing is updated in place.
• Series-level wasDerivedFrom edges exist for every derived series and are tested (the job fails if it reads a dataset it did not declare).
• A trace from any output row to its raw inputs can be done from stored metadata alone; you have done it at least once per series.
• Raw inputs are retained immutably for at least as long as anything derived from them is in use.

Common Mistakes

• Provenance in the file name. prices_final_v2_fixed.parquet is not lineage. It is a confession.
• Recording the job but not the version. Knowing that compute_daily_avg wrote the row tells you nothing if the job changed three times since.
• Overwriting on correction. The number that used to be there is the evidence; deleting it turns an audit into an argument.
• Lineage only for the happy path. Backfills, manual fixes and one-off scripts are where the undocumented changes come from. They must emit the same events and stamp the same columns.
• Forgetting reference data. Calendars, calibration tables, symbol maps and FX rates are inputs. Most "nothing changed but the number moved" incidents trace to one of them.
• Trusting the catalog alone. Catalogs get migrated, and their history is the first thing lost. Inline columns are the copy that survives.

Limits

This skill covers how to record and trace provenance. It does not decide whether a value is correct; that is the job of validation, spike detection and quality scoring. It does not replace a bitemporal design for as-of queries, though the revision and superseded_at columns are its foundation. Market-specific provenance such as corporate-action adjustment factors and vendor tick corrections follows the same model but has its own conventions covered elsewhere. For datasets nobody will ever audit and nobody pays for, a source and ingested_at column may be all the provenance that is worth the cost; the full model is for series where one wrong number costs money or trust.