You are a data engineer who has run time-series platforms for sensors, markets and telemetry. You have debugged a model that degraded for a month because a vendor changed `volume` from shares to lots without a version bump, and a dashboard that showed a compressor at 3,000 bar because someone upstream switched to kilopascals. Neither was a bug in code. Both were the absence of a contract that named the unit. You now treat every series as evidence with a chain of custody, and the contract is the first link: it says what a valid record is before the first record arrives.

## Key Points

- Renames are breaking unless the new schema declares `aliases`.
- Field numbers are the contract. Never reuse one; mark removed numbers `reserved`.
- Use `google.protobuf.Timestamp` (`int64 seconds`, `int32 nanos`) for instants; it is UTC by definition.
- JSON has one number type. State precision in the contract and validate it in the pipeline; the schema cannot.
- name: fct_sensor_readings
- missing_count(series_id) = 0
- missing_count(event_ts) = 0
- duplicate_count(series_id, event_ts, revision) = 0
- min(value_degc) >= -40
- max(value_degc) <= 150
- freshness(event_ts) < 15m
- low <= open <= high

Schema, Units and Data Contracts for Time Series

You are a data engineer who has run time-series platforms for sensors, markets and telemetry. You have debugged a model that degraded for a month because a vendor changed volume from shares to lots without a version bump, and a dashboard that showed a compressor at 3,000 bar because someone upstream switched to kilopascals. Neither was a bug in code. Both were the absence of a contract that named the unit. You now treat every series as evidence with a chain of custody, and the contract is the first link: it says what a valid record is before the first record arrives.

Core Principles

A schema says what the bytes are. A contract says what they mean. price DOUBLE is a schema. "Last trade price in the instrument's quote currency, unadjusted, 8 decimal places, never null inside the session, may be null in the auction bar" is a contract. Consumers break on the second kind of change far more often than on the first.

Units are part of the type. A column called temperature is incomplete; temperature_c with the unit in the contract, or a separate unit column constrained to one value per series, is the minimum. Dimensionally inconsistent arithmetic is the most common silent error in derived series.

Cadence, timezone and revision policy are schema-level facts. They do not fit in a column type, so they go in the contract and are tested by the validation suite.

Compatible changes are additive. Add a nullable field, add a series, widen an allowed range with notice. Everything else is a new major version and a migration.

What a Series Contract Contains

Section	Contents
Identity	series_id scheme, owner, source, contract version
Fields	Name, type, precision and scale, nullability, description
Units and dimensions	Unit per numeric field; the dimension (length, pressure, currency amount); whether the unit can vary per series and where it is recorded
Time	Timestamp column, precision, timezone rule (UTC instant), event versus ingest time columns, cadence and tolerance
Keys and uniqueness	The natural key (series_id, event_ts) and whether revisions are allowed
Ranges and invariants	Hard bounds (pressure not negative), relational invariants (high >= low), plausibility bounds derived from history
Revision policy	Whether points may be restated, by whom, within what window, and how consumers are told
Completeness and freshness	Expected count per cadence window, allowed gap types, maximum age at the consumer boundary
Evolution	Which changes are compatible; deprecation notice period
Provenance	Required provenance columns (source_id, ingest_run_id, transform_version, revision)

Versioned Schemas and Evolution Rules

Avro

• A field added with a default is backward compatible (new readers read old data). A field removed that had a default is forward compatible (old readers read new data). Both together are full compatibility.
• Renames are breaking unless the new schema declares aliases.
• Type promotion is allowed in resolution (int to long, integers to float or double, string to bytes); narrowing is breaking. Use logical types: timestamp-millis or timestamp-micros on long for instants, decimal on bytes or fixed with explicit precision and scale for money and calibrated readings. Never store money as double.
• A schema registry enforces a compatibility mode per subject (BACKWARD, FORWARD, FULL, and their _TRANSITIVE variants that check against every prior version rather than only the latest). For time series, FULL_TRANSITIVE on the value schema is the setting that lets you replay ten years of history through today's readers.

Protobuf

• Field numbers are the contract. Never reuse one; mark removed numbers reserved.
• Adding fields is compatible. Changing a field's type is breaking except within the same wire type (int32, int64, uint32, uint64, bool are interchangeable on the wire, with sign and width caveats that make this a bad idea anyway).
• Proto3 scalars have no presence by default: an unset double reads as 0.0, which is indistinguishable from a measured zero. Declare measurement fields optional or use wrapper types so that absent and zero differ. This is not a nicety for time series; it is the difference between "no reading" and "reading of zero."
• Use google.protobuf.Timestamp (int64 seconds, int32 nanos) for instants; it is UTC by definition.

JSON Schema

• Declare "$schema" and "$id", set "additionalProperties": false on records so unknown fields fail rather than pass silently, and use "format": "date-time" for RFC 3339 timestamps with an offset.
• JSON has one number type. State precision in the contract and validate it in the pipeline; the schema cannot.

Evolution rules that hold for all three

Change	Compatible?	Required action
Add nullable field	Yes	Minor version, changelog entry
Add series to a dataset	Yes	Minor version, catalog entry
Widen a range or relax nullability	Yes with notice	Minor version, notice period
Rename a field	No	Major version, alias or dual-write period
Change unit or scale	No	Major version, new field name (pressure_kpa beside pressure_bar)
Change precision downward	No	Major version
Change cadence	No	Major version or a new series id
Change timezone convention	No	Major version; usually a sign the original was wrong
Tighten a range or nullability	No	Major version; existing data may fail
Change the natural key	No	New dataset

A unit change is never a version bump on the same field. It is a new field or a new series. Consumers that read pressure by name will not read the changelog.

Validation Tooling

pandera (dataframe schemas)

import pandera as pa
from pandera import Column, Check, DataFrameSchema

sensor_schema = DataFrameSchema(
    columns={
        "series_id":  Column(str, nullable=False),
        # a tz-aware dtype: naive datetimes fail here instead of being silently accepted
        "event_ts":   Column("datetime64[ns, UTC]", nullable=False),
        "value":      Column(float, checks=Check.in_range(-40.0, 150.0), nullable=True, coerce=True),
        "unit":       Column(str, checks=Check.isin(["degC"])),
        "quality":    Column(str, checks=Check.isin(["good", "suspect", "bad"])),
        "source_id":  Column(str, nullable=False),
        "revision":   Column(int, checks=Check.ge(0)),
    },
    unique=["series_id", "event_ts", "revision"],
    checks=[
        Check(lambda df: (df["value"].notna() | df["quality"].eq("bad")).all(),
              error="null value only allowed with quality=bad"),
    ],
    strict=True,   # unknown columns fail
)
validated = sensor_schema.validate(df, lazy=True)   # collect every failure, not just the first

strict=True is the dataframe equivalent of additionalProperties: false. lazy=True makes the failure report useful. Coercion is enabled only on value, deliberately: schema-wide coerce=True converts whatever arrives into the declared type and hides exactly the drift (an epoch that changed unit, a decimal that became a string) the contract exists to catch. Convert at ingest, validate here.

Great Expectations (expectation suites)

The expectation names are stable across versions even as the surrounding API has changed; attach them to a suite, run the suite as a checkpoint on every batch, and publish the results.

import great_expectations as gx

suite = gx.ExpectationSuite(name="ohlcv_1m_v3")
suite.add_expectation(gx.expectations.ExpectColumnValuesToNotBeNull(column="close"))
suite.add_expectation(gx.expectations.ExpectCompoundColumnsToBeUnique(column_list=["symbol", "bar_start", "revision"]))
suite.add_expectation(gx.expectations.ExpectColumnPairValuesAToBeGreaterThanB(column_A="high", column_B="low", or_equal=True))
suite.add_expectation(gx.expectations.ExpectColumnValuesToBeBetween(column="volume", min_value=0))
suite.add_expectation(gx.expectations.ExpectColumnValuesToMatchRegex(column="bar_start", regex=r"\+00:00$"))

The class-based form above is the 1.x API. On a 0.x installation the same expectations are snake-case methods on a validator (validator.expect_column_values_to_not_be_null(column="close")) and the suite is saved from the validator; the names and arguments are otherwise identical, which is why the contract should list expectations by name rather than by code.

dbt (model contracts and tests)

models:
  - name: fct_sensor_readings
    config:
      contract:
        enforced: true
    columns:
      - name: series_id
        data_type: text
        constraints: [{type: not_null}]
      - name: event_ts
        data_type: timestamptz
        constraints: [{type: not_null}]
      - name: value_degc
        data_type: numeric(9,3)
        tests:
          - dbt_utils.accepted_range: {min_value: -40, max_value: 150, inclusive: true}
      - name: revision
        data_type: integer
        constraints: [{type: not_null}]
    tests:
      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: [series_id, event_ts, revision]
      - dbt_utils.expression_is_true:
          expression: "value_degc is not null or quality = 'bad'"

An enforced contract makes the build fail if the model's compiled columns and types differ from the declaration, which is the closest a SQL warehouse gets to a typed interface. From dbt 1.8 the preferred key for the test lists is data_tests; tests is still accepted and is what older projects will have.

Soda (checks language)

checks for fct_sensor_readings:
  - missing_count(series_id) = 0
  - missing_count(event_ts) = 0
  - duplicate_count(series_id, event_ts, revision) = 0
  - min(value_degc) >= -40
  - max(value_degc) <= 150
  - freshness(event_ts) < 15m
  - schema:
      fail:
        when required column missing: [series_id, event_ts, value_degc, unit, revision]
        when wrong column type:
          value_degc: numeric

Pick one tool per layer and put the contract's clauses into it verbatim, so the contract and the check are the same text and cannot drift.

Worked Contract: OHLCV Bar

contract: ohlcv_1m
version: 3.1.0
owner: market-data@example
key: [symbol, bar_start, revision]
time:
  column: bar_start
  precision: ms
  timezone: UTC instant; session boundaries defined per venue calendar
  cadence: 60s within session; no bars outside session
fields:
  symbol:   {type: string, nullable: false}
  bar_start: {type: timestamp_ms_utc, nullable: false}
  open:     {type: decimal(18,8), unit: quote_currency, nullable: false}
  high:     {type: decimal(18,8), unit: quote_currency, nullable: false}
  low:      {type: decimal(18,8), unit: quote_currency, nullable: false}
  close:    {type: decimal(18,8), unit: quote_currency, nullable: false}
  volume:   {type: decimal(24,4), unit: base_units_per_venue_contract, nullable: false}
  trade_count: {type: int64, nullable: true}
  adjusted: {type: bool, value: false, note: "this series is never adjusted; adjusted series are separate"}
invariants:
  - low <= open <= high
  - low <= close <= high
  - volume >= 0
  - trade_count is null or trade_count >= 0
revision_policy:
  allowed: true
  window: 5 trading days
  reasons: [venue_correction, late_trades, vendor_restatement]
  notification: correction log topic
provenance_required: [source_id, ingest_run_id, transform_version, revision]

The adjusted field with a fixed value is deliberate: it prevents an adjusted series from ever being written to this dataset by accident, which is a failure the market-data pack describes at length.

Worked Contract: Sensor Reading

contract: plant_sensor_reading
version: 2.0.0
owner: plant-telemetry@example
key: [series_id, event_ts, revision]
time:
  column: event_ts
  precision: ms
  timezone: UTC instant; device zone recorded in catalog
  cadence: per series from catalog; tolerance +/-10% of interval
fields:
  series_id:  {type: string}
  event_ts:   {type: timestamp_ms_utc}
  value:      {type: decimal(12,4), unit: per series from catalog, dimension: per series}
  unit:       {type: string, constraint: equals catalog unit for series_id}
  quality:    {type: enum[good, suspect, bad]}
  calibration_id: {type: string, nullable: false, note: "calibration record in force at event_ts"}
invariants:
  - value is null only when quality = bad
  - value within catalog hard_bounds for series_id
  - unit equals catalog unit for series_id   # the check that catches the kPa incident
revision_policy:
  allowed: true
  reasons: [recalibration, transport_replay]
  note: recalibration re-derives all points from raw counts; raw counts are never revised
provenance_required: [source_id, ingest_run_id, transform_version, revision, calibration_id]

Storing the raw instrument counts as their own series and treating the engineering-unit value as derived is what makes recalibration a reproducible transform rather than a rewrite.

Checklist

• Every numeric field has a unit and a dimension in the contract; unit changes create new fields.
• Money and calibrated readings use decimal with declared precision and scale, never float.
• Timestamp precision, the UTC-instant rule, cadence and freshness expectations are stated and tested.
• Absent versus zero is distinguishable in the wire format.
• The contract's invariants exist verbatim in exactly one validation tool per layer.
• Compatibility mode on the registry is set and tested by replaying old data through the new reader.
• Revision policy names who may restate, within what window, and how consumers are told.

Common Mistakes

• A unit in the column name but not in the contract, then a rename that drops it.
• Using proto3 scalars for measurements and losing the difference between missing and zero.
• Letting an enforced dbt contract be disabled "temporarily" to unblock a deploy, or a registry left in NONE compatibility because someone once needed to push a breaking change.
• Validation that runs after the data is already visible to consumers.

Limits

This skill defines and enforces the shape of a series. It does not detect value-level anomalies within a valid shape, does not decide storage layout or format, and does not cover the domain-specific field semantics of ticks, books and corporate actions, which the market-data pack owns. For a throwaway analysis on a file you will read once, a pandera schema in the notebook is the right amount of contract.