Technology & Engineering14 skills

Data Engineering skills for AI agents

14 practitioner-grade data engineering skills, each a focused Markdown document your agent loads into context on demand. Search them from Claude Desktop, Cursor or any MCP client, or pull one with the CLI.

$ npx skilldb search "data engineering"

All 14 skills

Analytics Engineering
Triggers when users need help with analytics engineering, dbt, dbt models, tests, macros, packages, incremental models, semantic layer, metrics definitions, documentation-as-code, CI/CD for data transformations, data mesh, self-serve analytics, or reverse ETL. Activate for questions about building dbt projects, designing transformation layers, implementing metrics layers, or enabling self-service data access.
131 lines
Batch Processing
Triggers when users need help with Apache Spark, batch data processing, RDDs, DataFrames, Catalyst optimizer, partitioning strategies, shuffle optimization, memory tuning, Spark SQL, data skew handling, broadcast joins, adaptive query execution, PySpark best practices, or cluster sizing. Activate for questions about Spark job optimization, batch ETL performance, distributed data processing, or large-scale data transformation.
124 lines
Data Governance
Triggers when users need help with data governance, data cataloging, DataHub, Amundsen, OpenMetadata, data lineage tracking, access control policies, PII detection, data classification, retention policies, regulatory compliance (GDPR, CCPA, HIPAA), data stewardship, or metadata management. Activate for questions about organizing data assets, tracking data lineage, managing data access, handling sensitive data, or meeting regulatory requirements.
156 lines
Data Integration
Triggers when users need help with data integration, Change Data Capture (CDC), Debezium, Fivetran, Airbyte, API data extraction, file-based ingestion, CSV, JSON, Parquet, Avro, webhook processing, data replication, schema registry, Confluent Schema Registry, AWS Glue Schema Registry, or format selection. Activate for questions about ingesting data from external sources, choosing ingestion tools, implementing CDC, or managing data formats and schemas.
144 lines
Data Lake Storage
Triggers when users need help with data lake storage design, object storage (S3, GCS, ADLS), partitioning strategies, file format selection (Parquet, ORC, Avro, JSON), compression codecs (Snappy, Zstd, LZ4), small file problem, storage tiering (hot/warm/cold), lifecycle policies, or storage cost optimization. Activate for questions about organizing data on object storage, choosing file formats, optimizing storage costs, or managing data lake layout.
186 lines
Data Lakehouse
Triggers when users need help with lakehouse architecture, Delta Lake, Apache Iceberg, Apache Hudi, open table formats, ACID transactions on data lakes, time travel, schema enforcement vs evolution, Z-ordering, data skipping, compaction, or medallion architecture (bronze/silver/gold). Activate for questions about combining data lake and warehouse capabilities, choosing table formats, or designing lakehouse storage layers.
133 lines
Data Migration
Triggers when users need help with data migration, large-scale migration strategies, zero-downtime migration, dual-write patterns, reconciliation, shadow traffic, cutover planning, data validation during migration, rollback strategies, cloud-to-cloud migration, or legacy system decommissioning. Activate for questions about migrating databases, moving between cloud platforms, planning migration cutovers, or decommissioning old systems.
170 lines
Data Modeling
Triggers when users need help with data modeling, dimensional modeling, Kimball vs Inmon methodology, data vault 2.0, activity schema, one big table (OBT), normalization vs denormalization, surrogate keys, conformed dimensions, fact table design, entity-relationship modeling, or choosing a modeling approach. Activate for questions about designing data models, choosing between modeling methodologies, or structuring tables for analytics.
137 lines
Data Orchestration
Triggers when users need help with data orchestration, Apache Airflow, DAGs, operators, executors, sensors, Dagster, Prefect, orchestration patterns, fan-out/fan-in, conditional branching, dependency management, retry strategies, backfill operations, or pipeline monitoring and alerting. Activate for questions about scheduling data pipelines, choosing orchestration tools, designing DAG structures, or managing pipeline operations.
141 lines
Data Pipeline Architecture
Triggers when users need help with data pipeline design, ETL vs ELT patterns, batch or streaming pipeline architecture, idempotency, exactly-once semantics, backfill strategies, schema evolution, pipeline orchestration, dead letter queues, or data contracts. Activate for questions about building reliable data pipelines, choosing between batch and streaming, handling pipeline failures, or designing producer-consumer data interfaces.
125 lines
Data Quality
Triggers when users need help with data quality, data testing, data validation, Great Expectations, dbt tests, Soda, anomaly detection in data, data profiling, SLA management, circuit breakers in pipelines, data quality scoring, or data quality dimensions. Activate for questions about ensuring data reliability, building data quality frameworks, monitoring data freshness, or implementing data quality checks in pipelines.
148 lines
Data Warehousing
Triggers when users need help with cloud data warehouse design, Snowflake, BigQuery, Redshift, dimensional modeling, star schema, snowflake schema, slowly changing dimensions, materialized views, query optimization, warehouse compute management, or cost control. Activate for questions about warehouse architecture, data modeling for analytics, query performance tuning, multi-cluster warehouses, or warehouse cost optimization.
140 lines
Real-Time Analytics
Triggers when users need help with real-time analytics, real-time dashboards, OLAP engines, ClickHouse, Apache Druid, Apache Pinot, StarRocks, pre-aggregation vs on-the-fly computation, materialized views for real-time, approximate query processing, HyperLogLog, Count-Min Sketch, lambda architecture, or kappa architecture. Activate for questions about building real-time analytical systems, choosing OLAP databases, sub-second query performance, or real-time aggregation strategies.
154 lines
Stream Processing
Triggers when users need help with stream processing, Apache Kafka architecture, topics, partitions, consumer groups, exactly-once processing, Kafka Streams, Apache Flink, Spark Streaming, event sourcing, CQRS, windowing strategies, watermarks, late data handling, stream-table duality, or real-time aggregation. Activate for questions about building streaming pipelines, choosing streaming frameworks, event-driven architectures, or real-time data processing.
127 lines