Data engineering skills in demand (2026)
Every chapter of this guide claims to teach what the job market wants. This page is the receipt. In July 2026 we collected 78 real, currently-open data-engineering job postings — scraped from the public careers boards of 25+ US tech companies (Okta, Anthropic, Databricks, Cloudflare, Coinbase, Pinterest, Stripe, Lyft, Scale AI, Samsara, and others) plus the month's Hacker News "Who is hiring?" thread — and counted how many postings name each skill. Not "top skills" listicle folklore: every number below is a document count over full job-description text, with the counting rules published at the bottom of the page.
Two headline findings before the table:
- The lakehouse era is measurably here. Apache Iceberg appears in 23% of postings — more than twice Hadoop (9%) and three times Hive (8%). If your mental model of "big data" is HDFS-era, Chapter 10 is the correction.
- Data quality is no longer a nice-to-have. "Data quality" language appears in 37% of postings — more than dbt, Snowflake, or Kafka. Chapter 11 exists for a reason.
The top 20, overall
| # | Skill | Postings | % of 78 |
|---|---|---|---|
| 1 | Communication & cross-functional collaboration | 54 | 69% |
| 2 | ETL / ELT / data pipelines | 50 | 64% |
| 3 | SQL | 43 | 55% |
| 4 | Python | 39 | 50% |
| 5 | Spark | 38 | 49% |
| 6 | AWS | 33 | 42% |
| 7 | Distributed systems | 33 | 42% |
| 8 | Observability & monitoring | 32 | 41% |
| 9 | Security | 32 | 41% |
| 10 | Airflow | 30 | 38% |
| 11 | Data quality | 29 | 37% |
| 12 | Machine learning (adjacent) | 28 | 36% |
| 13 | Data lake / lakehouse | 27 | 35% |
| 14 | dbt | 27 | 35% |
| 15 | Snowflake | 26 | 33% |
| 16 | Streaming / real-time data | 25 | 32% |
| 17 | Kafka | 23 | 29% |
| 18 | Kubernetes | 23 | 29% |
| 19 | Data modeling | 22 | 28% |
| 20 | Go | 22 | 28% |
Read it as a shape, not a shopping list. The spine is exactly what this guide is ordered around: SQL and Python as the floor, one heavy processing engine (Spark), one orchestrator (Airflow), one transformation framework (dbt), a cloud (AWS first), and the quality/observability practices that keep it all trustworthy. And the single most-named "skill" isn't a tool at all — communication shows up in 7 of 10 postings, usually phrased as partnering with analysts, ML engineers, and product teams.
The data-engineering stack, ranked
Just the data-engineering tools and concepts (other categories below):
| Skill | Postings | % | Where the guide covers it |
|---|---|---|---|
| ETL / ELT / pipelines | 50 | 64% | Ingestion, Transformation |
| Spark | 38 | 49% | Chapter 5 |
| Airflow | 30 | 38% | Chapter 8 |
| Data quality | 29 | 37% | Chapter 11 |
| Data lake / lakehouse | 27 | 35% | Chapter 10 |
| dbt | 27 | 35% | Chapter 7 |
| Streaming / real-time | 25 | 32% | Chapter 9 |
| Data modeling | 22 | 28% | Chapter 4 |
| Databricks | 19 | 24% | Chapter 5, Chapter 10 |
| Apache Iceberg | 18 | 23% | Chapter 10 |
| Orchestration (as a concept) | 18 | 23% | Chapter 8 |
| Data governance | 16 | 21% | Chapter 11 |
| Flink | 15 | 19% | Chapter 9 |
| Delta Lake | 10 | 13% | Chapter 10 |
| Data lineage | 9 | 12% | Chapter 11 |
| Dagster | 8 | 10% | Chapter 8 |
| Fivetran | 8 | 10% | Chapter 6 |
| Trino / Presto | 8 | 10% | Chapter 3 |
| Hadoop | 7 | 9% | Chapter 5 (as history) |
| Kinesis | 7 | 9% | Chapter 9 |
| Hive | 6 | 8% | Chapter 10 (metastore) |
| Looker | 6 | 8% | — (BI, downstream of Chapter 7) |
| Parquet | 5 | 6% | Chapter 2 |
| Tableau | 5 | 6% | — (BI) |
| Change data capture | 4 | 5% | Chapter 6 |
Worth noticing: every generation of the stack is still on payroll somewhere. Hadoop and Hive haven't vanished — they've dropped to single digits while Iceberg, Delta Lake, and Trino take their slots. You don't need to learn the old stack, but knowing why it lost is cheap and interviews well.
Languages, clouds, datastores
| Languages | % | Cloud & DevOps | % | Datastores | % | ||
|---|---|---|---|---|---|---|---|
| SQL | 55% | AWS | 42% | Snowflake | 33% | ||
| Python | 50% | Kubernetes | 29% | Kafka | 29% | ||
| Go | 28% | CI/CD | 18% | BigQuery | 13% | ||
| Java | 27% | Terraform | 17% | PostgreSQL | 13% | ||
| Scala | 18% | GCP | 15% | ClickHouse | 9% | ||
| Rust | 6% | S3 | 15% | Redshift | 8% | ||
| C++ | 5% | Docker | 12% | DynamoDB | 6% | ||
| Kotlin | 5% | Azure | 10% | Elasticsearch | 6% |
Three practical reads:
- SQL + Python is the entry ticket — SQL is named in over half of postings and Python in exactly half, and the pairing is where this guide starts. Go (28%) and Java (27%) reflect the platform end of the field: data-infrastructure teams building services, not writing transformations.
- AWS is the default cloud (42%, with S3 named separately in 15%), GCP second, Azure third. Multi-cloud fluency is rarely demanded; depth in one is.
- Snowflake leads the warehouses (33% vs BigQuery's 13% and Redshift's 8%) — consistent with the cloud-warehouse chapter's framing.
And the AI adjacency is real but bounded: 36% of data-eng postings mention machine learning, 15% mention LLMs, and 15% mention agents — almost always as a consumer of the pipelines you build, which is exactly the framing of 12.5 AI in the data stack.
Methodology (and what this can't tell you)
Collection. July 2026. Sources: the public Greenhouse/Lever careers APIs of ~50 named US tech companies (25 of which had qualifying roles open), the July 2026 Hacker News "Who is hiring?" thread, We Work Remotely, and Remotive (scanned; it contributed no qualifying postings). A posting entered the corpus only if its title names a data-engineering role (data engineer, analytics engineer, data platform/infrastructure, big data, ETL, data warehouse, lakehouse, streaming platform) — a posting that merely mentions pipelines somewhere in the body doesn't count. Sales, solutions-architect, product, and analyst roles were excluded by title. Multi-location duplicates of the same role were collapsed to one. Final corpus: 78 unique postings, every one with a full job description.
Counting. A skill "appears in" a posting if any of its aliases matches the title or full description, case-insensitively, on word boundaries ("Go" never matches inside "Google"). Synonyms are folded into one canonical skill (ETL/ELT/"data pipelines" count once; PySpark counts as Spark). A vendor naming itself is not demand: a Databricks job description saying "Databricks" doesn't count toward Databricks' number.
Honest limitations. n=78 is a directional snapshot, not a market index. The corpus skews toward well-known US tech companies with public boards, and toward senior/staff roles (that's who those companies were hiring in July 2026) — a Fortune-500-enterprise corpus would show more Azure, Informatica, and Teradata. Ubiquitous skills that postings don't bother naming (Git, Excel, basic SQL) undercount. And "named in a posting" measures demand, not importance — the ranking tells you what filters résumés, not what matters most on the job.
The corpus and counting engine are re-runnable; we plan to refresh this page as the market moves. One reading note: the tables above are scoped — the stack table is complete for data-engineering tools named in 4+ postings, but the languages/cloud/datastores columns show only the top eight each, and cross-cutting practices (testing 15%, Git 14%, code review 12%, system design 6%) aren't tabled at all. If a data-engineering tool you're betting on is missing from the stack table, though, that's a data point: fewer than 4 of the 78 postings named it.