Get Started
Hunting for you, 24x7Get the iPhone app

Your hunt runs in your pocket, and tells you the moment a strong match opens.

Download on theApp Store
Intercontinental Exchange, Inc.Posted 3 days ago

Senior Developer, Data Engineer

Who we'd reach at Intercontinental Exchange, Inc.Found and reached by Power Apply
Best leadEngineering manager

Leads the team you would join

Technical recruiter

Owns the opening

Team member

On the team, can refer you

Power Apply

From your own Gmail and LinkedIn.

How this role compares

One of 11 open Senior Data Engineer roles in Hyderabad we are tracking today.

Posted 3 days ago, with 3 of the others posted since and 7 still open from before.

Counted across the roles Frontrunner has open right now.

About the role

This is a senior data engineering role in Hyderabad, India, on the AI Centre of Excellence team at Intercontinental Exchange. The engineer owns the Airflow orchestration layer, builds production pipelines, and helps shape the lakehouse and transformation work that feeds analytics and machine learning teams.

What you’ll do

  • Own Apache Airflow DAGs for ingestion, transformation, validation and delivery, including idempotency, backfill safety and SLA alerting.
  • Run Airflow on Kubernetes, managing Helm upgrades, pod templates, resource limits and scheduler performance.
  • Build fault-tolerant ETL and ELT pipelines across APIs, message queues, databases and object storage.
  • Embed schema validation, row-count reconciliation and anomaly detection into pipelines.
  • Design a Bronze, Silver and Gold lakehouse on Apache Iceberg, including schema evolution and partitioning.
  • Manage Databricks compute, Delta Lake workflows, Unity Catalog and scheduled jobs.
  • Build and maintain dbt projects alongside Airflow.
  • Contribute reusable templates, custom operators and utility libraries to the shared platform.
  • Deliver feature pipelines and training datasets for data scientists and ML engineers.
  • Maintain metrics, dashboards, alerts, runbooks and on-call cover.

What they’re looking for

  • At least three years running Airflow in production, including scheduler internals, executors, XComs, pools and the TaskFlow API.
  • Hands-on experience running Airflow on Kubernetes with KubernetesExecutor or CeleryKubernetesExecutor.
  • Strong Python for DAG authoring, dynamic task mapping, custom operators and tested code.
  • Experience building batch and incremental pipelines at scale.
  • Practical knowledge of lakehouse design and the Apache Iceberg table spec.
  • Practical Databricks experience covering clusters, Delta Lake, Workflows and Unity Catalog.
  • Working dbt experience, including models, tests, snapshots and documentation.
  • Engineering-grade Python with pandas, PySpark, SQLAlchemy and Pydantic.

Nice to have

  • Kafka, Kafka Connect or Flink for near-real-time ingestion.
  • PySpark tuning for shuffle and partition sizing.
  • Kubernetes or OpenShift knowledge beyond Airflow, including Helm and RBAC.
  • Data contract practice and catalogue or quality tools such as DataHub, Collibra, Great Expectations or Soda.
  • Managed data services on AWS, Azure or GCP.
  • Terraform or Pulumi for infrastructure as code.
  • Feature store experience.

What’s on offer

  • The posting describes a collaborative team with input into tooling and engineering standards, but it does not list pay or benefits.

Skills

Questions about this role

How much Airflow experience is needed?

At least three years running Airflow in production is expected.

Is Kubernetes experience required?

Yes, hands-on experience running Airflow on Kubernetes is listed as required.

Does the role involve on-call work?

Yes, the engineer participates in on-call cover and owns runbooks.