Leads the team you would join
Owns the opening
On the team, can refer you
From your own Gmail and LinkedIn.
How this role compares
One of 11 open Senior Data Engineer roles in Hyderabad we are tracking today.
Posted 3 days ago, with 3 of the others posted since and 7 still open from before.
Counted across the roles Frontrunner has open right now.
About the role
This is a senior data engineering role in Hyderabad, India, on the AI Centre of Excellence team at Intercontinental Exchange. The engineer owns the Airflow orchestration layer, builds production pipelines, and helps shape the lakehouse and transformation work that feeds analytics and machine learning teams.
What you’ll do
- Own Apache Airflow DAGs for ingestion, transformation, validation and delivery, including idempotency, backfill safety and SLA alerting.
- Run Airflow on Kubernetes, managing Helm upgrades, pod templates, resource limits and scheduler performance.
- Build fault-tolerant ETL and ELT pipelines across APIs, message queues, databases and object storage.
- Embed schema validation, row-count reconciliation and anomaly detection into pipelines.
- Design a Bronze, Silver and Gold lakehouse on Apache Iceberg, including schema evolution and partitioning.
- Manage Databricks compute, Delta Lake workflows, Unity Catalog and scheduled jobs.
- Build and maintain dbt projects alongside Airflow.
- Contribute reusable templates, custom operators and utility libraries to the shared platform.
- Deliver feature pipelines and training datasets for data scientists and ML engineers.
- Maintain metrics, dashboards, alerts, runbooks and on-call cover.
What they’re looking for
- At least three years running Airflow in production, including scheduler internals, executors, XComs, pools and the TaskFlow API.
- Hands-on experience running Airflow on Kubernetes with KubernetesExecutor or CeleryKubernetesExecutor.
- Strong Python for DAG authoring, dynamic task mapping, custom operators and tested code.
- Experience building batch and incremental pipelines at scale.
- Practical knowledge of lakehouse design and the Apache Iceberg table spec.
- Practical Databricks experience covering clusters, Delta Lake, Workflows and Unity Catalog.
- Working dbt experience, including models, tests, snapshots and documentation.
- Engineering-grade Python with pandas, PySpark, SQLAlchemy and Pydantic.
Nice to have
- Kafka, Kafka Connect or Flink for near-real-time ingestion.
- PySpark tuning for shuffle and partition sizing.
- Kubernetes or OpenShift knowledge beyond Airflow, including Helm and RBAC.
- Data contract practice and catalogue or quality tools such as DataHub, Collibra, Great Expectations or Soda.
- Managed data services on AWS, Azure or GCP.
- Terraform or Pulumi for infrastructure as code.
- Feature store experience.
What’s on offer
- The posting describes a collaborative team with input into tooling and engineering standards, but it does not list pay or benefits.
Skills
Questions about this role
How much Airflow experience is needed?
At least three years running Airflow in production is expected.
Is Kubernetes experience required?
Yes, hands-on experience running Airflow on Kubernetes is listed as required.
Does the role involve on-call work?
Yes, the engineer participates in on-call cover and owns runbooks.