Who we'd reach at CapcoFound and reached by Power Apply
Best leadHiring manager
Leads the team this role joins
Team director
Heads the group this role sits in
Recruiter
In-house, owns the opening
From your own Gmail and LinkedIn.
About the role
This is a remote big data engineering role for someone who writes Scala and uses Apache Spark to build data processing systems. The work centers on distributed pipelines for large volumes of structured and unstructured data, plus tuning and troubleshooting Spark jobs in production. The hiring firm is Capco, a consulting company that serves banks, insurers, payment providers and energy businesses.
What you’ll do
- Design and maintain large-scale big data applications and their pipelines.
- Write high-performance data processing code in Scala on Apache Spark.
- Build and refine ETL and ELT pipelines for mixed structured and unstructured data.
- Work with Hadoop, HDFS, Hive, Kafka and Spark SQL.
- Tune Spark jobs for speed and resource use.
- Investigate and fix production issues in pipelines and distributed applications.
- Add data quality checks, monitoring and error handling.
- Build reusable data processing frameworks.
- Collaborate with architects, analysts and other engineering teams.
- Follow engineering practices such as Git, code review, testing and CI/CD.
What they’re looking for
- Strong hands-on Scala programming.
- Solid experience with Apache Spark and Spark SQL.
- Working knowledge of big data and distributed computing concepts.
- Experience with the Hadoop ecosystem, including HDFS and Hive.
- Experience building large-scale ETL or data engineering pipelines.
- Strong SQL skills.
- Experience with Kafka or similar messaging and streaming tools.
- Familiarity with Parquet, Avro and JSON data formats.
- Experience diagnosing and optimizing Spark applications.
- Understanding of the software development lifecycle and version control.
Nice to have
- Experience with AWS, Azure or GCP.
- Knowledge of Databricks.
- Experience with Airflow or similar orchestration tools.
- Exposure to Delta Lake or other lakehouse technologies.
- Knowledge of Docker, Kubernetes and CI/CD pipelines.
- Experience in Agile or Scrum teams.
What’s on offer
- Remote working arrangement, as listed.
- Client work on engaging projects for banks, insurers, payment providers and energy companies.
Skills
Questions about this role
Is this Big Data Engineer role remote?
Yes, the listing is remote and open globally.
Which skills are required?
Strong Scala, Apache Spark and Spark SQL, plus Hadoop, HDFS, Hive, Kafka and strong SQL.
Are cloud or Databricks skills required?
No, cloud platforms and Databricks are listed as nice to have rather than required.