Lead Data Engineer designing and implementing large-scale data pipelines using Teradata, Hadoop/Cloudera, and Spark within a banking environment. The role involves technical leadership, performance optimization, and infrastructure automation using Terraform and CI/CD tools.
About the Role
We are seeking a Senior Data Engineer with deep expertise in enterprise data platforms, banking domains, and large-scale data delivery. In this role, you will lead the design and development of robust data pipelines across Teradata, Hadoop/Cloudera, and cloud ecosystems while mentoring engineering talent and driving technical excellence.
What you'll do
Architecture & Development:
Lead the design and implementation of scalable batch, streaming, and API-based data pipelines, historical-data processing frameworks, SCD/CDC mechanisms, and semantic-layer data products.
Performance & Optimisation:
Optimise complex SQL queries, Spark workloads, partition strategies, and storage frameworks for large-volume processing within regulated enterprise banking environments.
Infrastructure & Automation:
Implement infrastructure-as-code using Terraform and establish automated CI/CD pipelines, containerised deployments, and rigorous testing standards.
Leadership & Collaboration:
Drive code reviews, enforce engineering standards, mentor junior engineers, and partner closely with architects, product teams, DevOps, and business stakeholders.
What You Will Need to Succeed
Education:
Bachelor's degree in Computer Science, Engineering, Information Technology, or a related discipline
Experience:
At least 10 years of data-engineering experience, with technical leadership in Teradata and Hadoop/big-data delivery.
Preferred Certifications:
Teradata, Cloudera, Databricks, Cloud Data Engineering, or Terraform
Core Technologies:
Teradata, Hadoop/Cloudera, Spark, Hive, Impala, HBase, SQL, Python, Scala/Java, Kafka, Airflow, and Denodo.
Infrastructure & DevOps:
Terraform, Kubernetes/OpenShift, Git, and CI/CD automation tools.
Domain Expertise:
Enterprise banking data (Customer, Payments, Risk, Finance, AML, and Regulatory Reporting) with a strong focus on data quality, reconciliation, lineage, and historical reconstruction.
Core Competencies:
Demonstrated technical leadership, disciplined delivery in regulated settings, exceptional communication skills, and accountability for production-ready outcomes.