Principal Software Engineer (AWS/Python)

Location: London, UK

Department: Engineering

Work Model: Hybrid (8 days per month onsite)

Position Type: Full-time, Permanent

Industry: Energy Data, Commodities & Financial Technology (FinTech)

Reports To: Director of Engineering, London

Visa Sponsorship: Not available.

REQ: 0031.

About Us:

Founded in 1995, Zema Global Data Corporation empowers organizations to simplify complexity, reduce risk, and make faster, more confident decisions that drive measurable results. Over the past two years, Zema Global has accelerated its growth through strategic investment and acquisition to strengthen our global leadership. Together we’re helping our customers gain a Decisioning Advantage - one bold idea at a time. With a presence across global energy, commodity, and financial markets, Zema Global empowers customers to simplify complexity, reduce risk, and make faster, more confident decisions that drive measurable results. At Zema Global, we Think Big, Make It Happen, and Win as One. We thrive on collaboration, creativity, and respect, united by a shared drive to innovate and deliver meaningful impact for our customers and communities. If you’re inspired by solving complex challenges and contributing to a culture that values purpose and performance, we invite you to join us.

Overview:

Zema Global is embarking on a transformational journey to modernize our data infrastructure through the Data Surface Programme with a strategic vision to build a cloud-native AWS Lakehouse architecture. Zema Global’s Data Surface will enable:

• Unified data governance with full lineage, metadata visibility, and automated quality checks

• Scalable, elastic architecture with decoupled compute and storage optimized for AI/ML workloads

• Enhanced security, compliance, and multi-tenant support with RBAC and encryption

• Real-time and batch data processing supporting both human-in-the-loop and fully automated AI workflows

• Self-service analytics and ML capabilities through open data formats and extensible APIs

• AI and machine learning on our own data through AWS platforms such as Amazon Bedrock and SageMaker, made possible by storing data in open source Apache Iceberg format

We are building a UK-based team to own the technical delivery of this programme. You will collaborate with our global engineering teams across the UK, US, Canada, and India, working with cutting-edge cloud-native technologies to shape the future of commodities data infrastructure as part of an ambitious, forward-thinking organization.

Our team operates using an innovative AI-driven development methodology, leveraging Claude Code to accelerate development cycles and enhance code quality. Working in weekly sprints, you will experience a modern, efficient approach to software engineering that combines the best of agile practices with AI-assisted development tools.

Position Overview:

We are looking for a Principal Software Engineer to define and own the technical architecture of our Data Surface programme, designing the end-to-end data platform on AWS including the Lakehouse layer (Apache Iceberg on S3), ingestion patterns, data modelling standards, and governance frameworks that the engineering team builds against. The architecture you define will also underpin our AI/ML ambitions: storing data in open source Apache Iceberg format so that AWS platforms such as Amazon Bedrock and SageMaker can consume it directly.

This is a foundational role. The standards, patterns, and models you establish will shape how the platform is built and how our customers access commodities and energy market data for years to come. You will work alongside the UK engineering team and with global teams across the US, Canada, and India, with the autonomy to make architectural decisions on a programme that has senior sponsorship and genuine strategic weight.

What You'll Be Doing

• Define and maintain the overall data architecture for the AWS Lakehouse platform, including Apache Iceberg table design, partitioning strategies, and storage layout on S3.

• Design data ingestion patterns from DynamoDB, Oracle, and streaming sources (Kinesis, Kafka) into Bronze-Silver-Gold transformation layers.

• Establish data modelling standards, naming conventions, and schema evolution policies across the platform.

• Architect data governance frameworks covering lineage tracking, access control, data quality rules, and metadata management using AWS Glue Data Catalog and related tooling.

• Evaluate and recommend technologies for query performance optimisation across Spark, Trino, Athena, and other SQL engines.

• Design the platform so that data held in Apache Iceberg is consumable by AWS AI/ML services, including Amazon Bedrock and SageMaker, covering table layout, access patterns, and feature engineering workflows.

• Use Claude Code to generate architecture specifications and design documents, and review specifications produced by other team members to ensure AI-generated code accounts for all operational nuances of the technology.

• Collaborate with engineering teams across the UK, US, Canada, and India to ensure architectural consistency and provide technical guidance on implementation.

• Define infrastructure-as-code patterns and reference architectures for EKS, Lambda, and other AWS services used by the platform.

• Produce and maintain architecture decision records (ADRs), data flow diagrams, and technical roadmaps for the Data Surface programme.

What You'll Bring

Essential

• 8+ years of experience as a Principal Software Engineer in data engineering, data platform design, or data architecture roles, including several years hands-on as a software or data engineer building production systems

• Proven track record of designing and delivering large-scale data platforms on AWS (S3, Glue, EMR, Athena, Kinesis, Lambda, EKS) or equivalent cloud environments

• Experience architecting for high-volume, high-frequency data, such as market, trading, or timeseries data, including partitioning, compaction, and retention strategies for very large tables

• Deep expertise in Lakehouse architecture patterns including Apache Iceberg, Delta Lake, or Apache Hudi

• Strong data modelling skills across dimensional, Data Vault, and schema-on-read paradigms

• Advanced SQL proficiency and experience optimising complex analytical queries across distributed engines (Trino, Presto, Athena)

• Hands-on experience with data governance and cataloguing tools (AWS Glue Data Catalog, Apache Atlas, Collibra, or similar)

• Experience with infrastructure as code (Terraform, CloudFormation) and CI/CD pipelines

• Strong communication skills with the ability to translate complex technical concepts for both engineering and business audiences

• Bachelor’s or Master’s degree in Computer Science, Engineering, or STEM related field

Desirable

• Experience with Apache Spark for batch and streaming data processing at scale

• Experience with Apache Kafka, Confluent Platform, or AWS Kinesis for real-time streaming architectures

• Background in migrating Oracle or legacy database workloads to cloud-native architectures

• Familiarity with data mesh or data product principles

• Experience working as an ML engineer, or hands-on experience with ML platform integration (Amazon Bedrock, SageMaker, Databricks) and feature store design

• Knowledge of financial services, commodities, or energy market data, and/or experience working with timeseries data

• AWS Solutions Architect Professional or Data Analytics Specialty certification

Why Join Zema Global?

• Global Impact

  Shape the future of energy, commodities, and financial markets through innovative technology.

• Innovation at Scale

  Work with cutting-edge software, advanced analytics, and complex data that solve real-world challenges.

• Career Growth

  Join a fast-growing global organization where your ideas, development, and career progression are valued.

• Exceptional People

  Collaborate with talented colleagues and industry experts across North America, Europe, and Asia.

• Meaningful Ownership

  Take ownership of your work, influence strategic initiatives, and make a measurable impact on our customers and business.

• One Global Team

  Be part of a collaborative culture built on trust, respect, and our values: Think Big. Make It Happen. Win as One.

Application Information

Please submit your English CV in PDF format. If your experience aligns with our requirements, our Talent Acquisition team will contact you regarding the next steps.

Please note: *Visa sponsorship is not available for this position.*

Equality, Diversity & Inclusion

Zema Global is committed to building an inclusive workplace where everyone can thrive. We encourage applications from all qualified candidates regardless of background or protected characteristic.