Data Engineer

Senior Data Engineer | Trading Analytics | London, 3 days on site

A London Trading firm is building a trading data platform from scratch, in house, on open source. Spark, Kafka, Polaris, Ranger, Airflow, MinIO. This is the person who builds it.

Why now

The analytics function is running on legacy warehouses that were never built for the volume or the velocity of trading data. They are consolidating them into a single lakehouse handling streaming and batch together, and building it in house rather than buying it. A Principal Data Engineer is being hired alongside this role to own the architecture. You will be the one turning that design into a working platform.

What you would do

• Build ingestion pipelines for both streaming and batch sources
• Write the Spark jobs that clean, transform and land data in the lakehouse
• Build and own Airflow orchestration
• Work on cataloguing and access control with Polaris and Ranger
• Help migrate data off the legacy warehouses and retire them

What we need

• Strong SQL and Python. These are the working languages of the platform.
• Real experience building production data pipelines, not just maintaining them
• Spark, and some exposure to Kafka or another streaming technology
• Comfort with infrastructure that someone runs themselves rather than only managed cloud services
• Financial markets background, any asset class. Commodities not required.

Useful, not essential

• Java or another JVM language
• Hadoop, HDFS, S3-compatible object storage, open table formats
• Kubernetes, Terraform, CI/CD

Who this suits

Someone two to six years into a data engineering career who wants to build a platform from the ground up rather than add features to one that already exists. You will be closer to the metal here than in most roles at this level, and you will learn more in the first year than in three years of pipeline maintenance somewhere else. If you want a title and a team, the Principal role is the one to look at instead.

The honest version

It is 3 days a week on site in London. Some of the estate is on premise and some of what you inherit is legacy that needs retiring. In exchange you get to build a modern lakehouse at, on open source, with someone senior to learn from.

Process

Three stages: an introductory call, a technical discussion built around real scenarios rather than a timed coding test, and a conversation with the CTO. Moving quickly, filling before year end.