Data pipelines don’t fail in staging. They fail at 3 a.m. on month-end, when compliance needs the numbers and the dashboard is showing yesterday’s data. You’ll build the pipelines that don’t break—and when they do, you’ll know before anyone else does.
This isn’t a role where you maintain someone else’s SQL. You’ll build data infrastructure that powers trading decisions, product analytics, compliance, and AI/ML systems across a global fintech. You’ll own data accuracy end-to-end, raise the engineering bar for the team, and catch problems before they reach your stakeholders.
Why This Matters
Deriv’s mission is Trading for Anyone, Anywhere, Anytime. Millions of traders, around the clock, across regulatory regimes. Every trade generates data. Every data point feeds analytics, compliance checks, fraud detection, and AI systems that serve customers in real time.
When a pipeline delivers stale data, a trader sees the wrong price. When a schema drifts undetected, a compliance report goes wrong. When governance is an afterthought, regulators ask questions nobody can answer. Data engineering at Deriv isn’t back-office plumbing. It’s the infrastructure that the entire business trusts.
Why Deriv
We’re in production, not planning.
- Natural language interfaces querying the data warehouse directly—leaders ask questions in English, answers come back in seconds
- Continuous KPI monitoring with anomaly detection surfacing problems before stakeholders notice
- Dozens of fraud detection models running against production data, continuously
- 400+ users on our internal workflow orchestration platform, fed by the data infrastructure you’ll help build and scale
We share openly. Deriv⟨ed⟩ is where we write about what we’re shipping, what breaks, and what we figured out the hard way. You’ll join a transformation that’s underway, not one waiting for approval.
What You’ll Do
Your work cuts across trading and product analytics, compliance and regulatory reporting, AI/ML feature pipelines, and business intelligence. Your placement depends on team needs and your strengths—your impact won’t be limited to one domain.
Build pipelines that are governed by design
- Design and build ETL/ELT pipelines across batch and real-time workloads using AI-assisted development—reducing build time without cutting reliability
- Bake in observability from the start: freshness checks, completeness monitoring, schema drift detection, lineage tracking, and alerting. Not retrofitted after launch
- Implement automated data quality checks and anomaly detection into every pipeline—governance built in, not bolted on
Own data accuracy before anyone has to ask
- Identify and resolve data issues before they surface to analysts or stakeholders
- Define and maintain data contracts: SLAs, SLOs, schema agreements, and producer-consumer alignment
- Handle PII, access control, and auditability correctly in a regulated financial environment
Make the platform better than you found it
- Optimise warehouse performance and cost—query efficiency, partitioning, clustering, orchestration reliability
- Build data models designed to scale beyond the immediate use case: dimensional modelling, semantic layers, reusable abstractions
- Spot gaps in the data platform and address them without waiting to be assigned
Raise the bar for the team
- Partner with analysts, product, finance, and compliance teams to translate requirements into reliable, governed data products
- Peer-review pipeline code and push quality standards higher across the team
- Help onboard new engineers—share context, catch mistakes early, make others productive faster
Who You Are
You build pipelines, not just queries
- 2-8 years in data engineering. You know that writing SQL is the easy part. The hard part is making sure data arrives on time, matches the schema you promised, and doesn’t silently break when upstream changes. You think about data quality before someone files a bug.
You know the cloud data stack cold
- GCP, BigQuery, Airflow, Python—or the equivalent at comparable scale. You’ve built and maintained real pipelines across batch and streaming workloads. You’ve shipped with dbt or Dataform and understand why transformation layers matter. You use AI coding assistants daily as part of your workflow, not as a novelty.
You think in contracts, not assumptions
- Data modelling techniques like Kimball star schema, Data Vault, or Medallion architecture aren’t buzzwords to you—they’re tools you pick based on the problem. You understand that a pipeline without a data contract is a pipeline waiting to fail. Even better if you’ve implemented data contract frameworks or schema registries at scale.
You fix root causes, not symptoms
- When data is wrong, you don’t patch the dashboard. You trace the problem to its source, fix it systematically, and make sure it doesn’t recur. You explain what happened clearly—to engineers and to non-technical stakeholders who need to trust the numbers again.
You make the people around you better
- You share learnings actively, create reusable resources, and give feedback that helps others ship better code. When a junior engineer is stuck, you unblock them. When a stakeholder’s requirements are vague, you ask the right questions and turn ambiguity into a spec.
You’ve seen what regulated data looks like
- A fintech, trading, or compliance-heavy background is a strong plus. You know the difference between handling PII in a textbook and handling it when auditors are checking your work. Experience with warehouse cost optimisation and containerised data platforms rounds out the picture.
Tech Stack
- Cloud & Warehouse: GCP, BigQuery
- Orchestration: Airflow
- Languages: Python, SQL
- Transformation: dbt / Dataform
- Streaming: Kafka, Pub/Sub
- Practices: CI/CD, version-controlled pipelines, peer review, AI-assisted development
The Honest Reality
This is demanding work. You’ll own pipeline reliability in a business where stale data means wrong trading decisions and missed compliance deadlines. You’ll debug schema mismatches at month-end when nobody can wait until Monday. You’ll push for governance practices when it’s easier to just ship the quick fix.
But you’ll build infrastructure the entire company depends on. You’ll see your pipelines powering dashboards, feeding AI models, and passing compliance audits. And you’ll work with a team that treats data engineering as a craft, not a cost centre.
If you want predictable work and clean datasets handed to you, this isn’t it. If you want to build data systems that actually matter, it might be.