Role Summary

Design, build, and run core services and reusable components for the open-architecture data platform, enabling scalable ingestion, metadata-driven processing, governance, and reliable multi-engine data consumption. Partner with Platform Engineering and Data Engineering teams to provide secure, observable, production-grade capabilities that support Core Reference, IBOR, and Stamford data initiatives.

Key Responsibilities

• Design and build backend services and shared libraries that provide platform capabilities, including:

• Streaming & batch ingestion frameworks

• Metadata-driven pipelines

• API and consumption layer integrations

• Transformation engine and MDM engine

• Engineer reusable platform components aligned with open formats (Iceberg, Parquet), catalog systems (Polaris, Horizon, Unity Catalog), and query engines (Spark, Trino)

• Implement platform security controls including role-based access control, authentication, and integration with enterprise entitlements

• Build and maintain platform observability (logging, metrics, traces), alerting, and operational dashboards in support of SLAs

• Collaborate with architects to define scalable, extensible designs and ensure adherence to open standards

• Drive engineering excellence through code reviews, automated testing, CI/CD, and production readiness practices

• Ensure platform services meet enterprise security and compliance requirements (SCC, governance)

• Participate in on-call / major incident management and troubleshoot production issues to maintain platform stability

Qualifications

• Strong programming skills in Python and Java; knowledge of Rust is a bonus

• Strong programming skills in Spark, with hands-on knowledge of optimization and debugging

• Hands-on experience with Snowflake and Databricks as data platforms

• Strong understanding of SQL optimization, execution, and large-scale analytics

• Deep understanding and working knowledge of open table formats such as Apache Iceberg are preferred

• Deep understanding of distributed computing and hyperscale

• Experience building production-grade services in cloud environments; AWS,GCP and/or Azure is preferred

• Familiarity with data catalogs (any of Polaris, Horizon, Unity) and metadata management

• Familiarity with streaming pipelines and Kafka is preferred

• Understanding of data modeling and data product design

• Experience with CI/CD, containers, and Kubernetes

• Financial/data platform background preferred but not required