Principal Data Engineer – Databricks | Spark | Delta Lake | PySpark | Data Lakehouse | AWS/Azure
Job Description
Location: Toronto
Work Model: Onsite (4 days/week)
Key Requirements
• 12–18 years of overall Data Engineering experience.
• 8+ years of experience with Enterprise Data Warehouse and Data Lake platforms.
• 5+ years of hands-on experience with Databricks and Apache Spark at scale.
• Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala, Spark) to Databricks Lakehouse.
• Experience redesigning ingestion, transformation, and consumption patterns from HDFS-based architecture to cloud object storage and Delta Lake.
• Experience refactoring legacy Hive/Impala logic into PySpark and Spark SQL ELT pipelines.
• Experience ensuring data reconciliation, audit integrity, and consistency during migration.
• Experience designing and governing Enterprise Data Warehouse and Data Lake/Lakehouse architectures.
• Experience implementing layered architectures including:
• Raw/Landing Layer
• Curated/Conformed Layer
• Semantic/Consumption Layer
• Experience modernizing traditional Enterprise Data Warehouse platforms into scalable Lakehouse architectures.
• Strong experience with finance and risk data models, including:
• General Ledger
• Sub-ledger
• Financial Hierarchies
• Credit Risk Models
• Liquidity Risk Models
• Market Risk Models
• Experience enabling reporting use cases including aggregation, drill-down, and drill-back capabilities.
• Experience building and managing semantic/consumption layers for BI, reporting, and analytics.
• Ability to define business metrics, dimensions, hierarchies, and KPIs.
• Experience with Databricks SQL, Delta Tables, and dbt or similar frameworks.
• Strong experience developing and optimizing large-scale data pipelines using:
• PySpark
• Spark SQL
• Delta Lake
• Experience implementing Medallion Architecture:
• Bronze Layer
• Silver Layer
• Gold Layer
• Experience optimizing workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations.
• Experience implementing data governance, data quality frameworks, reconciliation controls, and exception handling.
• Experience establishing data lineage and metadata management.
• Knowledge of data security, access control, and compliance standards.
• Experience with cloud platforms such as AWS or Azure.
• Experience with CI/CD pipelines using:
• Git
• Terraform
• Jenkins
• Azure DevOps
• Familiarity with orchestration tools such as:
• Apache Airflow
• Databricks Workflows
• Experience with dbt is a plus.
• Ability to act as a technical authority and lead architecture decisions.
• Experience mentoring senior engineers and establishing engineering standards.
• Strong stakeholder management skills with finance, risk, analytics, and governance teams.
• Ability to translate complex data structures into business-ready insights.
Nice to Have
• Experience in Banking, Financial Services, Insurance (BFSI), Capital Markets, or regulatory reporting.
• Exposure to:
• SAP Finance
• Oracle Financials
• SAP S/4HANA
• Experience supporting AI/ML workloads.
• Databricks or cloud certifications.
Key Responsibilities
• Lead Cloudera to Databricks transformation initiatives.
• Design and implement enterprise Data Lakehouse and Data Warehouse solutions.
• Build scalable, high-performance data pipelines and modern data architectures.
• Drive data modernization, governance, quality, and security initiatives.
• Support regulatory, management, and analytical reporting platforms.
• Provide technical leadership, mentor engineering teams, and establish best practices.