Lead and mentor a distributed team of data engineers building enterprise data solutions on Databricks and Lakehouse architecture, using Python, Spark, and SQL, with real-time streaming, Unity Catalog governance, and Terraform on AWS. Permanent full-time hybrid role in Toronto (3 days in office), CAD $135k-$175k.
Team Lead, Data Engineering - Databricks to lead and mentor a team of data engineers, conducting code reviews, design reviews, and knowledge-sharing sessions across multiple locations
8+ years of experience in
data engineering
3+ years of hands‑on experience with
Databricks
platform
Proven experience
leading a team
Strong expertise in
Python and Spark
programming
Demonstrable experience in using
AI
in development
Proven experience with
AWS
or other similar cloud services
Deep understanding of data modeling and SQL
Experience with
Delta Lake and Lakehouse
architecture
Strong knowledge of
ETL/ELT
principles and patterns
Experience with version control systems (Git)
Demonstrated ability to optimize data pipelines
Strong problem-solving and analytical skills
Excellent communication and collaboration abilities
Nice to Have:
Financial services industry experience
Experience with multiple cloud providers
Knowledge of AI/ML implementation patterns
Experience with real time data processingData governance framework experience
What Sets You Apart:
Active interest in emerging technologies and industry trends
Experience implementing complex data solutions
Track record of continuous learning and skill development
Strong technical documentation abilities
Experience using AI tools to enhance productivity
Ability to mentor junior team members
Technical Environment:
Primary Platform: Databricks
Languages: Python, SQL
Tools: Delta Lake, Unity Catalog, Git
Additional: Real-time processing, API integrations
Responsibilities
Your Role:
You will lead and mentor a team of data engineers, conducting code reviews, design reviews, and knowledge-sharing sessions across multiple locations
Drive the Agile/Scrum SDLC process and collaborate with team members
Design and develop Databricks solutions leveraging Lakehouse architecture for enterprise data processing and analytics
Develop and optimize ETL/ELT pipelines
Create and manage structured streaming pipelines for real-time data processing
Configure and optimize Databricks clusters and Spark jobs for optimal performance
Utilize Delta Live Tables for data ingestion and transformations
Apply Unity Catalog features and IAM best practices for security governance and access control
Support infrastructure and resource management using Terraform
Implement monitoring solutions for pipeline performance and data quality
Contribute to code reviews and knowledge-sharing sessions
Salary Range: CAD $135,000 - CAD $175,000
Permanent Full Time
Hybrid (3 days in the office)
Must Haves:
8+ years of experience in
data engineering
3+ years of hands‑on experience with
Databricks
platform
Proven experience
leading a team
Strong expertise in
Python and Spark
programming
Demonstrable experience in using
AI
in development
Proven experience with
AWS
or other similar cloud services
Deep understanding of data modeling and SQL
Experience with
Delta Lake and Lakehouse
architecture
Strong knowledge of
ETL/ELT
principles and patterns
Experience with version control systems (Git)
Demonstrated ability to optimize data pipelines
Strong problem-solving and analytical skills
Excellent communication and collaboration abilities
Nice to Have:
Financial services industry experience
Experience with multiple cloud providers
Knowledge of AI/ML implementation patterns
API development experience
Experience with real time data processing
Data governance framework experience
What Sets You Apart:
Active interest in emerging technologies and industry trends
Experience implementing complex data solutions
Track record of continuous learning and skill development
Strong technical documentation abilities
Experience using AI tools to enhance productivity
Ability to mentor junior team members
Technical Environment:
Primary Platform: Databricks
Cloud Platform: AWS (S3, Glue, Lambda)
Languages: Python, SQL
Tools: Delta Lake, Unity Catalog, Git
Additional: Real-time processing, API integrations
Responsibilities
Your Role:
You will lead and mentor a team of data engineers, conducting code reviews, design reviews, and knowledge-sharing sessions across multiple locations
Drive the Agile/Scrum SDLC process and collaborate with team members
Design and develop Databricks solutions leveraging Lakehouse architecture for enterprise data processing and analytics
Develop and optimize ETL/ELT pipelines
Create and manage structured streaming pipelines for real-time data processing
Configure and optimize Databricks clusters and Spark jobs for optimal performance
Utilize Delta Live Tables for data ingestion and transformations
Apply Unity Catalog features and IAM best practices for security governance and access control
Support infrastructure and resource management using Terraform
Implement monitoring solutions for pipeline performance and data quality
Contribute to code reviews and knowledge-sharing sessions
Salary Range: CAD $135,000 - CAD $175,000
Disclaimer:
AI may be used in evaluating candidates.
This posting is for an existing vacancy.