Data Engineer - DataBricks

DESCRIPTION

We are currently looking for a Data Engineer for a client in Toronto. They are looking for someone with experience working with Azure, DataBricks and experience working in the Banking/Finance sectors.

RESPONSIBILITIES

• Analyze and understand business requirements related to data and information, with a particular focus on the financial services industry.
• Design data integration and ETL processes using tools such as Databricks, Spark, Hadoop, and Hive, following a holistic, scalable, pragmatic, and effective approach to data processing.
• Ingest and transform data in Databricks for batch and near real-time processing using technologies such as PySpark and SparkSQL, while leveraging orchestration tools like Azure Data Factory within the broader Microsoft Azure ecosystem. Design and implement data pipelines integrated with Continuous Integration/Continuous Delivery (CI/CD) frameworks as required (DevOps).
• Develop automated unit-testing to ensure data quality and reliability.
• Develop design and operational documentation for job schedules and data pipelines.
• Produce estimates for data analysis and mapping, design, development, and testing.
• Ensure compliance with business, data, and technical requirements.
• Comply with client’s enterprise architecture standards, policies, and procedures are followed.

REQUIREMENTS

• Minimum 5 years experience in Information Technology, participating in complex projects.
• Minimum Bachelors degree in Computer Science.
• Strong communication skills.
• Strong knowledge in data analysis, database development, big data, data lakehouse, data warehousing life cycle and data integration methodologies.
• Strong SQL knowledge of using in Databricks workspace and Delta Lake features.
• Strong programming skills in Pyspark and Apache Spark Architecture.
• Experience with DLT for building reliable and maintainable data pipelines for streaming data using Databricks Structured Streaming.
• Strong understanding of at least one major cloud provider (e.g., AWS, Azure, or GCP) and its data-related services is essential. This includes knowing about services for data storage (like S3 or Azure Blob Storage) and how they integrate with Databricks.
• Designing schemas for databases and understanding concepts like the medallion architecture (Bronze, Silver, Gold layers) and slowly changing dimensions (SCD).
• Familiarity with concepts of data quality, security, and governance, often utilizing Databricks's Unity Catalog.
• Knowledge of ETL development using SSIS, Informatica, Data Stage.
• Knowledge of and experience with Copilots and LLMs to gain higher productivity.
• Strong knowledge of Oracle, Microsoft SQL databases.
• Knowledgeable in both relational and dimensional data modeling (with both Kimball and Inmon approaches).
• Experience in .NET development and UNIX shell scripting (nice to have).
• Good overall business knowledge in financial industries (i.e. one or more in retail banking, commercial banking, capital markets, wealth management, insurance, pension fund, and fintech).
• Ability to work independently and excel in a team environment.
• Preferably as Databricks Certified Data Engineer Associate.