Job Title: Senior Data Engineer
Overview:
We are seeking a skilled and motivated Azure Databricks Data Engineer to join our dynamic team. The ideal candidate will have strong experience with Python, Spark programming, and expertise in building and optimizing data pipelines in Azure Databricks. You will play a pivotal role in leveraging Databricks workflows, Databricks Asset Bundles, and CI/CD pipelines using GitHub to deliver high-performance data solutions. A solid understanding of Data Warehousing and Data Mart architecture in Databricks is critical for success in this role. If youre passionate about data engineering, cloud technologies, and scalable data architecture, wed love to hear from you!
Key Responsibilities:
Python and Spark Programming:
- Develop and maintain scalable data pipelines using Apache Spark within Azure Databricks/ Python/ Pyspark.
- Write optimized, high-performance Spark jobs to process large volumes of data efficiently.
- Utilize PySpark for distributed data processing, transformation, and aggregation tasks.
Databricks Workflows:
- Design and implement Databricks Workflows to automate data pipeline execution, orchestrating complex workflows and batch jobs.
- Set up task dependencies, triggers, and notifications to ensure smooth and reliable execution.
- Monitor, troubleshoot, and optimize Databricks workflows for optimal performance and minimal failures.
Databricks Asset Bundles:
- Create and manage reusable components such as Databricks Asset Bundles, including notebooks, libraries, and models.
- Share and reuse asset bundles across teams to increase efficiency and ensure consistency in development.
CI/CD for Databricks Artifacts using GitHub:
- Implement CI/CD pipelines using GitHub Actions for the continuous integration and deployment of Databricks notebooks, jobs, and libraries.
- Automate the testing, building, and deployment processes to ensure smooth, consistent code delivery across environments.
- Collaborate with teams to implement version control practices and code reviews using GitHub.
Data Warehousing & Data Mart Design:
- Design and implement Data Warehousing and Data Mart solutions using Databricks, ensuring high-performance storage and retrieval of structured data.
- Integrate data from multiple sources into a central data warehouse using Spark-based transformations, ensuring efficient schema design and query performance.
- Implement dimensional modeling, including star and snowflake schemas, within Azure Databricks for data marts to support business intelligence and reporting.
- Data Pipeline Optimization and Management:
- Continuously monitor and optimize Databricks-based data pipelines for performance, scalability, and cost efficiency.
- Implement best practices for data partitioning, caching, and query optimization within the Databricks platform.
- Troubleshoot and resolve issues related to data integrity, performance, and workflow execution.
Collaboration and Stakeholder Communication:
- Work closely with data scientists, analysts, and other teams to understand requirements and build data solutions that meet business needs.
- Communicate technical concepts effectively to both technical and non-technical stakeholders.
- Provide mentorship and guidance to junior data engineers on Databricks best practices, data architecture, and efficient coding techniques.
Preferred Qualifications:
Cloud Certifications:
- Azure certifications, particularly in Databricks, Data Engineering, or Cloud Solutions, are a plus.
- Big Data Technologies:
- Familiarity with other big data tools such as Kafka, Hadoop, or Flink for streaming and real-time data processing is a plus.
Data Science/ML Experience:
- Exposure to machine learning workflows and model management within Databricks (e.g., using MLflow) is beneficial.