Senior data engineer

• Data Pipeline Design & Development (ETL/ELT)

o Develop ETL/ELT Pipelines: Build scalable ETL/ELT pipelines using Azure Databricks, integrating multiple data sources like Azure Blob Storage, Azure Data Lake, SQL Databases, etc.

o Automation of Data Workflows: Automate data ingestion, transformation, and loading processes through Azure Data Factory (ADF), Databricks workflows, and Azure Functions.

o Real-time Data Processing: Implement real-time streaming data pipelines using Databricks Structured Streaming for use cases such as IoT or event-driven architectures.

• Data Transformation & Modeling

o Data Cleaning & Transformation: Leverage Apache Spark in Databricks to process large datasets efficiently, performing data cleansing, transformation, and enrichment.

o Data Modeling: Design and implement dimensional data models (e.g., star schema) optimized for performance and querying in Azure Synapse Analytics or other reporting layers.

o Delta Lake Implementation: Use Delta Lake for reliable and scalable ACID-compliant data storage and to optimize data for batch and stream processing.

• Optimization & Performance Tuning

o Optimize Data Processing: Tune Databricks notebooks and jobs for performance, leveraging Databricks' autoscaling features and optimizing Apache Spark configurations for specific workloads.

o Data Partitioning & Indexing: Implement best practices for partitioning large datasets, managing table storage formats (Parquet, Delta), and indexing data for faster querying.

o Cluster Management: Manage Databricks clusters (autoscaling, sizing, and costs), ensuring efficient resource utilization on Azure.

• Collaboration & Integration with Azure Services

o Azure Integration: Integrate Databricks with other Azure services like Azure Data Lake Storage, Azure SQL, Azure Synapse, Azure Key Vault (for security), and Power BI for seamless data flow and analysis.

o Monitoring & Alerts: Set up monitoring, alerting, and logging of data pipelines using tools like Azure Monitor, Databricks Jobs, and Azure Log Analytics to ensure smooth operations.

• Data Governance & Security

o Data Security: Ensure security measures such as data encryption, role-based access control (RBAC), and compliance with GDPR and other regulations using Azure Active Directory (AAD) and Databricks Secrets.

o Data Quality Management: Implement data quality checks in Databricks pipelines, ensuring consistency, accuracy, and validity of data.

o Version Control & CI/CD: Use version control systems like Git and implement CI/CD pipelines using Azure DevOps for Databricks notebooks and data workflows.

• Collaboration & Documentation

o Cross-functional Collaboration: Collaborate with data scientists, analysts, and business users to develop insights and solutions that drive business objectives.

o Documentation: Maintain thorough documentation of data architectures in DF Confluence, processes, and pipelines in Databricks for future scalability and team collaboration.

• Advanced Analytics & Machine Learning

o Machine Learning Integration: Collaborate with data science teams to build and deploy machine learning models using Databricks MLflow and integrate with Azure's machine learning services for operationalization.

o Data Exploration: Support exploratory data analysis and business intelligence needs using Databricks Notebooks and integrate with Azure Power BI or other visualization tools.

Requirements