Azure Databricks Architect / SME

Job Title: Azure Databricks Architect / SME

Location: Remote
Rate: $60/hr

About the Role

We are seeking an experienced Azure Databricks Architect / SME to design and implement scalable, high-performance data solutions on Microsoft Azure. You will lead end-to-end data engineering projects, from architecture to deployment, while mentoring teams and ensuring best practices in big data processing, ETL, and cloud analytics.

Key Responsibilities

Design & implement advanced analytics solutions using Azure Databricks, Microsoft Fabric, and PySpark.
Develop and optimize data pipelines, Delta Lake architectures, and ETL/ELT processes.
Integrate Azure Databricks with Synapse, Data Factory, ADLS Gen2, and on-prem systems.
Ensure scalability, security, and performance of data solutions.
Lead CI/CD deployments (Git, DevOps) and Infrastructure as Code (IaC) best practices.
Collaborate with stakeholders to translate business needs into technical solutions.
Mentor data engineering teams and drive innovation in big data & cloud analytics.

Required Skills & Experience

8+ years in data engineering/architecture, with strong Azure Databricks expertise (Delta Lake, notebooks, cluster optimization).
Deep knowledge of Azure cloud services (Data Lake, Synapse, Data Factory).
Expert-level PySpark & SQL (query optimization, Delta tables).
Experience with ETL/ELT, data warehousing, and lakehouse architectures.
CI/CD pipelines (Git, DevOps) and cloud security best practices.
Strong leadership, problem-solving, and communication skills.

Nice-to-Have Skills

Power BI (dashboarding & reporting).
Microsoft Fabric (unified analytics).
Spark Streaming (real-time data processing).
ARM templates (Infrastructure as Code).