Azure Databricks DevOps Engineer

Azure Databricks DevOps Engineer

Key Responsibilities:

• Databricks Management: Deploy, configure, and maintain Azure Databricks environments, ensuring scalability, security, and efficiency.

• Infrastructure Automation: Implement and manage IaC using Terraform, ARM templates, or Bicep to automate cloud resource provisioning.

• CI/CD Pipelines: Develop and manage CI/CD pipelines using tools like Azure DevOps, GitHub Actions, Jenkins to automate data pipeline deployments.

• Monitoring & Logging: Set up and maintain monitoring solutions using Azure Monitor, Log Analytics, and Application Insights for proactive issue resolution.

• Security & Compliance: Implement best practices for security, governance, and compliance, including RBAC, encryption, and network security policies.

• Data Integration & Optimization: Work with Data Engineers to optimize ETL pipelines, ensuring high performance and cost efficiency.

• Collaboration: Work closely with Data Scientists, Engineers, and Business Analysts to enable efficient data workflows.

• Incident Management: Troubleshoot and resolve Databricks, Azure Kubernetes Services (AKS), and networking issues efficiently.

• Documentation & Best Practices: Maintain technical documentation and establish best practices for Azure Databricks and DevOps processes.

Required Skills & Qualifications:

• Bachelors or Masters degree in Computer Science, IT, or a related field.

• 5+ years of experience in DevOps, Cloud Engineering, or Data Engineering roles.

• Strong hands-on experience with Azure Databricks, Azure Data Factory, and Azure Synapse Analytics.

• Proficiency in Terraform, ARM Templates, or Bicep for infrastructure automation.

• Experience with CI/CD tools such as Azure DevOps, GitHub Actions, or Jenkins.

• Knowledge of Python, Scala, or PowerShell for automation and scripting.

• Familiarity with containerization (Docker, Kubernetes, AKS).

• Solid understanding of Azure networking, security, and identity management (IAM, RBAC, AAD).

• Experience with monitoring and logging tools (Azure Monitor, Log Analytics, Prometheus, Grafana).

• Strong problem-solving skills and ability to work in an Agile/Scrum environment.

Preferred Qualifications:

• Azure Certifications (e.g., AZ-400, AZ-104, DP-203, DP-900).

• Experience with Big Data processing frameworks (Apache Spark, Delta Lake).

• Understanding of ML Ops and working with machine learning models in Databricks.

• Knowledge of Event-driven architectures (Azure Event Hubs, Kafka, Service Bus).