Databricks Platform Engineer
Gyansys Infotech Pvt ltd. · Bangalore ·
- Employment
- Full time
- Category
- Devops
- Experience
- 8+ years
Gyansys Infotech Pvt ltd. · Bangalore ·
Databricks Platform Engineer
Experience : 8+ Years
Work Timings : 2 : 00 PM - 11 : 00 PM IST
Notice Period : Immediate to 15 Days
Cloud Platform : AWS preferred
Employment Type : Full-time
Role Overview :
We are looking for an experienced Databricks Platform Engineer with strong hands-on expertise in designing, managing and optimising enterprise Databricks platforms. The ideal candidate should have a deep understanding of Databricks capabilities and limitations, along with practical experience across Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, Infrastructure as Code (IaC), GitHub and platform monitoring.
The role will focus on ensuring that the Databricks platform is scalable, reliable, secure and cost-efficient while supporting data engineering, data science and MLOps workloads across development, staging and production environments.
Key Responsibilities :
- Design, implement and manage the Databricks platform to support data engineering, data science and MLOps workflows.
- Ensure the platform is scalable, reliable, secure and highly available.
- Manage Databricks environments across Development, Stage and Production.
- Configure and manage native Databricks capabilities including Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, and Compute clusters.
- Establish and maintain best practices for Databricks workspace, compute and platform management.
- Monitor Databricks compute and platform performance from security, reliability and cost-performance perspectives.
- Identify opportunities to optimise compute usage and cloud costs without impacting performance.
- Implement appropriate security controls, access management and governance across Databricks environments.
- Manage platform configurations and deployments using Infrastructure as Code (IaC).
- Use GitHub for version control, collaboration and deployment workflows.
- Support CI/CD processes for Databricks platform and workload deployments.
- Troubleshoot platform, compute, networking, permissions and performance-related issues.
- Work closely with Data Engineers, Data Scientists, MLOps Engineers, Cloud teams and Security teams.
- Establish monitoring, alerting and operational processes for the Databricks platform.
- Document platform architecture, configurations, standards and operational procedures.
- Stay updated with new Databricks features and evaluate their applicability to enterprise use cases.
Required Skills :
- 8+ years of experience in data platforms, cloud engineering, platform engineering or related areas.
- Strong hands-on experience with Databricks administration and platform management.
- Deep understanding of Databricks features, capabilities and limitations.
- Hands-on experience with Delta Live Tables (DLT), Databricks Workflows, Unity Catalog, and Databricks Compute.
- Strong understanding of Development - Stage - Production lifecycle management.
- Experience with AWS Cloud, preferably with Databricks deployed on AWS.
- Hands-on experience with Infrastructure as Code (IaC).
- Strong experience using GitHub for version control.
- Experience monitoring and optimising Databricks compute and platform performance.
- Strong understanding of cloud security, access control and governance.
- Experience with cost optimisation and performance tuning of cloud-based data platforms.
Good to Have :
- Experience with Terraform for Databricks infrastructure provisioning.
- Knowledge of AWS services commonly integrated with Databricks.
- Experience with CI/CD tools and deployment automation.
- Understanding of networking and security architecture in cloud environments.
- Experience supporting MLOps and ML workloads on Databricks.
- Knowledge of Delta Lake and data governance frameworks.
- Experience with enterprise-scale Databricks implementations.
Key Competencies :
- Databricks Platform Administration, Databricks Architecture, AWS Cloud, Unity Catalog, DLT & Workflows, Infrastructure as Code, GitHub & CI/CD, Security & Governance, Cost & Performance Optimisation, Production Support.