Job Summary
We are seeking a skilled Senior Consultant – Databricks to design, develop, and implement modern data engineering solutions using the Databricks platform. The ideal candidate will work closely with customers and internal teams to understand business requirements, build scalable data pipelines, and deliver high-quality data solutions leveraging cloud-based Lakehouse architecture.
The role requires strong hands-on experience in Databricks, Apache Spark, Python, and SQL, along with the ability to collaborate with stakeholders and contribute to end-to-end data platform implementations.
Experience
- 6+ years of overall experience in Information Technology.
- 3+ years of hands-on experience with Databricks.
- Experience in developing and implementing enterprise data engineering solutions.
Key Responsibilities
- Collaborate with customers and business stakeholders to understand data and analytics requirements.
- Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
- Build and optimize ETL/ELT workflows for batch and near real-time data processing.
- Implement Data Lakehouse solutions following the Medallion Architecture (Bronze, Silver, Gold).
- Develop data transformation logic using Python, PySpark, and SQL.
- Ensure data quality, reliability, and performance through testing and optimization.
- Troubleshoot and resolve data processing and performance issues.
- Work closely with Solution Architects, Data Engineers, and DevOps teams to deliver end-to-end solutions.
- Participate in code reviews and follow development best practices.
- Create technical documentation and provide support during deployment and production activities.
Required Technical Skills
- Strong hands-on experience with Databricks.
- Proficiency in Python, SQL, and Apache Spark (PySpark).
- Good understanding of Data Lake and Lakehouse architecture.
- Experience implementing Bronze, Silver, and Gold data layers.
- Experience designing and developing ETL/ELT pipelines.
- Knowledge of Delta Lake and Spark performance optimization.
- Understanding of data modeling and data integration concepts.
- Experience working with structured and semi-structured data.
Preferred Skills
- Experience with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform.
- Familiarity with Delta Live Tables, Unity Catalog, Databricks Workflows, and MLflow.
- Experience with orchestration tools such as Azure Data Factory, Apache Airflow, or similar.
- Knowledge of Git, CI/CD pipelines, and Agile development methodologies.
- Databricks or cloud platform certifications are an added advantage.
Soft Skills
- Strong analytical and problem-solving skills.
- Good communication and customer interaction skills.
- Ability to gather and analyze business requirements.
- Strong collaboration and teamwork skills.
- Ability to manage multiple tasks and deliver within project timelines.
- Willingness to learn new technologies and adapt to evolving project requirements.
What You'll Bring
- Hands-on expertise in Databricks and modern data engineering practices.
- A customer-focused approach with the ability to translate business requirements into technical solutions.
- A passion for building scalable, reliable, and high-performance data platforms.
- The ability to work independently as well as collaboratively in cross-functional teams.