Data Engineer (Databricks Specialist)

Job Title: Data Engineer (Databricks Specialist)

Employment Type: Full-time
Experience Level: Mid-Sen

About Us

Client company: We specialize in empowering businesses through data and AI solutions. As trusted advisors to our clients, we design and implement cutting-edge data & AI platforms that transform raw data into actionable insights. We are looking for a talented Data Engineer skilled in Databricks to join our dynamic team and contribute to impactful, innovative projects across industries.

About the Role

As a Data Engineer, you will be responsible for building, optimizing, and maintaining Databricks-based data pipelines and architectures. You will work closely with clients and internal teams to ensure data solutions are scalable, efficient, and tailored to meet specific business objectives.

This role requires technical expertise in data engineering, hands-on experience with Databricks, and a passion for solving complex data challenges in a consulting environment.

Key Responsibilities

• Data Pipeline Development: Design and implement robust ETL/ELT workflows using Databricks to process and transform large datasets efficiently.

• Data Integration: Develop scalable data ingestion processes from various sources, integrating them into Delta Lake or data lakes.

• Collaboration: Work with architects, data scientists, and business analysts to understand requirements and deliver data solutions that align with business goals.

• Performance Optimization: Optimize Databricks workflows for performance, scalability, and cost efficiency, including Spark tuning and cluster management.

• Cloud Integration: Implement cloud-native data engineering solutions on platforms like Azure, AWS, or GCP.

• Governance & Quality: Ensure data accuracy, consistency, and security by implementing best practices in data governance and quality frameworks.

• Automation: Automate repetitive tasks and implement CI/CD pipelines for data workflows.

• Documentation: Maintain comprehensive documentation of processes, workflows, and architecture to ensure knowledge sharing and reproducibility.

Skills & Qualifications

Required:

• Experience:

  • 3-5+ years of experience in data engineering, with hands-on expertise in Databricks.

  • Proven ability to deliver large-scale data pipelines in a consulting or enterprise environment.

• Technical Expertise:

  • Proficiency in Databricks, Delta Lake, and Apache Spark.

  • Strong programming skills in Python, SQL, and optionally Scala.

  • Experience with cloud platforms like Azure (preferred), AWS, or GCP.

  • Knowledge of relational and non-relational databases, data lakes, and real-time data processing.

  • Hands-on experience with PySpark, SQL-based ELT pipelines, and data modelling for lakehouse and warehouse layers

• Soft Skills:

  • Strong problem-solving skills and a proactive approach to addressing challenges.

  • Effective communication and teamwork abilities in a collaborative environment.

Preferred:

• Experience with tools like MLflow or other AI/ML frameworks.

• Familiarity with data visualization tools such as Power BI or Tableau.

• Certification in Databricks or cloud platforms (e.g., Azure Data Engineer Associate).

• Experience in automation and CI/CD practices for data engineering pipelines