AWS Data Engineer/ Data Bricks/Pyspark

AWS Data Engineer / Databricks / PySpark Technical Lead

Experience: 5-10 Years

Location: Bangalore (Whitefield)

Job Summary

We are seeking an experienced AWS Data Engineer / Databricks / PySpark Technical Lead with strong expertise in designing, developing, and implementing large-scale data solutions on AWS and Databricks platforms. The ideal candidate should have extensive hands-on experience with Python, PySpark, Spark SQL, Kafka, Airflow, and cloud-based data engineering solutions.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using Python, PySpark, Spark SQL, Databricks, and AWS Services.
• Lead end-to-end architecture, design, development, and implementation of big data solutions.
• Develop and maintain real-time and batch data processing workflows using Apache Kafka and Airflow.
• Build and support data integration solutions using APIs, streaming platforms, and cloud-native services.
• Perform data modeling, database design, performance tuning, and optimization activities.
• Support production systems, troubleshoot critical issues, and provide timely resolutions.
• Collaborate with business stakeholders, cross-functional teams, and vendors to deliver high-quality solutions.
• Mentor junior team members and provide technical leadership in Agile development environments.
• Participate in code reviews, architecture discussions, and best practice implementation.

Required Skills

• Strong experience in Python, PySpark, Spark SQL, Databricks, AWS Cloud Services.
• Minimum 2+ years of Apache Kafka streaming experience.
• Hands-on experience with Apache Airflow.
• Experience working with Oracle, PostgreSQL, and NoSQL databases.
• Good knowledge of SQL, PL/SQL, Unix Shell Scripting, and Autosys.
• Experience with GitHub, Agile methodologies, and CI/CD practices.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and stakeholder management skills.

Preferred Qualifications

• Experience in Banking or Capital Markets domain.
• Proven ability to lead Agile development teams and deliver projects within tight deadlines.
• Experience in designing enterprise-scale data platforms and cloud migration initiatives.

Note: This role requires approximately 90% hands-on development, design, and architecture work, along with 10% team mentoring and technical leadership responsibilities.

Provide your feedback on BizChat

Make this JD more conciseAdd key performance indicators for this roleCreate interview questions based on this JD