Experience Required - 8+ Years.
Work Locations - Bengaluru, Chennai, Hyderabad, Pune.
Notice Period - max 30 Days.
Primary Skills –
Spark Structured Streaming, Apache Spark, Scala, Apache Kafka, Databricks, Any cloud, preferably GCP.
In depth coding knowledge in Spark Structured Streaming.
Kafka producers and Spark structure streaming consumers.
- Strong technical knowledge on Big Data Technologies ( Data bricks, Spark Streaming , GCP , Scala )
- Strong technical skills in Spark Structure streaming, Scala, python, data bricks for lake flow engineering and orchestration, Cloud SQL, Git.
- Spark internals and execution architecture.
- Spark execution concepts - including query planning, DAG generation, shuffle operations, partitioning strategies, and memory management.
- Understanding of Spark optimization techniques such as broadcast joins, caching/persistence strategies, partition pruning, adaptive query execution, and skew handling
- Databricks platform expertise and optimization techniques - cluster sizing strategies, job optimization, Delta Lake internals, workflow orchestration, and performance tuning.
- Advanced batch and streaming ingestion patterns.
- Data Ingestion and Architecture Patterns - Exposure to ingestion frameworks and modern data integration patterns.
- Performance and Optimization Experience - troubleshooting performance bottlenecks, optimizing large-scale data pipelines, or implementing cost and performance improvements.
- Performance tuning and troubleshooting at scale.
- Data architecture and design decision-making.
- Technical leadership, mentoring, and stakeholder communication.
Key Responsibilities
- Possess very strong communication skills and stakeholder management.
- Hands on project technical expertise and involve 80% in technical work and 20% management activities
- Carries very strong Development project experience and manage a team of 10 to 15 technical project team.
- Involve in Technical architecture review meetings and understanding the proposed technical architecture for the project requirements
- Provide required technical mentoring to delivery team
- Efficient coordination with Onshore Tech Lead on the deliverables and ensure to meet quality & schedule adherence
- Report the Monthly Metrics (Internal & Client)
- Generate all the required reports.
- Review and arrange training for the team depending on the need basis.
- Part of deployment activities
- Good analytical and Problem-solving skills