Company description
Publicis Sapient is a digital transformation partner helping established organizations get to their future, digitally enabled state, both in the way they work and the way they serve their customers. We help unlock value through a start-up mindset and modern methods, fusing strategy, consulting, and customer experience with agile engineering and problem-solving creativity. United by our core values and our purpose of helping people thrive in the brave pursuit of next, our 20,000+ people in 53 offices around the world combine experience across technology, data sciences, consulting, and customer obsession to accelerate our clients’ businesses through designing the products and services their customers truly value.
Overview
We are looking for a Big Data Manager / Data Engineering Lead with 8+ years of experience in Data Engineering, Big Data, or Data Warehousing to lead the design and delivery of enterprise-scale, cloud-native data solutions on Microsoft Azure.
This is a hands-on technical leadership role focused on Azure Databricks, Azure Data Lake Storage (ADLS), Apache Spark, Scala, SQL, and streaming technologies. You will lead a team of 3-4 Data Engineers, providing technical direction, design guidance, code reviews, mentoring, and delivery oversight while remaining close to solution architecture and engineering.
Responsibilities
Your Impact
- Design scalable Azure cloud-native data solutions and define technical approaches for enterprise batch and streaming data pipelines using Azure Databricks, ADLS, Apache Spark, Scala, and SQL.
- Lead a team of 3-4 Data Engineers, providing technical direction, mentoring, code reviews, design guidance, delivery coordination, and support removing technical blockers.
- Build and standardize reusable data engineering components and patterns supporting batch and streaming ingestion, transformation, data quality, and movement across raw, curated, and consumption layers.
- Implement reliable and maintainable Databricks and ADLS pipelines while addressing performance, scalability, resilience, logging, monitoring, testing, and other non-functional requirements.
- Collaborate with architects, analysts, QA, client stakeholders, and upstream/downstream application teams to translate business and technical requirements into solution and implementation designs.
- Debug and optimize existing pipelines and codebases, identifying opportunities for automation, performance improvement, scalability, reliability, operational efficiency, and reduced run cost.
- Drive engineering delivery from backlog planning and estimation through QA/UAT, deployment readiness, production implementation, operational handoff, and troubleshooting.
Qualifications
Skills & Experience
- 8+ years of hands-on experience in Data Engineering, Big Data Engineering, Data Warehousing development, or related data-platform engineering, including delivery of production data solutions.
- Strong hands-on expertise with Azure Databricks, Apache Spark, ADLS, Scala, and SQL, including Spark performance optimization and SQL transformation/tuning.
- Proven experience designing and implementing enterprise-grade ETL/ELT pipelines, including reusable ingestion and data transformation patterns.
- Strong understanding of batch and streaming architectures, including Spark Structured Streaming or comparable streaming implementation patterns.
- Demonstrated experience leading and mentoring a small engineering team, with the ability to manage 3-4 Data Engineers, guide technical decisions, review code/designs, unblock engineers, and maintain delivery accountability.
- Experience translating business and technical requirements into solution designs and technical designs, including appropriate consideration of scalability, reliability, performance, maintainability, and data quality.
- Experience working within Agile delivery models and collaborating across engineering, architecture, data, QA, and client/business stakeholder groups.
- Working knowledge of CI/CD, testing, source control, observability, logging/monitoring, deployment, and production-support practices for modern data platforms.
Set Yourself Apart With
- Python / PySpark development experience.
- Broader experience across the Microsoft Azure data ecosystem.
- Experience with MongoDB or other NoSQL technologies.
- Experience designing or implementing data governance and data quality frameworks.
- Experience in the Healthcare domain, particularly working with claims, clinical, eligibility, provider, or member data and HIPAA-aware data handling.
- Experience communicating complex technical decisions and trade-offs to both technical and non-technical client stakeholders.