We are looking for an experienced and driven Senior Data Engineer with 6+ years of hands-on experience designing, building, and scaling robust data pipelines. The ideal candidate will combine deep technical expertise in PySpark, SQL, and Python with strong business acumen, acting as a crucial bridge between complex data infrastructure and our business stakeholders.
Experience with platforms like Dataiku (DataIQ) is highly preferred to support rapid pipeline development and cross-functional collaboration.
Key Responsibilities
- Pipeline Architecture & Development: Design, build, and optimize scalable ETL/ELT data pipelines using Python, PySpark, and modern orchestration tools.
- Dataiku (DataIQ) Integration: Leverage visual and code-based recipes within Dataiku to rapidly prototype, deploy, and monitor end-to-end data workflows and machine learning pipelines.
- Database Optimization: Write, tune, and maintain highly complex SQL queries and data models to ensure high-performance analytical reporting.
- Stakeholder Collaboration: Partner directly with business units, product managers, and leadership to understand data requirements, translate them into technical specifications, and present insights clearly.
- Data Governance & Quality: Ensure data integrity, reliability, and security across all pipelines through robust testing, monitoring, and best-practice implementation.
- Mentorship: Guide junior and mid-level engineers on best practices for coding, optimization, and scalable data architecture.
Required Qualifications & Skills
- Experience: 6+ years of professional experience in data engineering, data warehousing, or software engineering focused on data systems.
- Big Data Technologies: Advanced proficiency in PySpark and distributed computing frameworks.
- Database Expertise: Exceptional SQL skills, including performance tuning, complex window functions, and schema design for modern cloud data warehouses.
- Platform Preference: Hands-on experience building pipelines using code or Dataiku (DataIQ).
- Programming: Strong software development fundamentals in Python.
- Communication: Exceptional verbal and written communication skills with a proven track record of interacting with non-technical business stakeholders, gathering requirements, and presenting technical concepts clearly.
Preferred Qualifications
- Experience with cloud platforms (AWS, Azure, or GCP).
- Familiarity with workflow orchestrators like Apache Airflow or dbt.
- Prior exposure to agile development environments and cross-functional teams.
You can also share your profile at [email protected]