Senior Data Engineer
The Senior Data Engineer is responsible for collecting, designing, and converting complex data into information that can be interpreted by Data Scientists and Business Analysts. Data accessibility is the ultimate goal, which enables our organization to utilize data for performance evaluation and optimization. As a Senior Data Engineer, one of your primary responsibilities is to guide and mentor junior Data Engineers to ensure they have the necessary skills and knowledge to perform their tasks effectively. This includes setting up guardrails and protocols that help the team achieve their goals efficiently while maintaining the quality of their work. Your role is crucial in ensuring the team's success, as you play a significant part in streamlining their workflow and maximizing their productivity.
What you’ll be doing:
- Implementing pipelines to move raw data in Azure Synapse using spark, python, SQL and C# in line with established architectural standards related to Data Warehouse and Data Lakehouse modeling standards.
- Developing machine learning and regression analysis skills in spark-python-pandas, openai and Azure ML.
- Mentoring other members of the Data Engineering team, specifically those in a junior role
- Work closely with Data Analysts and Business Analysts to clarify their requirements and provide clear guidance on the execution process.
- Providing peer review support to work produced by others, confirming use of relevant coding standards.
- Demonstrate mastery of the Software Development Life Cycle.
What we’re looking for:
- 4+ years of experience in Python development related to data engineering (spark, pandas, etc.).
- 4+ years of experience in SQL related to data engineering.
- Experience in designing and implementing complex data pipelines ensuring data quality & consistency.
- Solid understanding of data warehouse and delta lake design concepts
- Solid data analytics background
- Solid understanding of the Software Development Life Cycle
Bonus Points:
- Microsoft Cloud Certification
- Familiarity with Machine Learning and AI
- Web Development is a plus.
Education:
- Bachelor’s Degree or Relevant Certifications & equivalent years of experience
Travel:
Addendum A – OPS – Product Solutions
In addition to the responsibilities outlined in the standard Data Engineer Job Description, employees assigned to AI Data Engineering functions are responsible for designing, developing, and maintaining data platforms, pipelines, and integrations that support machine learning, generative AI, retrieval augmented generation (RAG), and related technologies. The role partners closely with Data Engineering, AI Engineering, Infrastructure, Security, and Business teams to ensure scalable, reliable, and compliant data solutions.
Responsibilities:
- Design, develop, and maintain scalable batch and streaming data pipelines that ingest, transform, and deliver data for AI and machine learning workloads.
- Build and support ETL/ELT processes using Python, SQL, and other approved technologies to prepare training, testing, and production datasets.
- Support feature store development and management to enable efficient model training and inference.
- Assist in the development and maintenance of data lakes, lakehouses, and data warehouse solutions that support AI and analytics initiatives.
- Integrate and maintain vector databases and vector storage solutions used to support retrieval augmented generation (RAG) and other AI applications.
- Partner with senior engineers and architects to design data architectures, optimize pipeline performance, and implement best practices for AI-ready data platforms.
- Implement and maintain data quality, validation, observability, and monitoring capabilities to ensure data integrity and reliability.
- Support compliance with organizational security, governance, privacy, and regulatory requirements related to data usage and AI systems.
- Contribute to the documentation of data flows, architectures, schemas, and operational processes.
- Work with Model Context Protocol (MCP) integrations, including supporting existing connections and implementing modifications under the guidance of senior team members.
- Collaborate with AI Data Engineering, IT Data Engineering, AI Engineering, Infrastructure, Security, and business stakeholders to support AI-driven products and services.
Additional Basic Qualifications
- Understanding of AI/ML data workflows.
- Proficiency in Python and SQL.
- Understanding of ETL/ELT and data modeling.
- Knowledge of relational and NoSQL databases.
- Ability to support AI-focused data pipelines and data quality practices.
Additional Preferred Qualifications
- Vector database and RAG experience.
- Spark, Kafka, dbt, dlt, Hadoop, or NiFi.
- Cloud data services (Azure Data Factory, AWS Glue, GCP Dataflow).
- PyTorch or other machine learning frameworks.
- Docker, Kubernetes, and CI/CD.
- Experience with MCP integrations.
Travel: