Information Technology - Lead Data Engineer
SINGAPORE AIRLINES LIMITED · Singapore ·
- Seniority
- Lead
- Category
- Data engineering
SINGAPORE AIRLINES LIMITED · Singapore ·
gsbsolutions1 · Jalisco, Mexico
慨正橡扯 · Swansea / Abertawe, United Kingdom
First Citizens Bank · NCSDAC-Data Center
GSB Solutions · Jalisco, Mexico
Lead Data Engineer builds and maintains open-source data ingestion and MLOps platforms using Python, Spark, Kafka, and Databricks to deploy ML models at scale.
Job Description
The lead data engineer is a senior software developer with strong softwareengineering skills who is responsible for building custom open-source-baseddata ingestion and MLOps platforms. He/she has deep appreciation of thecomplexity of the data engineering process, such as the challenges of dataingestion involving large or near-real-time datasets, the maintenance of highdata quality, and the importance of automation for increasing pipelinerobustness and reducing the need for human intervention.
Responsibilities
• Be an effective distributed-system implementer in the following coreactivities:
o Design and develop data engineering services and their ecosystem usingdistributed databases (relational, columnar, graph, in-memory); orchestration(Apache Airflow); and distributed stream/batch data processing (Kafka, Kinesis,Spark).
oDesign and develop MLOps production pipelines; provide technical support todata scientists/ML engineers by getting their ML/DL models deployed at scaleand meeting SLAs on both cloud and on-premises GPU and CPU instances.
o Design data models for mission-critical, high-volume, near-real-time/batchdata; build idempotent/atomic production data pipelines to make data ingestionmore fault tolerant.
o Design and develop intuitive, highly automated, self-service data platformfunctions for business users.
o Design, build, and operate scalable and reliable data pipelines on theDatabricks platform.
• Explore, evaluate and champion the introduction of next-generationtechnologies in the data-ingestion workflow. Participate in project planningand provide technical guidance on cloud architecture for data projects.
Requirements
• BS in Computer Science or other related discipline is required. Advanceddegrees in Computer Science (PhD, MS) are highly desirable.
•5+ years of relevant industry experience in some or most of the followingtechnical areas:
o Advanced programming skills in Python. Conversant with data structures andalgorithm design.
o Experience in building data pipelines (including data collection,warehousing, processing, analysis, monitoring, and governance) usingopen-source data ingestion platforms.
o Intermediate-level knowledge and experience with AWS cloud components andbest practices. Good understanding in deploying data stores such as S3,RedShift, Elasticache, PostgreSQL, and EMR.
o Hands on experience with Databricks workspace, cluster management, AI Agentcapabilities, and job orchestration
o Prior experience in modern software development is required (such as webfrontend UI, backend API microservices, understanding of CI/CD and Scrum/Kanbanagile development). Strong grasp on object-oriented or functional programming(using e.g. Python, Java, Scala, or C#).
srijantech · Gurgaon