Designs and maintains ETL/ELT pipelines, vector ingestion, and streaming workflows to feed a data platform, ensuring data quality and observability.
- Build ETL and ELT workflows
- Build vector ingestion and embedding pipelines
- Design batch and streaming pipelines
- Develop transformations and orchestration
- Handle late arriving data and schema drift
- Implement CI/CD for data pipelines
- Ingest operational data into data platform
- Maintain data quality and observability
- Manage backfills and replays
- Monitor pipeline health and resolve incidents
- Own data pipelines
- Package data products with contracts and versioning