Data Engineer — ML Training Data Pipeline

What We Expect:

• Build end-to-end data pipelines: raw trace ingestion → dedup → format conversion → quality gating → training-ready datasets
  
• Process large-scale JSONL data on AWS S3 (tens of thousands of traces per batch)
  
• Convert between chat-completion formats (e.g., OpenAI → Llama 3.1 tool-calling format)
  
• Implement smart deduplication and sampling to balance training distribution
  
• Design identity-aware train/test splits that measure true generalization
  
• Build data validation gates to detect schema drift and format anomalies
  
• Create a continuous pipeline that auto-processes new production traces for retraining
  

Requirements

• Experience: 6+ years data engineering focused on ML data pipelines
  
• Python: Strong — pandas, pyarrow, JSONL processing at scale
  
• ML Data Libraries: HuggingFace Datasets, Arrow-based storage
  
• Data Formats: Multi-turn conversation/chat data structures and tokenizer-specific formatting
  
• Deduplication: Content hashing, identity-based grouping strategies
  
• AWS: S3, EC2, batch processing workflows
  

Preferred (Not Required): LLM training data prep (chat templates, tool-calling schemas); Axolotl or similar dataset formats; data versioning (DVC, LakeFS); browser-automation trace data or Playwright.

Benefits

• Comprehensive Medical Coverage:
  Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.
• Robust Protection Plans:
  Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.
• Retirement Benefits:
  PF and Gratuity provided as per standard government regulations.
• Flexible Work Options:
  Enjoy hybrid work arrangements & flexible working hours.
• Generous Leave Policy:
  21 days of annual leave, in addition to 10 company-declared holidays.
• Employee Well-being Spaces:
  Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.