We are looking for an experienced Python + PySpark Developer with a strong background in designing, developing, and optimizing large-scale data processing solutions. The ideal candidate should have strong expertise in Python, PySpark, SQL, and big data technologies. You will build the large-scale data pipelines and agentic AI features that power our platform on AWS. You will apply strong Python and PySpark skills to data processing on EMR on EKS, and build LLM-based agents that assist and automate work on our data under human oversight.
• Write and tune PySpark jobs on EMR (EMR on EKS) and Delta Lake to prepare and transform data at scale
• Design and build LLM-powered agents that plan tasks, call tools, and propose or apply actions against our services and data, under confidence thresholds and human-in-the-loop review
• Define the tools and APIs agents can use, and connect agents to internal systems (e.g., via Model Context Protocol)
• Build retrieval-augmented (RAG) pipelines: chunking, embeddings, vector search, and grounding responses in our data
• Design agent workflows with clear guardrails, confidence gating, human-in-the-loop exception handling, and fallback behavior
• Write prompts and evaluation harnesses; run evaluation in CI and measure agent quality, cost, and latency, then iterate
• Build and maintain Python services and shared libraries, and Kafka producers/consumers for moving data between services
• Serve agents behind APIs and monitor them with LLM/agent observability (quality, cost, latency, failures)
• Write design docs and collaborate with engineers and data scientists on architecture decisions
• Use AI coding agents in your own day-to-day workflow while owning correctness and quality
- These are the core skills we screen for. A strong candidate has all three.
- Python 3 — solid experience writing production code, with strong fundamentals and readable, tested code, using a web framework such as FastAPI, Flask, or Django
- PySpark & SQL — hands-on experience writing and tuning Spark jobs on EMR (we run EMR on EKS), reading/writing Delta Lake tables, and strong SQL for large-scale data
- Agentic AI — hands-on experience building LLM or agent-based features: tool/function calling, multi-step agent workflows, RAG, and grounding output in real data.
- Experience writing production Python 3, including designing and optimizing large-scale data processing solutions
- Experience building services or APIs with a Python web framework such as FastAPI, Flask, or Django
- Hands-on experience with Spark / PySpark, ideally on EMR (EMR on EKS a plus)
- Strong SQL and familiarity with big data technologies
- Hands-on experience building LLM or agent-based features (tool calling, function calling, or multi-step agent workflows)
- Experience with an agent or LLM framework or platform such as Amazon Bedrock, LangChain, LangGraph, LlamaIndex, or the OpenAI/Anthropic SDKs
- Experience with RAG: embeddings, vector search, and grounding model output in real data
- Practical prompt engineering and experience evaluating LLM/agent output for quality and reliability
- Experience handling sensitive data and PII responsibly, including data-governance and privacy considerations when sending data to external LLMs
- Experience with at least one database (MongoDB/DocumentDB, PostgreSQL, or similar)
- Experience with AWS services such as S3 and EMR
- Experience writing tests with pytest (or similar) and working with Git
- Comfortable with Docker and Linux
- Practical experience using AI coding assistants or agents (e.g., Kiro, Cursor, GitHub Copilot, Claude Code) in real projects
- Strong communication: able to write design docs and collaborate across teams
- Experience with Model Context Protocol (MCP) or wiring agents to external tools and data sources
- Experience keeping LLM systems reliable: guardrails, evaluation pipelines, cost/latency tuning, caching
- LLM/agent observability and lifecycle experience: prompt/version management, monitoring, and quality regression detection
- Solid ML fundamentals and quality metrics (precision, recall, F1); familiarity with scikit-learn, pandas, NumPy
- Delta Lake, AWS Glue, and Athena
- Kafka streaming pipelines
- Kubernetes / EKS, Helm, and GitOps deployment
- Experience in a polyglot microservice environment (Node.js / Python / Java)
- Awareness of bias and explainability in AI systems