Principle SDET

About the technology organization

Technology at Lilly builds and operates mission-critical digital products and platforms that support the discovery, development, and delivery of medicines that make life better for people around the world. Our teams operate in highly regulated, high-availability environments, where operational excellence, reliability, and quality are non-negotiable.

Within Tech@Lilly, the Digital Core organization applies a product, platform, and reliability-first mindset, ensuring that operational capabilities scale sustainably across the enterprise.

About the Team

Tech@Lilly builds and maintains capabilities using pioneering technologies like the most prominent tech companies. What differentiates Lilly IT is that we redefine what's possible through tech to advance our purpose, creating medicines that make life better for people around the world, including data-driven drug discovery, connected clinical trials, resilient enterprise platforms, and intelligent digital operations. We hire the best technology professionals from a variety of backgrounds, so they can bring an assortment of knowledge, skills, and diverse thinking to deliver creative solutions in every area of our business.

The Digital Core team leads Lilly's transformation into the Digital and AI era. They inspire digitally empowered teams to new ways of working and accelerate innovation and agility. This team powers and advances the entire company by building and maintaining world-class technology capabilities and platforms.

Lilly Capability Centre India (LCCI), Hyderabad, is Lilly's premier Global Technology Hub, harnessing data, AI, analytics, and digital solutions to revolutionize healthcare and improve patient outcomes worldwide.

The Reliability Engineering team is the engineering-first function that owns operations, incident response, runbook automation, and the safe evaluation of agentic remediation across a multi-application production estate. This role sits inside that team and owns test engineering for the work it ships, partnering closely with the peer engineering team that builds the agentic automation platform.

Role summary

As a Principal Software Development Engineer in Test, you own the quality bar for the Reliability Engineering function and act as the shift-left engineering partner across the wider application support organization. You are an engineer first and a tester second. You write code, you build test frameworks, you instrument quality signals into pipelines, and you raise the engineering standard for everything that ships through the team.

This is not a manual QA role and it is not a role that catches defects late. You move quality left into design reviews, code reviews, CI gates, and observability, and you build the automated test machinery that lets a small engineering team ship agentic runbooks, remediation logic, and operational platforms with confidence. You are the engineer the team relies on to make sure what reaches production has been engineered for verifiability from the start.

You partner with the senior reliability engineering leads, the agentic platform engineering team, and the senior architect. You are the single named owner for test strategy across your function, and the consulting voice on quality engineering across the wider organization.

What you'll be doing

1) Shift-left test engineering and design partnership

• Embed in design reviews and architecture conversations before code is written. Raise testability, observability, and failure-mode questions early, and make sure the answers are engineered in, not bolted on.

• Define the test strategy for each platform, runbook, or agent the team ships, including unit, contract, integration, end-to-end, and behavioural tests for agentic and AI-augmented components.

• Set the quality gates that releases must clear in CI, and own the conversation when a gate blocks a release. The gates exist because you defended them.

2) Test automation framework engineering

• Build and maintain the test automation frameworks used across the reliability and agentic engineering teams. Engineers consume what you build; you do not write tests case by case for them.

• Engineer test harnesses for agentic systems: deterministic evaluation of agent outputs, regression suites for prompt and policy changes, golden-set comparison, and replay-based testing for runbook automation.

• Build contract tests between the agentic automation layer, the observability platform, the ITSM system of record, and the applications they act on, so changes on either side surface in CI rather than in production.

• Build chaos and resilience tests that exercise failure paths the team would not otherwise see until an incident.

3) CI/CD quality gating and pipeline ownership

• Own the quality stages in the team's CI/CD pipelines, including static analysis, security scanning hooks, coverage thresholds, performance regression checks, and gating policies.

• Drive shift-left static analysis, security linting, and dependency-risk checks into pull-request workflows, and tune them so they catch real issues without becoming noise.

• Engineer pre-production environments that mirror production sufficiently to validate runbook automation and agentic actions safely, including data masking, synthetic traffic, and replay harnesses.

4) Quality observability and signal engineering

• Treat observability as a testing surface. Engineer test signals into OpenTelemetry instrumentation so the same telemetry that runs production also validates correctness in lower environments.

• Build dashboards and queries (Splunk or equivalent) that surface quality signals — escaped defects, flaky tests, coverage drift, agent decision accuracy — to engineering leads and senior technology leadership.

• Validate SLO instrumentation in partnership with the reliability engineering leads, including burn-rate alerting tests and synthetic probes.

5) Regulated-environment readiness and audit-grade evidence

• Engineer test evidence that holds up under GxP, SOX, and HIPAA scrutiny: traceability from requirement to test to result, repeatable runs, signed artefacts, and change-controlled test assets.

• Partner with the validation and quality functions on computer system validation expectations for operational platforms and agentic remediation, and translate those expectations into automated checks where possible.

• Make sure the test estate itself is treated as code: versioned, reviewed, and deployed through the same pipelines as production software.

6) Quality leadership across the organization

• Act as the consulting quality engineer for the agentic platform engineering team and the broader reliability function when those teams need shift-left advice or framework support.

• Coach engineers in writing the tests that should ship with their code, and raise the engineering bar on test design through code reviews.

• Mentor junior engineers and operational anchors on test automation skills as part of the team's capability uplift.

How you will succeed

At the principal level, success is defined by leverage and by the quality of what reaches production:

• You ship test frameworks and CI machinery that other engineers reuse, not isolated test suites tied to one project.

• Defects that should have been caught in CI are caught in CI; the ones that escape are rare, well-understood, and inform the next gate you build.

• Agentic and automated remediation in production behaves the way the team said it would, because you engineered the evaluation that proved it.

• The team writes more tests, not because they are told to, but because the frameworks you built make doing the right thing the easy thing.

What you should bring

Required

• 10+ years of professional engineering experience, with at least 5 years specifically in test engineering, SDET, or quality engineering roles where you owned automation frameworks and CI quality gates, not just test execution.

• Hands-on engineering background in Python and at least one of Java, TypeScript, or Go, with the kind of fluency where you ship the test framework yourself rather than specify it for someone else.

• Deep, hands-on experience with at least two major test automation stacks (for example: pytest plus a suitable end-to-end framework, Playwright or Selenium for UI, Postman/Newman or REST Assured or schemathesis for APIs, Locust or k6 for performance).

• Strong CI/CD engineering background, including authoring pipelines (GitHub Actions, Azure DevOps, Jenkins, or equivalent), designing quality gates, and tuning them so they remain useful rather than ignored.

• Working fluency with a modern observability and ITSM toolchain — Splunk or equivalent for observability and quality signal surfacing, ServiceNow or equivalent as the system of record, Terraform for infrastructure-as-code, at least one major public cloud (AWS preferred), and OpenTelemetry for instrumentation.

• Demonstrated shift-left practice: participation in design reviews, contract-first development, test-driven or behaviour-driven development at meaningful scale, and a point of view on where shift-left pays off versus where it adds friction.

• Experience operating in a regulated or audited environment (GxP, SOX, HIPAA, PCI, or equivalent), with comfort around change control, audit evidence, and validated-system constraints.

• Bachelor's degree or higher in Computer Science, Information Technology, or a closely related engineering field.

Preferred

• Hands-on experience testing AI-augmented or agentic systems, including evaluation harnesses, golden-set regression, prompt and policy drift detection, and human-in-the-loop validation patterns.

• Experience contributing to chaos and resilience testing programs (AWS Fault Injection Service, Gremlin, LitmusChaos, or equivalent) tied to measurable reliability outcomes.

• Familiarity with computer system validation (CSV) and GxP test documentation expectations in pharma or life sciences technology environments.

• Background in performance and load engineering, including baseline-and-regress practice and the ability to defend performance targets with data.

• Mentoring engineers on test craft and influencing peers without org-chart authority through code, design reviews, and engineering documents.

• Formal certifications welcome but not required, such as ISTQB Advanced, AWS Certified DevOps Engineer, Certified Kubernetes Administrator, or equivalent.

Leadership expectations

• Leads through code, frameworks, and design reviews — not through ceremony.

• Treats test code with the same engineering rigour as production code.

• Holds the quality bar consistently and without it becoming political, even when a deadline is in tension with a gate.

• Brings shift-left thinking into rooms where it is not yet the default, and earns the right to be heard there by doing the engineering work.

• Operates as the named quality-engineering owner for the reliability function, and as the consulting voice on quality across the wider organization.

Additional information

Availability to work flexible work hours is/may be required. This team supports continuous operations and may require non-standard work hours, including some work on weekends and holidays. Appropriate adjustments in benefits will be provided for employees working non-standard hours where applicable.

This is an onsite role based in Hyderabad. Candidates should be open to working with global delivery partners across time zones, with occasional shift flexibility when production-critical validation work requires it.

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form for further assistance.

Lilly does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability, or any other legally protected status.