Robotics Software Engineer, Test Infrastructure (SITL/HITL)

About The Team

The Sentry Tower Software team develops robotic systems that provide force protection capabilities, monitoring the perimeter of secure areas, land or sea, for approaching people, vehicles, and vessels. We live in a world where security officers are increasingly overwhelmed by sensor data feeds. Our products leverage advanced sensor fusion and autonomy to seamlessly render activity in the environment to Lattice’s common operating picture.

About the Job

Sentry Towers are robotic systems: motors, cameras, radars, radios, and sensor boards, running on NVIDIA Jetson compute in the field. Verifying that a system like this works means accounting for physical actuation, real sensor data, and hardware that fails in ways no unit test anticipates. We’re hiring a Senior Software Engineer to own our simulation and hardware test infrastructure and make automated, whole-system testing the default for everything we ship. A regression that escapes here does not surface on a developer’s laptop. It surfaces on a tower at a site someone has to fly to, guarding a perimeter that does not pause while we debug.

You will work across both halves of the problem. Our software-in-the-loop environments are declaratively defined multi-node virtual towers, complete with a simulated ground control station, synthetic sensor feeds, and a game-engine-driven world, which lets us run whole-system scenarios in CI with no hardware attached. Our hardware-in-the-loop side drives real Jetson-based towers through a board test server: provisioning them over the network, flashing firmware, installing full system images, and reporting results back through our CI pipeline on a nightly and weekend cadence.

Both foundations are in place and working, so you are inheriting infrastructure rather than starting from nothing. What does not exist yet is most of what sits on top: the functional test suites, the scenario library, and the diagnostic tooling that turn a booting tower into a verified one are yours to design. Firmware integration, compute platform bring-up, and both our Air Defense and Counter-Intrusion business lines depend on this capability, and what you build becomes the standard they test against.

Test infrastructure is a job where the wins are invisible and the failures are loud. You will spend real time on work nobody asks for by name: chasing intermittent failures to root cause instead of retrying them, waiting on hardware that is scarce and shared, and rebuilding coverage when a compute generation changes underneath you. The measure of the role is bugs that never reach a tower, which means much of your best work leaves no trace anyone thanks you for.

Our Product:

WHAT YOU’LL DO

• Own and grow our SITL environments: add product variants and scenarios, keep the simulated sensor and video pipelines faithful enough to be trustworthy, and keep whole-system test runtimes short enough to fit inside a normal development cycle.
• Build out functional coverage on the HITL bench across the behavior that matters: motion control and trajectory execution, camera and sensor bring-up, firmware update paths, network failover, and clean recovery from power loss and unexpected reboots.
• Own the CI integration end to end: scheduling, dependency chaining between test stages, artifact and log capture, result reporting, and the flake triage that keeps a hardware pipeline trusted instead of ignored.
• Grow the test platform continuously as the product line does. New compute generations, new payloads, and new product variants each need coverage, and keeping the framework ahead of the roadmap is the enduring core of this role.
• Make failures diagnosable without touching the rig. Invest in log capture, serial console capture, and test artifacts so an engineer can root-cause from the CI output alone.
• Partner with hardware and manufacturing engineers to stand up the physical HITL benches, specifying the fixtures, switching, power control, and instrumentation needed to exercise towers unattended. This is a substantial upfront build with periodic expansion as new hardware arrives.
• Partner with firmware, platform, and business line teams so they can add their own tests against your framework, and support them as their first users.

REQUIRED QUALIFICATIONS

• 4+ years of professional software engineering, with meaningful time spent building test infrastructure, CI systems, or simulation for systems that involve real hardware.
• Proficiency in Python for test authoring and automation, plus working ability in Rust or C++ to read and debug the services under test.
• Demonstrated experience automating tests against physical hardware: device provisioning, remote power and reset control, serial consoles, network booting, or instrumented benches.
• Strong CI/CD engineering skills, including pipelines with expensive stages, hardware resource contention, and scheduled rather than per-commit execution.
• Experience with declarative configuration and reproducible builds. Nix or NixOS is what we use and what we most want to see; adjacent depth in Bazel, Buildroot, Yocto, or infrastructure-as-code tooling (Terraform, Ansible, Pulumi) is a solid substitute.
• Solid Linux systems fundamentals: systemd, networking, filesystems, and boot, since most of what you automate is a Linux system misbehaving.
• The judgment to know which tests are worth their cost, and the discipline to fix or delete a flaky test rather than tolerate it.
• US person status required, with eligibility to obtain and maintain a US security clearance.

NICE TO HAVE

• Experience with robotics or vehicle simulation, whether game-engine-based (Unreal, Unity), Gazebo, Isaac Sim, or a custom in-house simulator.
• Familiarity with NVIDIA Jetson provisioning and flashing workflows.
• Experience with virtualization and VM-based integration testing, including GPU passthrough.
• Background in hardware test engineering, HIL rigs, or manufacturing test systems.
• Familiarity with CAN, serial protocols, or bus-level diagnostic tooling.