We are seeking a Senior Software Engineer to support the design, development, and sustainment of a comprehensive Metrics and Monitoring solution for High-Performance Computing (HPC) systems. The ideal candidate will help build and maintain scalable monitoring capabilities that improve the reliability, performance, and operational visibility of complex HPC environments. This role requires strong experience with Splunk, automation, data analysis, and software development. The successful candidate will collaborate with engineering teams to develop monitoring solutions, optimize system performance, and support mission-critical infrastructure.
What you'll do
Design, develop, and maintain distributed metrics and monitoring solutions for HPC systems.
Develop software solutions using Python and automation scripting.
Build and optimize Splunk dashboards, reports, alerts, and data models.
Perform Splunk data onboarding, field extractions, performance tuning, and troubleshooting.
Develop and maintain monitoring workflows, including alerting, incident management, and event correlation.
Automate operational processes using scripting languages such as Shell and Perl.
Integrate and manage log data from tools such as Syslog-ng and rsyslog.
Support message-driven architectures using technologies such as Kafka.
Implement Infrastructure as Code (IaC) solutions using automation tools such as Ansible.
Support CI/CD pipelines and software delivery practices.
Collaborate with engineering teams to improve system reliability, scalability, and performance.
Qualifications
U.S. Citizenship with an active TS/SCI clearance with Polygraph.
One of the following education/experience combinations:
Master’s degree in Computer Science or a related discipline plus 10+ years of Software Engineering experience.
Bachelor’s degree in Computer Science or a related discipline plus 12+ years of Software Engineering experience.
14+ years of Software Engineering experience supporting programs and contracts of similar scope, type, and complexity.
Experience working with Linux command-line tools and environments.
Experience developing automation scripts using Shell or Perl.
Recent software development experience using Python.
Strong experience with Splunk, including:
SPL development
Dashboard creation
Report development
Data onboarding
Field extractions
Data models
Performance tuning
Alerting and event correlation
Experience interfacing with log management tools such as Syslog-ng and rsyslog.
Familiarity with message brokers such as Kafka.
Experience with Infrastructure as Code (IaC) concepts and automation tools such as Ansible.
Experience with CI/CD methodologies and tools, including GitLab CI.
Experience using Git source control.
Preferred Qualifications
Experience with the Atlassian Tool Suite, including Jira and Confluence.
Experience supporting HPC environments, distributed systems, and enterprise-scale monitoring solutions.
Experience developing operational dashboards and analytics for large-scale computing environments.
Familiarity with Agile software development practices and DevOps methodologies. the experience and attributes of the ideal candidate