Staff Site Reliability Engineer - Linux, Observability, Containers
We are seeking an experienced Staff Site Reliability Engineer (SRE) to join our dynamic Observability team. In this role, you will play a critical part in building and maintaining reliable, secure, and cost-effective observability solutions that support our global infrastructure.
As a Staff SRE, you will act as a key liaison between development and operations, ensuring our systems and services are efficient, scalable, and highly available. You will work closely with software engineers, system administrators, and other technical stakeholders to automate workflows, enhance system performance, and streamline deployments.
You will take ownership of your solutions, prioritizing system stability, performance, and security. You will design, implement, and improve monitoring, logging, and alerting systems to proactively identify and resolve issues. You will also participate in incident response, troubleshooting live issues, performing root cause analysis, and contributing to preventive measures.
In this position, you will mentor and support junior SREs and collaborate within Agile Scrum teams, leveraging your technical expertise in coding, scripting, testing, and debugging to deliver high-quality results. You will help drive continuous improvement in processes, tools, and automation across the organization.
Job Description
- Design, deploy, and optimize instrumentation for comprehensive monitoring and logging across distributed systems.
- Architect and implement automation to drive operational efficiency, reduce toil, and streamline system integration.
- Lead the support and management of observability platforms and tools, including Splunk, ClickHouse, Grafana, Prometheus, M3DB, OpenTelemetry, Fluent Bit, ElasticSearch, OpenSearch, and CloudWatch.
- Collaborate with development, operations, and security teams to design scalable monitoring solutions and assist in building robust development, staging, and production environments.
- Ensure system security and compliance by proactively applying hotfixes, patches, and hardening measures.
- Champion DevOps and SRE best practices across engineering teams, fostering a culture of reliability and continuous improvement.
- Design, build, and maintain CI CD pipelines, enabling rapid, automated, and reliable deployments.
- Oversee and optimize cloud infrastructure (AWS, GCP) to ensure high availability, scalability, performance, and security.
- Develop and manage infrastructure as code using tools such as Terraform, Ansible, or CloudFormation.
- Continuously monitor system health, analyze performance metrics, and implement solutions to improve reliability, efficiency, and scalability.
- Deploy, manage, and optimize containerization and orchestration technologies, including Docker and Kubernetes.
- Conduct root cause analysis on major incidents, lead postmortems, and drive implementation of preventative and corrective measures.
- Maintain and enhance comprehensive documentation for infrastructure, processes, and operational procedures.
- Provide advanced technical support and mentorship to resolve complex infrastructure and deployment issues.
- Identify and automate manual processes to further improve operational workflows and reduce human error.
- Lead incident response efforts, including on-call rotations, and drive improvements to incident management processes.
- Evaluate, recommend, and implement new tools and technologies that advance the organization’s reliability and observability capabilities.
- Collaborate with cross-functional teams on disaster recovery planning, business continuity, and capacity planning initiatives.
- Present technical findings, recommendations, and status updates to leadership and stakeholders.
- Foster knowledge sharing and best practice adoption through training, code reviews, and documentation initiatives.
Qualifications Exp. Level : 8-11 Years
Basic Qualifications:
- Bachelor’s degree with a 8-11 years of relevant professional experience.
Preferred Qualifications:
- Proven experience performing upgrades and maintenance for observability tools such as Splunk Enterprise and Universal Forwarder, Fluent Bit, Prometheus container images, Grafana, ElasticSearch, OpenSearch, Kibana, and ClickHouse.
- Demonstrated expertise in configuring and deploying exporters (e.g., Node Exporter, Cert Exporter, and others) for metrics collection.
- Extensive hands-on experience with containerization and orchestration platforms, including Docker and Kubernetes.
- Deep proficiency with a range of observability tools such as Splunk, ClickHouse, Grafana, Prometheus, M3DB, OpenTelemetry, Fluent Bit, ElasticSearch, OpenSearch, and CloudWatch.
- Strong experience managing CI CD pipelines using tools like GitHub and Ansible.
- Proficiency with Infrastructure as Code tools (e.g., Terraform) and configuration management practices (e.g., GitOps).
- Advanced scripting skills in Python and or Shell scripting.
- Working knowledge of query languages such as PromQL, MS SQL, or Splunk SPL is highly desirable.
- Solid background working in Linux environments and Unix shell scripting.
- GCP or AWS cloud certifications are a strong plus.
- Excellent analytical skills, with the ability to break down complex technical problems and communicate solutions at varying levels of detail for different stakeholders.
- Outstanding communication, collaboration, and leadership abilities, with a track record of mentoring and guiding technical teams.
Bachelor’s degree with a 8-11 years of relevant professional experience.