As a Site Reliability Engineer, your primary responsibility will be to review, optimize, and complete the monitoring and alerting systems for our applications. You will work closely with development, operations, and product teams to ensure that our monitoring systems provide clear, actionable data and that our alerting mechanisms are finely tuned to detect issues before they impact our customers. Your work will be pivotal in transforming raw data into actionable intelligence, improving system observability, and enhancing the overall user experience.
RESPONSIBILITIES AND DUTIES:
- Monitoring and Observability: Evaluate existing monitoring systems and implement improvements to ensure comprehensive observability across all systems and environments. Develop and maintain dashboards and reports that provide real-time visibility into system health, capacity/utilization trends, and performance.
- User Experience: Ensure that the overall system environment operates nominally by monitoring critical performance indicators. Provide insights into system status that help maintain a smooth and uninterrupted user experience.
- Alerting Optimization: Review and refine alerting mechanisms to minimize false positives and ensure timely and accurate notifications for critical issues. Develop escalation processes and response playbooks to streamline incident management.
- Data Analysis and Insights: Analyze monitoring data to identify trends, anomalies, and potential areas of improvement. Provide actionable insights to relevant teams and drive data-driven decision-making leveraging machine learning and normal versus abnormal system behaviors.
- Collaboration: Work closely with software engineers, DevOps teams, and other stakeholders to ensure monitoring and alerting systems are aligned with business goals and technical requirements.
- Automation and Tooling: Develop and maintain automation scripts and tools to streamline monitoring and alerting processes, reducing manual effort and improving efficiency.
- Documentation and Training: Document monitoring and alerting systems, processes, and best practices. Provide training and guidance to teams on how to use monitoring tools and interpret data.
- Continuous Improvement: Continuously assess and improve monitoring and alerting strategies to adapt to changing technologies and business needs. Stay updated with industry trends and emerging tools in the observability space.
KNOWLEDGE, SKILLS, AND ABILITIES:
Strong experience with monitoring and observability tools (e.g., Nagios, Prometheus, Grafana, ELK Stack, Datadog, New Relic).
Proficiency in scripting languages (e.g., Python, Bash, PowerShell) for automation.
Familiarity with cloud platforms (AWS, Azure, GCP) and hybrid cloud environments.
Understanding of infrastructure-as-code tools (e.g., Terraform, Ansible).
Knowledge of CI/CD pipelines and version control systems (e.g., Git, Jenkins).
Basic understanding of networking, security, and system administration.
EDUCATION AND EXPERIENCE:
Bachelor's degree in Computer Science, Engineering, a related field, or equivalent experience.
Minimum of 3 years of experience in a Site Reliability Engineering or similar role, with a focus on monitoring and alerting in a SaaS environment.
WORK ENVIRONMENT AND PHYSICAL DEMANDS:
Normal office environment with use of computers and telephone systems; no unusual physical demands.
Travel as needed, including business air travel and car rental.