Senior Site Reliability Engineer, Databases - remote

Grafana Labs

Posted 3 months ago • United States, Canada

This is a remote position and we’re considering candidates in the USA &Canada.

About the role:

We are looking for a Senior SRE to help us support our highest value Grafana Cloud customers by increasing the reliability of our Cloud databases that are based on Mimir, Loki, Tempo, and Pyroscope. We provide these databases as a SaaS product from AWS, GCP, and Azure across all regions.

The SRE team is a new team within the Databases department, that owns the environments (customer and product cells) for our largest customers, and acts as an overlay to existing teams that run the databases within the system. As an SRE within the team, you own the configuration of the software via Helm charts and Jsonnet, being involved with the PRR for new features, shepherding releases to the environment and ensuring new releases do not degrade the SLOs or user experience for the customer (learn what is special about each of these customers, and mitigate risks that might be produced by a change in the software), directly contributing design docs, code, PR review, and other engineering activities to the databases to further improve reliability for the customer, observability of the customer stack, and making recommendations to customers about their use of the system to further improve reliability.

Like all SRE roles there is an on-call element, unlike other roles this one is a shared pager where “if the Mimir team are paged for this customer, then we are also paged”, this allows you to focus your response on the experience the customer has, whilst also being supported by another on-call engineer who will focus on the system. As a company, we hire globally (remote-only) to ensure our on-call is as healthy as possible, and aligned to 12 daylight hours per day as the default.

What we seek:

Strong engineering background (at least 6 years), that lean towards SRE roles (at least 3 years)
- This may encompass but is not limited to experience as a reliability/production engineer, infrastructure/systems engineer, or software engineer with an infrastructure/systems focus.
Good communication, capable of engaging in deep technical conversations with other engineers and customers, and collaborating across organizational boundaries
Experience with Kubernetes on any of AWS, GCP, or Azure, and working with Helm charts or other IaC tools.
Experience with Site Reliability Engineering, Large System Design, and Distributed Computing
Experience with one or more programming languages (e.g. Go, Python, Java, etc)
Experience with Linux operating systems internals, and some knowledge of networking, cloud storage, and scaling.
Excellent problem-solving and troubleshooting skills.
Experience with calmly and actively participating in blame-free Incident Response, following up on actions, and writing high quality PIRs (Post Incident Reviews, a.k.a. post-mortem documents)
Comfortable working within an engineering team where individuals are encouraged to have a strong sense of autonomy and self-direction.
We highly value those who are intellectually curious, who default to transparency, possess a high bias towards action, and who are also kind (this is important)

Your day-to-day will include:

- Regular 1:1s to with your manager and colleagues
- Reviewing and creating SLOs, proactively investigating ways in which we can further reduce budget burn for those SLOs, which can be self-directed or as the result of learnings from incidents, and may include improvements to monitoring, automation, increasing self-healing, auto-scaling, etc.
- Improve observability of customers within their environments
- Designing and implementing solutions to ensure reliability and scalability of our environments can meet rapidly increasing demands
- Develop fault-tolerant design patterns ensuring that we are considering reliability at all stages of the service lifecycle.
- Collaborating with our Engineering Leaders to help define and influence product strategy, roadmaps and technical designs
- Participate in PR review and collaborating with other engineers on their Design Docs
- Teach others about Site Reliability Engineering and communicate best practices to be applied early in development of new features and functionality
- Participate in Incident Response when applicable, including investigation through to resolution, PIR, and communication with customers via Bridge calls where necessary

In the US, the base compensation range for this role is $148,505 –$178,206. Actual compensation may vary based on level, experience, and skillset as assessed in the interview process. Benefits include equity, bonus (if applicable) and other benefits listed here.

About Grafana Labs: There are more than 1,000,000 active installations of Grafana around the globe, monitoring everything from beehives to climate change in the Alps. The instantly recognizable dashboards have been spotted everywhere from a NASA launch and Minecraft HQ to Wimbledon and the Tour de France. Grafana Labs also helps companies including Bloomberg, JPMorgan Chase, and eBay manage their observability strategies with full-stack offerings that can be run fully managed with Grafana Cloud, or self-managed with Grafana Enterprise Stack. The Grafana stack has grown to include four other open source projects, Grafana Loki (for logs), Grafana Tempo (for traces), Grafana Mimir (for metrics), and Grafana OnCall (for on-call management).

Benefits: For more information about the perks and benefits of working at Grafana, please check out our careers page.

A note about covid-19: All Grafanistas who wish to attend in-person events or travel for Grafana Labs must be fully-vaccinated.

Equal Opportunity Employer: At Grafana Labs we’re building a company where a diverse mix of talented people want to come, stay, and do their best work. We know that our company runs on the hard work and the dedication of our passionate and creative employees. If you’re excited about this role but your experience doesn’t align perfectly with every qualification in the job description, we encourage you to apply anyways.

We will recruit, train, compensate and promote regardless of race, religion, colour, national origin, gender, disability, age, veteran status, and all the other fascinating characteristics that make us different and unique. We believe that equality and diversity builds a strong organisation and we’re working hard to make sure that’s the foundation of our organisation as we grow.

For information about how your personal data is used once you’ve applied to a job, check out our privacy policy.

About Grafana Labs: There are more than 20M users of Grafana, the open source visualization tool, around the globe, monitoring everything from beehives to climate change in the Alps. The instantly recognizable dashboards have been spotted everywhere from a NASA launch and Minecraft HQ to Wimbledon and the Tour de France. Grafana Labs also helps more than 3,000 companies —including Bloomberg, JPMorgan Chase, and eBay —manage their observability strategies with the Grafana LGTM Stack, which can be run fully managed with Grafana Cloud or self-managed with the Grafana Enterprise Stack, both featuring scalable metrics (Grafana Mimir), logs (Grafana Loki), and traces (Grafana Tempo).

Benefits: For more information about the perks and benefits of working at Grafana, please check out our careers page.

We will recruit, train, compensate and promote regardless of race, religion, color, national origin, gender, disability, age, veteran status, and all the other fascinating characteristics that make us different and unique. We believe that equality and diversity builds a strong organization and we’re working hard to make sure that’s the foundation of our organization as we grow.

For information about how your personal data is used once you’ve applied to a job, check out our privacy policy.

Apply