In the vast realm of Google’s production environment, there exists a hidden world of software engineering efforts that go beyond the consumer-facing products like Gmail or Maps. This realm is the domain of Site Reliability Engineering (SRE), a team tasked with maintaining the uptime and low latency of Google’s complex infrastructure. Within SRE, software engineering plays a crucial role in developing tools to solve internal problems related to keeping production running smoothly.
The Complexity of Google’s Production
Google’s production environment is one of the most intricate systems humanity has ever built. SREs, with their firsthand experience in production intricacies, are uniquely positioned to develop tools tailored for scalability, graceful degradation during failure, and seamless integration with existing infrastructure. Unlike quick hacks, these tools are full-fledged software engineering projects, reflecting a product-based mindset that considers internal customers and future plans.
Why Software Engineering Within SRE Matters
The sheer scale of Google’s production necessitates internal software development, as few third-party tools can match its needs. SREs bring a wealth of Google-specific production knowledge, enabling them to design and create software with scalability and efficiency in mind. The direct relationship between SREs and their users ensures high-signal feedback, facilitating rapid iteration.
From a pragmatic standpoint, SRE-driven software development benefits Google by maintaining a growth rate of supported services that exceeds the growth rate of the SRE organization. This aligns with the principle that “team size should not scale directly with service growth,” emphasizing the need for perpetual automation and streamlining of tools to handle exponential service growth.
On an individual level, SREs benefit from software development projects by providing career development opportunities and preventing coding skills from becoming rusty. Long-term projects offer a balance to interrupts and on-call work, contributing to job satisfaction for engineers seeking a mix of software and systems engineering.
The Birth of Auxon: A Solution to Capacity Planning
Auxon emerged from the minds of SREs and technical program managers tasked with the complex responsibility of capacity planning for Google’s vast infrastructure. Faced with the inefficiencies of manual planning in spreadsheets, the team envisioned a tool that could automate and optimize the allocation of resources based on intent-driven descriptions of service requirements.
The core functionality of Auxon revolves around collecting user intents expressed as requirements for service provisioning. These requirements, whether defined in a user configuration language or through a programmatic API, are translated into machine-parseable constraints. The tool prioritizes and represents these requirements as a giant mixed-integer or linear program, solving it to create a bin packing solution that forms the allocation plan for resources.
Major Components of Auxon:
Auxon’s major components work in harmony to transform user intent into actionable resource allocation plans. The figure below outlines the key components:
Performance Data: This component delves into how a service scales concerning demand and dependency. Scaling data is derived through methods such as load testing or inference based on past performance.
Per-Service Demand Forecast Data: Describing the usage trend for forecasted demand signals, this component allows services to anticipate future usage based on forecasts like queries per second, broken down by continent.
Resource Supply: Providing data about the availability of fundamental resources, the resource supply acts as an upper bound, limiting service growth and placement. The goal is to optimize resource supply based on intent-based descriptions.
Resource Pricing: Offering insights into the cost of fundamental resources, this component factors in global variations based on facility-specific charges. Prices inform overall calculated costs, acting as the objective to be minimized.
Intent Config: This key component defines what constitutes a service and how services relate to one another. Serving as a human-readable and configurable layer, it acts as the linchpin allowing all other components to be seamlessly wired together.
Auxon Configuration Language Engine: Acting upon information from Intent Config, this component formulates machine-readable requests (protocol buffers) for the Auxon Solver. It serves as the gateway between human-configurable intent and machine-parseable optimization requests.
Auxon Solver: Considered the brain of the tool, the solver formulates giant mixed-integer or linear programs based on optimization requests. Designed for scalability, it runs in parallel across hundreds or thousands of machines. It incorporates scheduling, worker management, and decision tree descent.
Allocation Plan: The output of the Auxon Solver, the Allocation Plan prescribes which resources should be allocated to which services in specific locations. It provides implementation details of the intent-based definition, including information on any unmet requirements.
Auxon Case Study: Intent-Based Capacity Planning
Auxon stands as a testament to the power of SRE-driven software engineering. Developed to automate capacity planning for services in Google’s production, Auxon addresses the laborious and imprecise nature of traditional approaches to capacity planning. It introduces the concept of Intent-Based Capacity Planning, emphasizing specifying requirements rather than implementation details.
The chain of abstraction, from explicit resource requests to the true intent behind these requests, involves capturing dependencies, performance metrics, and prioritization. Auxon, as an implementation of intent-based planning, collects user intents through a configuration language or programmatic API, translating human intent into machine-parseable constraints. It then formulates a giant mixed-integer or linear program, solving it to generate an allocation plan for resources.
Designing and Developing Auxon: Successes and Lessons Learned
Auxon’s success can be attributed to several key principles, including agnosticism – designing the software to be generalized and adaptable to myriad data sources. This approach allows Auxon to remain of sufficient general utility across diverse use cases. The team’s commitment to providing customer service, even to highly technical users, played a pivotal role in driving adoption.
The importance of socializing internal software tools, user advocacy, and gaining sponsorship from senior engineers and management is highlighted. A consistent and coherent approach, along with white glove customer support, ensures successful adoption and integration into the organization.
Team Dynamics: Fostering a Software Engineering Culture
Building a software engineering culture within SRE involves selecting a diverse team with a mix of generalists and specialists. A seed team with engineers capable of quickly adapting to new topics, coupled with specialists in relevant domains, helps cover blind spots and ensures a well-rounded perspective. Engaging specialists at the right time, such as during the later phases of development, enhances the project’s overall success.
In conclusion, software engineering within Google SRE has evolved into a crucial aspect of maintaining Google’s vast production environment. Projects like Auxon showcase the innovative approaches that stem from hands-on production experience. SRE-driven software projects contribute to a sustainable model for supporting services at scale, preventing linear growth in SRE teams. The unique blend of production experience and software development skills in SREs allows for the creation of tools that streamline processes, automate tasks, and ultimately contribute to the company’s success. The benefits extend to the SRE organization, ensuring a balance between software and systems engineering, and individual SREs, offering career development opportunities and job satisfaction.
As Google continues to grow, the lessons learned from successful software engineering projects pave the way for future endeavors, making SRE-driven software development an integral part of Google’s reliability and success.
Do you like to read more educational content? Read our blogs at Cloudastra Technologies or contact us for business enquiry at Cloudastra Contact Us .
As your trusted technology consultant , we are here to assist you.