On The State Of Continuous Profiling
What is continuous profiling and where are we with it, end of 2021?
A month ago I asked via Twitter about signal types uptake (in the context of Kubernetes):
The answer distribution was not overly surprising but one thing stood out to me: just shy of 1/5 of the folks who responded said they, in addition to logs, metrics, and traces, they’d also “capture profiles/eBPF”.
Alrighty, what does that even mean? :)
In 2010, Gang Ren, Eric Tune, Tipp Moseley, Yixin Shi, Silvius Rus, and Robert Hundt published Google-Wide Profiling: A Continuous Profiling Infrastructure for Data Centers in the IEEE Micro journal, summarizing the architecture with:
The core idea of continuous profiling (CP) is to capture CPU, memory, I/O, and networking usage over time, connecting it with the code base in a manner so that it’s possible to directly locate a certain line of source code.
With CP it’s possible to address a number of use cases from saving infrastructure costs to improving application performance. Increasingly, we see CP utilizing eBPF to capture the low-level values (CPU, RAM, etc.) at runtime and making them available to higher-level frameworks and/or for visualization.
Interestingly, for almost a decade after the Google engineers published the IEEE Micro article, the topic did not go mainstream. 2021, however, was the year when–seemingly out of nowhere–a number of projects are aiming to deliver in open source what the Google engineers described back then. Let’s have a closer look at those offerings.
Open Source Offerings
In alphabetical order, we now review existing open source CP offerings.
Parca by PolarSignals
Parca—with strong Prometheus roots, eBPF-based, for single node and Kubernetes environments. Building on the Prometheus labels and service discovery concepts, easy to set up and use.
CNCF Pixie
Pixie—all-in-one o11y solution based on eBPF that offers, amongst other things, profiling. Powerful and enables to write own eBPF programs, with focus on Kubernetes.
Pyroscope by Pyroscope, Inc.
Pyroscope— comes with wide language support (Java, Ruby, Python, .NET, etc.), eBPF integration, and advanced visualizations.
Further reading on the topic:
- Prometheus Pitfalls and the Rise of Continuous Profiling
(Open Observability podcast episode, 12/2021) - Gathering insights on Kubernetes applications, services, and network traffic with Pixie
(AWS open source blog, 05/2021) - What does the future hold for observability?
(KubeCon keynote, 2019) - Continuous Profiling in Production: What, Why and How
(QCon talk, 2019) - See also vendor-specific profiling offerings, for example, by Dynatrace, Datadog, or Google Cloud.
