We compare several concrete approaches and projects: eBPF-based auto-instrumentation using OpenTelemetry’s Go auto-instrumentation agent and OBI (OpenTelemetry eBPF Instrumentation), compile-time manipulation using tools like Orchestrion and the OpenTelemetry Compile-Time Instrumentation SIG, runtime injection via Frida/ptrace, and USDT (User Statically-Defined Tracing) probes — both via libstapsdt and a custom Go runtime fork.

Beyond what exists today, we look at how ongoing work in the Go runtime and diagnostics tooling could unlock cleaner, safer hooks for future auto-instrumentation, including flight recording proposals and native USDT support in the Go toolchain.

Throughout the talk, we use benchmark results and small, realistic services to compare these strategies along three axes: performance overhead (latency, allocations, CPU impact), robustness and upgradeability across Go versions and container images, and operational friction (rollout complexity, debugging, and failure modes).

Recording

Slides

• Slides - Markdown

Demo/Code

• fosdem-2026

Events

• FOSDEM 2026 — Go Devroom
  • Recording

Related

• Auto-Instrumenting Go: From eBPF to USDT Probes — full technical blog post expanding on this talk

In this talk, we walk through the full stack of reliable performance measurement — from controlling your benchmarking environment (bare metal instances, CPU affinity, disabling SMT and dynamic frequency scaling) to designing benchmarks that are both representative and repeatable. We cover the statistical methods needed to interpret results correctly (hypothesis testing, change point detection) and show how to integrate continuous benchmarking into development workflows so regressions are caught before they reach production.

Along the way, we share experiments demonstrating how environment control alone can reduce measurement variance by 100x, and practical tips for anyone who writes benchmarks — whether you’re optimizing a hot loop or validating a system-wide change.

Co-presented with Augusto de Oliveira.

Recording

Slides

• Slides - Markdown

Demo/Code

• fosdem-2026-software-performance — includes Jupyter notebooks in experiments/ with benchmark design and results interpretation visualizations

Events

• FOSDEM 2026 — Software Performance Devroom
  • Recording

Related

• Measuring Software Performance: Why Your Benchmarks Are Probably Lying — full technical blog post expanding on this talk

You’ll see how platform and tooling teams use -toolexec to weave organisation-wide practices directly into the build, from enforcing error-handling standards to automatically adding observability hooks. We’ll map the journey from a “hello-world” wrapper to full Aspect-Oriented compile-time transformations, and discuss the trade-offs that come with this new power.

Along the way we’ll spotlight real projects—such as Datadog’s Orchestrion—and community efforts in compile-time instrumentation, showing what’s already possible and where things are heading. Finally, we’ll share practical tips for keeping builds fast and reliable when you venture beyond the default toolchain path.

If you’re curious about bending the Go compiler to your will—and doing so responsibly—this talk will equip you with the concepts, examples, and inspiration to start experimenting the moment you’re back at your editor.

Recording

Slides

• Unleashing the Go Toolchain
  • Slides - Markdown

Demo/Code

• gopherconuk25-demo

Events

• GopherCon UK 2025
  • Recording

Building observable applications begins with proper instrumentation. While Prometheus tooling simplifies this process, there are still numerous opportunities for mistakes or misuse.

In this talk, Jéssica Lins and Kemal Akkoyun present several useful patterns, best practices, and idiomatic methods for instrumenting critical services. They discuss common pitfalls, failure cases, and instrumentation strategies, sharing valuable insights and methods to avoid these mistakes. Additionally, they provide tips for writing simple, maintainable, and robust instrumentation facilities using real-life examples. The talk also demonstrates how to enrich metrics by correlating them with other observability signals and discusses how to best use recent changes in client_golang, the Go client library for Prometheus.

Recording

Slides

• Best Practices and Pitfalls of Instrumenting Your Cloud-Native Application

Events

• PromCon EU 2022
  • Recording

Recording

Slides

• Story of Correlation: Integrating Thanos Metrics with Observability Signals

Events

• KubeCon EU 2022
  • Recording

Recording

Slides

• eBPF? Safety First!

Events

• Cloud-Native eBPF Day EU 2022
  • Recording

Profiling has long been part of the developer’s toolbox to analyze the resource usage of a running process. Go users are very familiar with the concept thanks to state-of-art Go tooling. For years Google has consistently been able to cut down multiple percentage points in their fleet-wide resource usage every quarter, using techniques described in their “Google-Wide Profiling” paper, which is called continuous profiling. Through continuous profiling, the systematic collection of profiles, entirely new workflows suddenly become possible.

In parallel, eBPF became a new promising technology, is likely not news to most people in cloud space. We are discovering more use cases where eBPF can be useful, especially when combined with Go and modern infrastructure, from security, over observability to performance tuning. For a long time, eBPF has struggled with portability, it needed to be compiled for each kernel, or a compiler and kernel headers needed to be shipped to execute effectively arbitrary code. The eBPF community acknowledged this and started the CO:RE (compile once-run everywhere) initiative, which is young but quickly maturing in the form of libbpf and libbpf-go.

In this talk, we will bring these two concepts together, and explain how to write portable eBPF programs and embed them in Go applications. And what libbpf-go does in order to achieve compile once-run everywhere, how it can be used in portable Go applications. We will demonstrate all concepts together by using real-life examples to help measure and improve performance systematically.

Recording

Demo Application

Slides

• Building a Go Profiler Using Go

Events

• GopherCon EU 2022
  • Recording

Recording

Slides

• Achieving Zero-Instrumentation Monitoring with eBPF

Events

• PrometheusDay North America 2022
  • Recording

Recording

Slides

• Parca - Profiling in the Cloud-Native Era

Events

• KubeCon NA 2021
  • Recording

The presenter will start this talk with an introduction to profiling applications, and demonstrate how one can practice it using open-source continuous profiling tools, and how continuous profiling allows for an unprecedented fleet-wide understanding of code at production runtime.

Attendees will learn how to continuously profile their code, guide themselves in building robust, reliable, and performant software and reduce cloud spending systematically.

Recording

Slides

• Profiling Go Applications in the Cloud-Native Era

Events

• GopherCon Turkey 2021
  • Recording

There exist several observability signals, such as “log”, “metric”, “tracing” and “profiling” that can be collected from a running service, which we can also call pillars of observability. Using these signals, we can create real-time, actionable alerts, create panels where we can monitor applications closely, and perform in-depth analysis to find the root of the systems’ failures. Within the Go and CNCF community, there are a variety of tools that can collect and make these observable signals useful.

During this talk, Kemal will first introduce the tools that can be embedded in the services to make critical services observable, and share the patterns that will enable them to be used efficiently in the applications and services. Moreover, he will demonstrate how to use these collected signals in real-life scenarios, using CNCF tools — Loki, Prometheus, OpenTelemetry, Jaeger, Conprof. He also aims to share the methods that are used to build and run applications running under heavy-traffic, and to understand the origin of the problems encountered in running systems.

Recording

Slides

• Building Observable Go Services

Events

• GopherCon Turkey 2020
  • Recording

Recording

Slides

• Absorbing Thanos Infinite Powers for Multi-Cluster Telemetry

Events

• KubeConNA 2020
  • Recording

In modern days, we run our applications as loosely coupled microservices on distributed, elastic infrastructure as (mostly) stateless workloads. Under these circumstances, observability is the key to understanding how our applications run and behave in action to deliver highly available and resilient service.

Prometheus is born in such an atmosphere as a solution to satisfy the observability needs of the cloud-native era. Among many other observability signals like logs and traces, metrics play the most substantial role. Sampled measurements observed throughout the system are crucial for monitoring the health of the applications and they enable real-time, actionable alerting. Although Prometheus tooling makes life a lot easier, there are still numerous possibilities to make mistakes or misuse them.

During this talk, Kemal will present several valuable patterns, best practices and idiomatic methods for instrumenting critical services. He will discuss common pitfalls and failure cases while sharing useful insights and methods to avoid those mistakes. Last but not least, he will give tips for writing simple, maintainable and robust alerts that derived from real-life experiences. By doing so he will propose “The Zen of Prometheus”.

Recording

Slides

• The Zen of Prometheus

Events

• PromCon Online 2020
  • Recording

Among many other observability signals like logs and traces, the metrics signal has a substantial role. Sampled measurements observed throughout the system are crucial for monitoring the health of the applications and, they enable real-time, actionable alerting. While there are many open-source robust libraries, in various languages, that allow us to easily instrument services for backends like Prometheus, there are still numerous possibilities to make a mistake or misuse those libraries.

During this talk, we discuss valuable patterns and best practices for instrumenting your Go application. The speakers will go through common pitfalls and failure cases while sharing valuable insights and methods to avoid those mistakes. Also, this talk demonstrates, how to use Go unit testing to verify the correctness of your observability signals. How it helps and why it is important. Last but not least, the talk covers a demo of the example instrumented Go application based on the experience and projects we maintain.

Recording

Slides

• Are You Testing Your Observability?

Events

• GoDays Berlin 2020
  • Recording
• FOSDEM 2020
  • Recording