Jfr And Async Profiler

---
name: jfr-and-async-profiler
description: >
  Selecting the least-perturbing JVM evidence source that matches the question: JFR events and
  timeline versus async-profiler sampling, CPU versus elapsed/off-CPU, allocation versus
  retention, lock versus queue/I/O, startup versus steady state, and one-off versus continuous
  capture. Covers adequacy, positive controls, target scope, version discovery, container
  access, overhead, artifact integrity, and cross-tool reconciliation. Use before a JVM profile
  is collected or when an empty/disagreeing recording may be a configuration artifact. Graph
  interpretation, JFR internals, async-profiler engines, and fleet operations have separate
  owners.
---

# JFR and async-profiler

## Purpose

Choose an observation whose event population and weighting answer the stated performance
question, then prove that the capture was active, adequate, bounded, and correlated to the
symptom. A CPU profile can be perfectly valid while saying nothing about off-CPU latency; an
empty contention event can mean no qualifying waits or a disabled/thresholded/wrong event.

Neither tool is categorically better. JFR provides JVM event chronology and correlations;
async-profiler provides targeted sampled stacks and additional native/perf/allocation/lock
modes. They can overlap and perturb each other. Start with the lowest-risk existing evidence
that separates hypotheses.

## Ownership boundary

- This skill owns first-instrument and capture-envelope selection.
- `jfr-advanced` owns JFC/event/schema/API details.
- `async-profiler-advanced` owns engine/stack/access/conversion details.
- `flame-graph-analysis` owns aggregate stack interpretation.
- `continuous-profiling` owns always-on fleet storage/query operations.
- `incident-evidence-capture` owns live-incident ordering and recovery budget.
- `performance-methodology` owns causal investigation and validation.

## Question contract

Reuse supplied recordings, commands, project/runtime facts, and applicable calibration before
asking for context or collecting again. Fill the relevant parts of this contract; request only
missing facts that could change selection, safety, or the conclusion, and continue independent
artifact checks while they remain unresolved:

```text
symptom, decision, outcome metric, and affected window:
CPU, elapsed latency, allocation, retention, contention, I/O, startup, or chronology question:
target population: process/thread/task/operation/version/instance:
event/sample weight and denominator required:
minimum contribution/duration/rate worth detecting:
workload, completed work, warm-up/lifecycle and control:
collection duration/trigger and expected observations:
overhead, storage, privilege, privacy, and recovery budget:
```

“Where is time going?” is incomplete: CPU time, thread elapsed residency, request critical-path
time, and summed parallel wait are different quantities.

## Selection matrix

| Question                                 | Start with                                            | Add/switch when                                                                    |
| ---------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Broad intermittent JVM incident/history  | existing rolling JFR                                  | event/threshold/stack coverage is insufficient; use targeted profile/JFC           |
| On-CPU Java/native/kernel work           | async-profiler CPU or validated JFR CPU-time sampling | platform/access or stack fidelity requires alternative/correlation                 |
| Latency with low CPU / off-CPU residency | JFR duration events + wall/off-CPU profile            | queue/dependency/task ownership remains ambiguous                                  |
| GC/pause chronology                      | GC log + JFR GC/safepoint events                      | allocation/live-set attribution needed                                             |
| Allocation source                        | sampled allocation JFR/async-profiler                 | exact TLAB/lifetime/retention question requires escalation                         |
| Retained heap/leak                       | GC/live-set trend, object statistics                  | heap dump/dominator paths—not allocation profile alone                             |
| Java contention                          | JFR monitor/wait/park + thread/lock profile           | logical resource/owner requires application/pool evidence                          |
| Startup/class loading/JIT                | JFR from process start                                | targeted compilation/profile/assembly after phase localization                     |
| Native/OS scheduling gap                 | async-profiler native/kernel + host metrics           | targeted eBPF when JVM-local view cannot distinguish                               |
| Very hot selected method calls           | sampling first                                        | JDK 25 method timing/trace or custom instrumentation for a narrow bounded question |

Use both tools only when the extra dimension can change the decision and combined cost is
calibrated. “Always take CPU plus allocation” wastes budget and can perturb the incident.

## CPU, wall, and duration events

CPU sampling selects CPU consumption or an engine-specific approximation; Java RUNNABLE
state alone does not prove a thread was executing on a CPU. It ranks
where CPU/event samples land, not response time. Wall sampling observes eligible threads over
elapsed time, including idle/waiting states; its volume and denominator depend on thread
population and implementation. JFR duration events record qualifying operations with start/
duration and fields; thresholds censor shorter events.

Wall is not off-CPU-only: it includes execution as well as waiting. A dedicated off-CPU
mechanism observes descheduled intervals with its own coverage; do not subtract unrelated
CPU and wall percentages to derive blocked time.

For latency, select the evidence that distinguishes the remaining hypotheses from:

- end-to-end latency/trace/queue measurement for critical path;
- JFR I/O/monitor/park/safepoint/GC events for typed intervals;
- wall/off-CPU stacks for code locations and thread roles;
- CPU stacks for actual compute;
- dependency, pool, cgroup, and host evidence for ownership.

Summed waits across concurrent threads can exceed wall time. A parked frame can be healthy idle
capacity. A socket event duration may overlap scheduler delay. Do not convert stack percentage
directly to request latency without scope/correlation.

## Allocation, GC, and retention

Allocation sampling attributes creation under its sampling/weight mechanism. Use weights when
the event provides them; event count alone may mis-rank sites. Allocation does not prove
retention or leak. GC logs/JFR establish allocation pressure, occupancy, collector phases,
pause/concurrent CPU, promotion/humongous/reference work; object statistics/heap dump establish
retained paths.

Allocation changes can affect GC frequency and collection work/duration. Do not infer “small
CPU frame + large allocation frame = GC is the cause” until GC/resource and user-impact
evidence align.

## Contention and waits

Map the code mechanism to target-JDK event types rather than one generic “blocking” event:

- monitor enter contention;
- `Object.wait`/notify protocol;
- `LockSupport.park`-based locks, synchronizers, pools, and queues;
- sleep;
- socket/file operations;
- virtual-thread pinning/scheduling/submit failures where supported;
- application/executor/connection/remote queues not automatically represented by JVM events.

Check event existence, enablement, threshold, stack setting, and workload opportunity. Treat
zero events as evidence of absence only within validated coverage and configuration. If shorter
events matter, test a lower threshold on a canary or bounded window while measuring event volume
and overhead; otherwise retain the current threshold and state its censoring limit.

## Warm-up and phase selection

Do not require steady state when startup, readiness, class loading, cache priming, deoptimization,
or phase transition is the question. For steady-state comparisons, define warm-up by observed
outcomes and runtime state—throughput/latency, compilation activity, GC/live set, caches,
connections, and workload mix—not a sleep or one compiler-queue snapshot.

Capture phase markers and process uptime. Reprofiling a warmed candidate against a cold baseline
is not a valid differential.

## Adequacy

Plan expected selected observations for the smallest frame/event contribution worth detecting.
Sample-count uncertainty is not universally `1/sqrt(n)`: samples can be weighted,
autocorrelated, batched, throttled, clustered, filtered, and lost. Report absolute count/weight,
duration/work denominator, unknown/truncated/lost fraction, and independent recordings.

Reuse relevant controls from the same capture configuration and environment when they still
establish the required coverage. For new or uncertain coverage, choose the controls that can
resolve that uncertainty, for example:

- known CPU loop should appear in CPU sampling, not off-CPU;
- known sleep/park/I/O should appear in appropriate wall/JFR event coverage;
- known allocation site should appear with correct weight semantics;
- known below/above-threshold contention should demonstrate censoring;
- target/non-target threads/processes should prove scope.

Use isolated controls or a representative canary; do not inject CPU/allocation/contention
load into an affected production JVM merely to satisfy this list. If controls are unavailable,
report the resulting uncertainty. Stop at the capture/incident budget and report insufficient
evidence rather than extending collection until one frame reaches a folklore count.

## JFR baseline decision

Continuous JFR is valuable when retroactive questions justify permanent measured cost and
durable handling. It is not mandatory for every JVM. `default.jfc` is designed as a low-overhead
starting point, not a workload-independent percentage guarantee; added events/periods/stacks and
concurrent recordings can change cost.

JFR is available in mainstream JDKs since 11, but a live `jcmd JFR.start` still depends on the
actual JVM/build, attach enabled, PID namespace, credentials, tooling, process responsiveness,
and storage. Some repository/stack/buffer configuration and evidence needed from startup cannot
be retrofitted safely. Test capture and survival before incidents.

Do not assert that `settings=default` is unsuitable or `profile` is required for all method
profiling. Inspect target JFC/event settings, question resolution, overhead, and positive
controls. Java 25 JFR CPU-time sampling and method timing/tracing are additional choices with
platform/experimental/instrumentation constraints; follow `jfr-advanced`.

## Async-profiler decision

Pin/download async-profiler through an approved artifact supply chain with release checksum/
provenance. Discover `asprof -v`, `list`, and help on the target. Generic CPU, perf-events,
`ctimer`, `itimer`, wall, stack walkers, batching, virtual-thread support, and converter options
change by release/platform.

The checked async-profiler v4.4 distribution targets HotSpot-based JVMs on Linux/macOS;
do not assume native Windows or another JVM implementation is supported. Use appropriate
JFR evidence or a matching supported environment without upgrading the application by default.

Container failure can be attach, PID/filesystem namespace, UID, seccomp/LSM, perf policy,
capabilities, PMU virtualization, limits, or symbols. `ctimer` may avoid perf-event privilege but
also changes selection/resolution/kernel visibility; it is not “same result minus kernel
stacks.” Prefer the least-privileged engine that meets the question and prove what actually ran.

Async-profiler typically allows one controlled session per target library/JVM at a time; JFR
supports multiple recordings. Coordinate with existing agents/profilers and test supported
multi-event JFR output rather than launching competing sessions.

## Instrumentation versus sampling

Instrumentation cost generally scales with selected invocation/allocation/event rate and can
change inlining/transformation behavior; sampling cost generally scales with sampling/event
rate and stack work. Neither is universally safe/unsafe in production. APM and JFR custom/method
events can be production-appropriate when narrow, calibrated, bounded, and reversible.

Use instrumentation when exact call/duration semantics are required and sampling cannot answer
the decision. For a new or changed configuration, compare control/enabled outcomes and exercise
relevant burst, exception, retransformation, agent-interaction, and tail-latency risks. Reuse
applicable calibration. Never say “instrumenting agents are forbidden in production.”

## Thread dumps are not profiles

Repeated thread dumps can show persistence/progress/ownership and are valuable for deadlocks/
liveness. They are not a statistically designed sampler, and capture/output cost can be material
with large platform/virtual-thread populations. Modern JDK dump commands/formats and virtual-
thread coverage differ. Do not claim every `jstack` invocation necessarily uses one global
safepoint; inspect target implementation/impact and use `concurrency-diagnostics`.

## Cross-tool disagreement

Disagreement can come from different clocks/populations/weights, thresholds, filters, time
windows, stack walking/symbols, JDK/tool versions, lost events, or genuine multi-layer behavior.
Reconcile this table before choosing one result:

| Dimension                            | JFR result | async-profiler result |
| ------------------------------------ | ---------- | --------------------- |
| exact interval/load/process uptime   |            |                       |
| event/engine and eligible population |            |                       |
| weight/unit/threshold/period         |            |                       |
| thread/state/context filters         |            |                       |
| stack depth/walker/symbol coverage   |            |                       |
| event/sample totals and loss         |            |                       |
| tool/JDK/config epoch                |            |                       |

The disagreement is a diagnostic lead, not automatically a finding and never permission to
select the convenient graph.

## Capture workflow

If existing evidence answers the question, finish with the supported result, its limits, and
what would warrant another capture. Otherwise propose or execute only the bounded next capture
needed; distinguish a proposal from work actually run. Do not delay authorized incident recovery
to complete a profiling checklist.

1. Preserve existing telemetry and incident window.
2. State the question contract and surviving hypotheses.
3. Select least-risk event/clock/tool and target cohort.
4. Discover target capabilities/settings/access; validate unresolved coverage with relevant controls.
5. Bound duration, event/sample rate, disk/memory, privilege, and abort threshold.
6. Capture workload/outcome markers and profiler/JFR health concurrently.
7. Verify file/readability, counts/weights, loss, scope, timestamps, checksum, and privacy.
8. Hand off interpretation with artifact/window, event/weight semantics, hypotheses, and limits;
   validate any resulting performance change with comparable external outcomes.

## Anti-patterns

**Anti-pattern: CPU profile closes a latency investigation.** It sees compute, not off-CPU/
dependency/queue critical path. Add typed duration and wall/trace evidence.

**Anti-pattern: allocation profile on every investigation.** It adds cost and answers a different
question. Require allocation/GC/memory hypothesis.

**Anti-pattern: zero event count proves absence.** Validate event schema/settings/threshold,
capture opportunity, and positive control.

**Anti-pattern: copy exact JFR/async-profiler defaults and percentages.** Settings and engines
are version/workload dependent. Discover, calibrate, fingerprint, and preserve effective config.

## Definition of done

- [ ] Result names the decision, relevant clock/event, population, weight, and detection limit.
- [ ] Tool choice and alternatives are justified by information gain and risk.
- [ ] Runtime/tool/settings and relevant access facts are established; material unknowns are explicit.
- [ ] Adequacy and coverage cite applicable controls, counts/weights, and loss evidence; unsupported
      conclusions remain conditional rather than treating unavailable checks as passed.
- [ ] Any new capture aligns to workload/lifecycle/outcome and has bounded overhead/storage/privilege.
- [ ] Artifacts used are readable and have checksum/provenance and handling appropriate to privacy,
      retention, and recovery requirements; missing evidence limits the result.
- [ ] When results are compared across tools, reconcile their semantics before causal interpretation.

## References

- [Choosing a profile](references/choosing-a-profile.md) — read when choosing the clock,
  population, controls or escalation for a symptom or empty recording.
- [Command and production capture protocol](references/commands.md) — read before capture
  to check target access, storage, lifecycle and artifact validation.
- [JFR API Programmer's Guide](https://docs.oracle.com/en/java/javase/25/jfapi/)
- [JDK 25 `jcmd`](https://docs.oracle.com/en/java/javase/25/docs/specs/man/jcmd.html)
- [async-profiler repository](https://github.com/async-profiler/async-profiler)