Pulsar

---
name: pulsar
description: "Expert coverage of Apache Pulsar 4.x distributed messaging and streaming platform: broker/BookKeeper architecture, multi-tenancy, subscription types, geo-replication, tiered storage, Pulsar Functions, Pulsar IO connectors, transactions, and topic compaction. Use for \"Pulsar\", \"Apache Pulsar\", \"BookKeeper\", \"bookie\", \"Pulsar broker\", \"Pulsar topic\", \"Pulsar namespace\", \"Pulsar tenant\", \"Pulsar subscription\", \"Exclusive subscription\", \"Shared subscription\", \"Key_Shared\", \"Failover subscription\", \"Pulsar geo-replication\", \"tiered storage Pulsar\", \"Pulsar Functions\", \"Pulsar IO\", \"pulsar-admin\", \"Pulsar proxy\", \"Pulsar transaction\", \"topic compaction\", \"Pulsar SQL\", \"KoP\", \"Kafka on Pulsar\", \"managed ledger\", \"Pulsar schema\"."
license: MIT
---

# Apache Pulsar

This skill covers Apache Pulsar 4.x (4.0 through 4.2), including:

- Broker/BookKeeper separated architecture (stateless brokers, durable storage)
- Multi-tenancy (tenant/namespace/topic hierarchy, resource isolation)
- Four subscription types: Exclusive, Shared, Failover, Key_Shared
- Geo-replication (async and sync, active-active, replicated subscriptions)
- Tiered storage (offloading to S3, GCS, Azure Blob with erasure coding)
- Pulsar Functions (serverless stream processing)
- Pulsar IO (source and sink connectors)
- Transactions (atomic consume-process-produce across topics)
- Topic compaction (latest-value-per-key)
- Schema registry (Avro, JSON, Protobuf, compatibility modes)
- Pulsar SQL (Trino integration for SQL queries on topics)
- KoP (Kafka-on-Pulsar protocol compatibility)

## How to Approach Tasks

1. **Classify** the request:
   - **Architecture / design** -- Load `references/architecture.md` for broker/BookKeeper, multi-tenancy, topics, subscriptions, geo-replication, tiered storage
   - **Best practices** -- Load `references/best-practices.md` for topic design, subscription selection, producer/consumer tuning, geo-replication, security
   - **Troubleshooting** -- Load `references/diagnostics.md` for backlog issues, BookKeeper problems, broker failures, consumer lag, CLI reference

2. **Recommend** -- Provide actionable guidance with `pulsar-admin` commands, broker configuration, and SDK code.

## Core Architecture

### Separated Compute and Storage

```
Producers/Consumers --> Brokers (stateless) --> BookKeeper (bookies, durable storage)
                                             --> Metadata Store (ZooKeeper/etcd/Oxia)
```

Brokers are stateless. BookKeeper provides durable storage via ledgers. Adding/removing brokers requires no data migration.

### Multi-Tenancy
```
Instance --> Cluster --> Tenant --> Namespace --> Topic
```
Tenants define auth, allowed clusters. Namespaces define retention, TTL, backlog quotas, dispatch rates, replication.

### Topic Types
- **Persistent:** `persistent://tenant/namespace/topic` -- stored in BookKeeper
- **Non-persistent:** `non-persistent://tenant/namespace/topic` -- broker memory only
- **Partitioned:** Logical topic backed by N internal partition topics

### Subscription Types

| Type | Consumers | Ordering | Ack | Use Case |
|---|---|---|---|---|
| Exclusive | 1 | Strict global | Cumulative | Single pipeline |
| Shared | Many (round-robin) | None | Individual | Work queue |
| Failover | Many (1 active) | Strict (active) | Cumulative | HA pipeline |
| Key_Shared | Many (per-key) | Per-key | Individual | Per-entity ordering |

## Anti-Patterns

| Anti-Pattern | Why It Fails | Instead |
|---|---|---|
| Batching with Key_Shared | Different keys in same batch break per-key routing | Disable batching or use key-based batching |
| Default BookKeeper config in production | Underperforming storage, GC issues | Tune journal, entry logs, GC per workload |
| Ignoring backlog quotas | Unbounded backlog fills BookKeeper storage | Set namespace backlog quotas with producer_request_hold |
| Single-partition topics for high throughput | Throughput limited to one broker | Use partitioned topics |
| No retention policy | Messages retained forever | Set namespace retention (size + time limits) |

## Reference Files

- `references/architecture.md` -- Broker/BookKeeper layers, multi-tenancy, topics, subscriptions, acknowledgment, batching, chunking, geo-replication, tiered storage, schema registry, transactions, compaction, Pulsar Functions/IO/SQL, KoP
- `references/best-practices.md` -- Topic design, subscription selection, producer/consumer tuning, namespace policies, geo-replication patterns, security, monitoring
- `references/diagnostics.md` -- Backlog growth, BookKeeper issues, broker failures, consumer lag, compaction, geo-replication lag, CLI reference (pulsar-admin)

## Cross-References

- The `overview` skill -- cross-broker comparisons and event-driven architecture design
- The `kafka` skill -- Kafka comparison and migration context