npx skills add AGCO-Global/org-skills --skill messaging-patterns -a github-copilot
# or with the org installer (adds .github/skills/messaging-patterns):
npx -y github:AGCO-Global/org-skills add skill messaging-patterns

npx skills add AGCO-Global/org-skills --skill messaging-patterns
# user-level instead of project-level:
npx skills add AGCO-Global/org-skills --skill messaging-patterns -g

/plugin marketplace add AGCO-Global/org-skills
/plugin install backend-shared-skills@org-skills
# or just this skill, in this repository:
npx skills add AGCO-Global/org-skills --skill messaging-patterns -a claude-code

npx skills add AGCO-Global/org-skills --skill messaging-patterns -a codex

npx skills add AGCO-Global/org-skills --skill messaging-patterns -a cursor

npx skills add AGCO-Global/org-skills --skill messaging-patterns -a gemini-cli

CREATE TABLE outbox (
  id            uuid PRIMARY KEY,
  occurred_at   timestamptz NOT NULL DEFAULT now(),
  aggregate_id  text  NOT NULL,                 -- partition / ordering key
  type          text  NOT NULL,                 -- 'orders.order-placed.v1'
  payload       jsonb NOT NULL,
  headers       jsonb NOT NULL DEFAULT '{}',    -- traceparent, tenant, correlation id
  sent_at       timestamptz
);
CREATE INDEX outbox_unsent ON outbox (occurred_at) WHERE sent_at IS NULL;

## Messaging design: <flow>
Flow table: message type · owner · partition key · consumer · effect · idempotency strategy
Outbox/inbox: tables, relay location and owner
Contract: <type>.v<n>, schema location, header set, compatibility rule
Failure handling: retry policy, DLQ name, alert, replay procedure
Ordering: per <key>; what breaks if reordered
Saga/workflow (multi-step only): steps, pivot step, compensations

---
name: messaging-patterns
description: >
  Design reliable asynchronous messaging between services or modules in any back-end stack — transactional outbox, inbox and idempotent consumers, at-least-once delivery, event schema versioning, retries with backoff and dead-letter queues, ordering per key, and sagas versus a workflow engine. Use this whenever the user adds a broker or queue (Kafka, RabbitMQ, Azure Service Bus, SQS/SNS, BullMQ, pg-boss), publishes an event right after saving to the database, asks "how do I make this consumer idempotent", sees duplicate or lost messages, designs or changes an event contract such as OrderPlaced, needs a retry or dead-letter policy, asks about MassTransit, NServiceBus, Rebus or Temporal, or wants to turn a synchronous call between modules into an event. Also apply it when a webhook receiver must tolerate redelivery.
metadata:
  technology: Backend (general)
  type: architecture
---

# Messaging Patterns
> **Targets:** Any broker (Kafka, RabbitMQ, Azure Service Bus, SQS/SNS, BullMQ, pg-boss) · **Verified:** 2026-09 against https://learn.microsoft.com/en-us/azure/architecture/patterns/saga and https://learn.microsoft.com/en-us/azure/architecture/databases/guide/transactional-outbox-cosmos

Every broker delivers at least once, and every network hop between "save" and "publish" can fail on its own. Design so that duplicates are harmless, lost messages are impossible and every failure has somewhere to go. This skill owns these patterns for the catalog; `dotnet-architecture` and `nodejs-architecture` link here and add only library choices.

## 1. Decide first

| Question | Default | Change when |
|---|---|---|
| Queue or log? | **Queue** (RabbitMQ, Service Bus, SQS, BullMQ, pg-boss) for commands and work distribution | Several consumer groups need the same events, or replay matters → **log** (Kafka, Redpanda, Event Hubs, Kinesis) |
| Save and publish in one request? | **Transactional outbox**, always | Never "save, then publish": the second step eventually fails alone |
| Consumer deduplication | **Inbox table** keyed by message id, written in the same transaction as the effect | The effect is naturally idempotent (upsert, conditional update); keep the inbox anyway when money or external calls are involved |
| Domain vs integration event | Domain events stay in-process and in the same transaction; integration events cross the boundary through the outbox | — |
| Ordering | Per key (aggregate id) only | You need global order → you have a workflow, not a stream |
| Multi-step business process | **Choreography** (events) for 2–3 steps without compensation | Compensation, timeouts or more steps → **orchestration** (saga) or a workflow engine |
| Broker library | One with outbox, retries and sagas built in (MassTransit, NServiceBus, Rebus in .NET; BullMQ or pg-boss plus your own outbox in Node) | You need broker-specific features (Kafka transactions, Streams) |

## 2. Transactional outbox

Write the message to an `outbox` table in the **same database transaction** as the state change; a relay reads unsent rows, publishes and marks them sent. Delivery becomes at-least-once, so consumers dedupe (§3).

```sql
CREATE TABLE outbox (
  id            uuid PRIMARY KEY,
  occurred_at   timestamptz NOT NULL DEFAULT now(),
  aggregate_id  text  NOT NULL,                 -- partition / ordering key
  type          text  NOT NULL,                 -- 'orders.order-placed.v1'
  payload       jsonb NOT NULL,
  headers       jsonb NOT NULL DEFAULT '{}',    -- traceparent, tenant, correlation id
  sent_at       timestamptz
);
CREATE INDEX outbox_unsent ON outbox (occurred_at) WHERE sent_at IS NULL;
```

Relay loop: `SELECT … FROM outbox WHERE sent_at IS NULL ORDER BY occurred_at LIMIT 100 FOR UPDATE SKIP LOCKED` → publish each row with `id` as the message id and `aggregate_id` as the partition key → `UPDATE … SET sent_at = now()` → commit. `SKIP LOCKED` lets several relay instances run without double-publishing; a crash after publish but before the update re-sends, which is why consumers dedupe. Archive or delete sent rows on a schedule. Relay code for .NET and Node is in `references/outbox-and-inbox.md`.

## 3. Inbox and idempotent consumers

- Consumer sequence: begin transaction → `INSERT INTO inbox (message_id, consumer) … ON CONFLICT DO NOTHING` → no row inserted means duplicate: commit and ack → otherwise apply the effect → commit → **ack after commit**. Dedup and effect share one transaction, so a crash between them replays cleanly.
- Prefer naturally idempotent effects (`UPSERT`, `UPDATE … SET status = 'paid' WHERE status = 'pending'`) so the inbox is the second line of defence.
- Side effects that cannot join the transaction (email, partner API) become their own outbox message or carry an idempotency key on the outbound call. Never call HTTP inside the consumer's database transaction.
- Bound consumer concurrency; keep the broker's visibility timeout or lock longer than the slowest handler, or renew it.

## 4. Event contracts and versioning

- Events are past-tense facts with a versioned type: `orders.order-placed.v1`. The payload carries ids and the fields consumers act on, not the whole aggregate.
- Within a version only **additive** changes: new optional fields; consumers ignore unknown fields. Removing or renaming a field, changing a type or a meaning → publish `.v2` **alongside** `.v1` (dual-publish), migrate consumers, measure `.v1` consumption, then retire it.
- Schemas (JSON Schema, Avro or Protobuf) are published from the owning module's contracts package or a registry; consumers validate on receipt and dead-letter what fails, they do not guess.
- Every message carries `id`, `type`, `occurred_at`, `aggregate_id`, `correlation_id`, `causation_id`, W3C `traceparent` and tenant in **headers**, so brokers and tooling read them without decoding the payload. Compatibility matrix and dual-publish recipe: `references/event-versioning.md`.

## 5. Retries, poison messages and dead letters

| Failure | Handling |
|---|---|
| Transient (timeout, deadlock, 503) | Retry with exponential backoff and jitter, capped (for example 5 attempts over ~10 minutes); the message id is preserved so the inbox still dedupes |
| Permanent (schema invalid, business rule violated) | No retry: dead-letter on the first attempt with the error and stack trace in headers |
| Retries exhausted | Dead-letter; alert on DLQ depth > 0; keep a replay tool for after the fix |
| Slow consumer | Bounded concurrency; visibility timeout or lock renewal longer than processing time |

Never delete from a dead-letter queue without a recorded decision.

## 6. Sagas versus a workflow engine

- **Choreography**: each service reacts to the previous event and emits the next. Fine for 2–3 steps; beyond that nobody can see the whole flow.
- **Orchestration** (saga / process manager): one component owns the state machine, sends commands, waits for replies and runs compensating actions in reverse on failure. Persist its state next to the inbox and outbox in the same database. MassTransit and NServiceBus ship saga persistence.
- **Workflow engine** (Temporal, Azure Durable Functions, Camunda) when you need durable timers measured in days, human steps, versioned long-running code or built-in visibility. It replaces your saga persistence, not your idempotency: activities still run at least once.
- Name the **pivot** step (the point of no return, such as charging the card): everything before it must be compensable, everything after it retryable.

## 7. Deliverable format

```
## Messaging design: <flow>
Flow table: message type · owner · partition key · consumer · effect · idempotency strategy
Outbox/inbox: tables, relay location and owner
Contract: <type>.v<n>, schema location, header set, compatibility rule
Failure handling: retry policy, DLQ name, alert, replay procedure
Ordering: per <key>; what breaks if reordered
Saga/workflow (multi-step only): steps, pivot step, compensations
```

## Checklist

- [ ] State change and outbox row are in one transaction; nothing publishes straight from request code.
- [ ] Every consumer dedupes by message id or is provably idempotent, and acks after commit.
- [ ] Message id, correlation id, `traceparent` and tenant travel in headers.
- [ ] Event type is versioned; the change is additive, or a new version is dual-published.
- [ ] Retry policy has backoff, jitter and a cap; permanent failures dead-letter immediately.
- [ ] DLQ depth is alerted and a replay path exists.
- [ ] Ordering is per key; the relay publishes with the aggregate id as partition key.
- [ ] Multi-step flows name the pivot step and the compensations.
- [ ] No HTTP call or non-transactional side effect inside a consumer's database transaction.

## Anti-patterns

- **Save, then publish, then hope** → outbox row in the same transaction; a relay publishes.
- **Ack before commit** → commit, then ack; accept redelivery and dedupe.
- **Global ordering requirement** → partition by aggregate id; make consumers tolerate reordering across keys.
- **Fat events carrying the whole aggregate** → ids plus the fields consumers act on; fetch the rest from the owner's API.
- **Retrying schema errors forever** → classify failures; permanent ones go to the DLQ on the first attempt.
- **HTTP call inside the consumer transaction** → separate outbox message or idempotent outbound step.
- **Shared "events" package containing entity classes** → contracts package with schemas and DTOs only.
- **Saga logic spread over five consumers with no owner** → orchestrator or workflow engine with persisted state.

## Go deeper

- `references/outbox-and-inbox.md` — relay implementations for .NET (EF Core + `BackgroundService`) and Node (`pg`), inbox DDL, consumer skeletons, BullMQ and pg-boss dedupe options, cleanup jobs.
- `references/event-versioning.md` — compatibility matrix, dual-publish recipe, header conventions, schema registry options.
- Sibling skills: `dotnet-architecture` and `nodejs-architecture` for library choice and module boundaries; `database-migrations` for adding the outbox and inbox tables safely; `service-operability` for trace propagation; `backend-code-review` for reviews.