Resilience Patterns¶
Principle · resilience-patterns
Loads automatically whenever the agent touches code this principle governs. Install the skills · View source
In a single process, a slow function is a performance problem. Across a network, a slow dependency is an outage: threads pile up on the caller, its own callers time out, and the failure walks upstream faster than anyone can page. Most microservice incidents are this, not a crash.
Two rules carry most of the weight: every remote call has a timeout, and every retry has a budget. Retries without a budget are a self-inflicted denial of service on a service that was already struggling.
Config Resolution¶
- Read
.msskills/config.yamlin the repo root; look uppaths.resilience. - A custom document exists at that path → read its frontmatter
mode: override→ use it alone; ignore the defaults below.overlay(default, or nomodekey) → read defaults first, then apply the custom document's sections on top, matched by exact heading; new sections append.- A path is configured but no file exists there → say which path is missing, then use the defaults.
- No config file or no
paths.resiliencekey → use defaults. .msskills/stack.mdexists → name that stack's actual library and idiom rather than the generic pattern: the platform's circuit breaker, its retry policy type, its client deadline mechanism, its outbox tooling. Never invent a library; if the profile does not name one, describe the behaviour and say the library choice is open.
Self-Validation Checklist¶
STOP after writing or reviewing any code that crosses a process boundary — HTTP, gRPC, database, cache, broker, filesystem. Verify every check. Fix failures before presenting.
- TIMEOUT: Does this call have an explicit timeout, shorter than the caller's own deadline? A default of "none" or "60s inherited from the client library" → set one derived from the latency budget.
- DEADLINE PROPAGATION: Is the remaining time budget passed downstream, so work nobody is waiting for is abandoned? If the framework supports it and it is unused → use it.
- RETRY SAFETY: Is the operation idempotent before it is retried? Retrying a non-idempotent write → add an idempotency key first, or do not retry.
- RETRY SHAPE: Does every retry use exponential backoff with jitter, a bounded attempt count, and a budget capping retries as a fraction of total traffic? Fixed-interval or unbounded retries → fix; synchronised retries → add jitter.
- RETRY CLASSIFICATION: Are only retryable failures retried — timeouts, connection errors, 429, 502/503/504? Retrying a 400 or 422 wastes the budget and never succeeds.
- NO NESTED RETRIES: Does more than one layer retry the same logical call? Retries multiply; pick exactly one layer, usually the outermost that still knows the operation is idempotent.
- CIRCUIT BREAKER: Does a call to a dependency that can fail persistently sit behind a breaker with a defined failure threshold, open duration, and half-open probe? If not → add one, or record why this dependency does not need it.
- ISOLATION: Can one slow dependency exhaust the resource pool the whole service shares? If yes → bulkhead it with a separate pool or concurrency limit.
- FALLBACK DEFINED: When this fails after all protection, what does the caller see — cached data, a partial response, a degraded feature, or a clean error? "It throws" is only acceptable if that is the deliberate choice.
- IDEMPOTENT CONSUMER: For a message handler, does redelivery of the same message produce the same result? At-least-once delivery is the norm → deduplicate on message ID or make the effect naturally idempotent.
- NO DUAL WRITE: Does anything write to the database and publish to a broker as two separate operations? → transactional outbox.
- COMPENSATION: For a multi-service workflow, is every step's compensating action defined, and is it itself idempotent and retryable?
- PROBES CORRECT: Does readiness reflect the ability to serve traffic, and liveness only unrecoverable state? A readiness probe that checks a downstream dependency will take the whole fleet out when that dependency blips.
All checks pass → state "Passes resilience: timeout
Active Anti-Pattern Scan¶
Any box you can check is a defect. Fix it before presenting.
- [ ] Infinite Timeout: any remote call using the library default → set an explicit one.
- [ ] Retry Storm: retries at client, gateway, and service layers compounding → one layer only.
- [ ] Synchronised Retry: fixed backoff with no jitter, so every caller returns together → full jitter.
- [ ] Unbounded Retry Budget: retries allowed regardless of overall failure rate → cap them as a fraction of traffic and shed when exceeded.
- [ ] Retrying the Unretryable: 4xx responses, validation failures, or non-idempotent writes retried → classify before retrying.
- [ ] Dual Write: database commit and event publish as separate steps → outbox.
- [ ] Fire and Forget: an async publish whose failure is logged and dropped → outbox, or an explicit dead-letter path.
- [ ] Missing Dead Letter: a consumer that retries forever on a poison message, blocking the partition → bounded retries, then dead-letter with enough context to replay.
- [ ] Cascading Readiness: readiness checking downstream dependencies, so one outage empties every load balancer → readiness reflects this instance only.
- [ ] Shared Pool Exhaustion: one connection or thread pool for every dependency → bulkhead the risky ones.
- [ ] Silent Fallback: degraded data served as if it were fresh, with no signal to the caller or the dashboards → make degradation visible.
- [ ] Compensation Assumed: a saga whose rollback path was never written or never tested → it does not exist.
- [ ] Unbounded Queue or Buffer: in-memory work queues with no limit → bound them and shed; an unbounded queue converts a latency problem into an out-of-memory crash.
Ambiguity Signals¶
Route these through collaborative-judgment. Each has two defensible answers.
- Fail closed or degrade. Serving stale or partial data protects availability; refusing protects correctness. Which one is right is a product decision — ask, do not assume.
- Timeout values. Too short sheds load that would have succeeded; too long ties up resources. Without latency percentiles this is a guess; say so, propose a starting point, and make it configurable.
- Circuit breaker thresholds. Sensitive breakers trip on noise; tolerant ones let cascades build. Depends on traffic volume and the cost of a false trip.
- Orchestration or choreography for a multi-step workflow. An orchestrator makes the flow
visible and debuggable in one place; choreography avoids the central component and scatters the
flow across services (see
service-boundaries). - Where the outbox relay lives. Change-data-capture is robust and adds infrastructure; a polling publisher is simple and adds latency and load.
- Whether this dependency needs a breaker at all. For a rarely used, non-critical call, the breaker may be more machinery than the risk justifies.