rust-concurrency

어떤 스킬인가요?

Design, implement, diagnose, and test Rust concurrency and parallelism with threads, Send and Sync, locks, atomics, channels, Tokio, Rayon, Crossbeam, bounded backpressure, actor ownership, task supervision, graceful shutdown, runtime diagnostics, and Loom model tests. Use when users ask about shared state, deadlocks, async tasks, CPU parallelism, high concurrency, daemon resource budgets, slow consumers, worker pools, lock-free structures, or concurrent correctness.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

interface:
  display_name: "Rust Concurrency"
  short_description: "Design safe bounded Rust concurrency and async systems"
  default_prompt: "Use $rust-concurrency to design and verify this concurrent or asynchronous Rust workflow."

# Concurrency Examples

## Thread pool pattern
```rust
use std::sync::{Arc, Mutex};
use std::thread;

let counter = Arc::new(Mutex::new(0));
let mut handles = vec![];

for _ in 0..10 {
    let c = Arc::clone(&counter);
    handles.push(thread::spawn(move || {
        *c.lock().unwrap() += 1;
    }));
}
for h in handles { h.join().unwrap(); }
println!("{}", counter.lock().unwrap());
```

## async/await with Tokio
```rust
use tokio::time;

async fn task(n: u32) -> u32 {
    time::sleep(time::Duration::from_millis(n as u64)).await;
    n
}

#[tokio::main]
async fn main() {
    let (a, b) = tokio::join!(task(100), task(200));
    println!("{a}, {b}");
}
```

## mpsc channel
```rust
use std::sync::mpsc;
use std::thread;

let (tx, rx) = mpsc::channel();
for i in 0..5 {
    let tx = tx.clone();
    thread::spawn(move || { tx.send(i).unwrap(); });
}
for received in rx { println!("Got: {received}"); }
```

# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4

[[package]]
name = "crossbeam-deque"
version = "0.8.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5181e0de7b61eb03a81e347d6dd8797bae9da5146707b51077e2d71a54ec0ceb"
dependencies = [
 "crossbeam-epoch",
 "crossbeam-utils",
]

[[package]]
name = "crossbeam-epoch"
version = "0.9.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d6914041f254d6e9176c01941b21115dcfb7089e55135a35411081bd106ef3f"
dependencies = [
 "crossbeam-utils",
]

[[package]]
name = "crossbeam-utils"
version = "0.8.22"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "61803da095bee82a81bb1a452ecc25d3b2f1416d1897eb86430c6159ef717c17"

[[package]]
name = "either"
version = "1.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e"

[[package]]
name = "rayon"
version = "1.12.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d"
dependencies = [
 "either",
 "rayon-core",
]

[[package]]
name = "rayon-core"
version = "1.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91"
dependencies = [
 "crossbeam-deque",
 "crossbeam-utils",
]

[[package]]
name = "rust-concurrency-golden"
version = "0.0.0"
dependencies = [
 "rayon",
]

[package]
name = "rust-concurrency-golden"
version = "0.0.0"
edition = "2024"
rust-version = "1.85"
publish = false

[dependencies]
rayon = "1.12"

[lib]
path = "src/lib.rs"

use std::sync::atomic::{AtomicUsize, Ordering};

use rayon::prelude::*;

pub fn count_workers(workers: usize) -> usize {
    let completed = AtomicUsize::new(0);
    std::thread::scope(|scope| {
        for _ in 0..workers {
            scope.spawn(|| {
                completed.fetch_add(1, Ordering::Relaxed);
            });
        }
    });
    completed.load(Ordering::Relaxed)
}

pub fn parallel_sum(values: &[u64]) -> u64 {
    values.par_iter().copied().sum()
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn joins_all_scoped_workers() {
        assert_eq!(count_workers(4), 4);
    }

    #[test]
    fn uses_cpu_parallelism_for_independent_items() {
        assert_eq!(parallel_sum(&[1, 2, 3, 4]), 10);
    }
}

# Concurrency Testing and Diagnostics

Concurrency verification requires complementary evidence. Ordinary tests prove selected executions; model checking explores controlled schedules; tracing and load tests expose runtime behavior under realistic work.

## Deterministic contract tests

- Inject clocks, cancellation, queue capacities, and worker counts.
- Replace sleeps with explicit barriers, notifications, or paused time where supported.
- Assert task and waiter counts return to zero after success, failure, cancellation, disconnect, and shutdown.
- Test queue-full, receiver-drop, sender-drop, lag, timeout, partial completion, panic, and shutdown races.
- Run multi-thread and current-thread Tokio configurations when both are supported.

Avoid tests that repeatedly sleep and hope a race appears. They are slow, flaky, and weak evidence.

## Loom model checking

Use Loom for small synchronization state machines implemented with threads, atomics, mutexes, condition variables, or custom cells. Put interchangeable synchronization types behind a `cfg(loom)` module and run the model separately:

```bash
RUSTFLAGS="--cfg loom" cargo test --release --test loom_model
```

Keep the model deterministic and small. Loom cannot observe ordinary synchronization hidden behind non-Loom types, arbitrary system calls, external runtimes, or random input. Bound preemptions only after an exhaustive model becomes impractical, and record that reduced coverage.

## Async runtime diagnostics

Instrument Tokio tasks with `tracing` spans and enable the runtime's tracing support only in diagnostic builds where appropriate. Use `tokio-console` to inspect:

- tasks with long poll durations;
- tasks woken repeatedly without progress;
- resources with long waits;
- tasks that remain alive after their owners stop;
- blocking operations on async workers.

Treat instrumentation overhead and telemetry cardinality as production budgets. Do not expose payloads, tokens, or unbounded identifiers in span fields.

## Load and failure testing

Measure at and beyond the intended concurrency limit:

1. ramp admission until the first budget saturates;
2. hold slow consumers and stalled dependencies;
3. inject connection resets, timeouts, partial responses, and cancellation;
4. trigger graceful shutdown while queues are non-empty;
5. confirm bounded memory, bounded queue depth, stable latency, rejected-work metrics, and no surviving tasks.

Use Criterion for local CPU primitives and a service load generator for end-to-end behavior. A microbenchmark cannot prove scheduler fairness, overload control, or graceful shutdown.

## Race and memory tools

- Miri can detect some undefined behavior in unsafe concurrent code but is not a general data-race detector for all external operations.
- ThreadSanitizer is useful on supported nightly/platform combinations; document toolchain and target limitations.
- Sanitizers and model tests complement, rather than replace, safe API and invariant review.

## Upstream sources

- [Loom](https://docs.rs/loom/)
- [Tokio testing](https://tokio.rs/tokio/topics/testing)
- [Tokio tracing](https://tokio.rs/tokio/topics/tracing)
- [tokio-console](https://github.com/tokio-rs/console)
- [Rust sanitizers](https://doc.rust-lang.org/beta/unstable-book/compiler-flags/sanitizer.html)

# Concurrency Tool Selection

Select a primitive from workload semantics and measured contention, not popularity. Inspect the locked crate version, enabled features, MSRV, supported targets, cancellation behavior, and maintenance status before adoption.

## Execution and scheduling

| Tool | Use it for | Required guardrails |
|---|---|---|
| `std::thread` and scoped threads | Bounded blocking work and borrowed parallel tasks | Join every thread; bound stack size and count |
| Tokio | Readiness-driven network and process I/O | Bound admission, queues, blocking submissions, and shutdown |
| Rayon | CPU-heavy divide-and-conquer and data parallelism | Use a dedicated pool when isolation or sizing matters |
| Crossbeam deque | Custom work-stealing schedulers | Define stealing, shutdown, panic, and memory-reclamation invariants |

Do not move CPU-heavy work to `spawn_blocking` without a submission limit. Tokio permits many queued blocking tasks; `max_blocking_threads` limits active threads, not queue growth. When an async service invokes Rayon, acquire an application-level permit before submitting work and return results through a cancellation-aware boundary.

## Channels and queues

| Requirement | Candidate | Notes |
|---|---|---|
| Async bounded MPSC | `tokio::sync::mpsc` | `.send().await` propagates backpressure; define close behavior |
| Async request/reply | Tokio `mpsc` plus `oneshot` | Treat either endpoint dropping as a protocol result |
| Latest configuration or status | `tokio::sync::watch` | Intermediate values may be skipped |
| Loss-tolerant fan-out | `tokio::sync::broadcast` | Handle `Lagged`; events must be discardable or reconstructible |
| Sync bounded MPMC | `crossbeam_channel::bounded` or a measured alternative | Bounded capacity is part of the API contract |
| Lock-free bounded queue | `crossbeam_queue::ArrayQueue` | Use only when non-blocking full/empty behavior is acceptable |
| Work-stealing queue | `crossbeam_deque` | Intended for scheduler-like ownership, not ordinary messaging |

Evaluate `flume` or `kanal` only when their synchronous/asynchronous semantics and measured behavior solve a concrete requirement. Do not choose a channel from microbenchmarks alone; cancellation, disconnect, fairness, select behavior, and memory bounds matter.

## Shared state

| State shape | Candidate | Boundary |
|---|---|---|
| Short critical section | `std::sync::Mutex` or `RwLock` | Never hold a guard across `.await`, callbacks, or slow I/O |
| Async critical section that must await | `tokio::sync::Mutex` | Prefer moving I/O outside the lock when possible |
| Lower-overhead synchronous locks | `parking_lot` | Recheck poisoning and fairness expectations |
| Read-mostly immutable snapshot | `arc-swap` | Writers replace whole snapshots; account for clone/update cost |
| Independent keyed mutations | `DashMap` | Multi-key operations and entry guards need explicit ordering |
| Concurrent cache | `moka` | Set maximum capacity, weights, TTL/TTI, and eviction observability |
| Counters and flags | atomics | Specify ordering and avoid split invariants across atomics |

Prefer partitioned ownership or an actor when an invariant spans multiple fields or keys. A faster lock cannot repair an invalid atomicity boundary.

## Selection evidence

Before replacing a primitive, capture a representative benchmark or load test with throughput, p95/p99 latency, allocation rate, queue depth, lock wait, CPU, and RSS. Include overload and shutdown behavior, not only steady-state throughput.

## Upstream sources

- [Tokio](https://tokio.rs/tokio/tutorial)
- [Rayon](https://docs.rs/rayon/)
- [Crossbeam](https://docs.rs/crossbeam/)
- [parking_lot](https://docs.rs/parking_lot/)
- [arc-swap](https://docs.rs/arc-swap/)
- [DashMap](https://docs.rs/dashmap/)
- [Moka](https://docs.rs/moka/)

# Asynchronous Service Mode Production

This resource uses the `rmux` local daemon/SDK as a case study, without treating its constants or crate versions as generic defaults. The current snapshot is `3e4eabd8534aab145523de9fe97e4ad164a75ac4`; design conclusions must be measured on the target project before deployment.

## Write Resource Budgets First

Before selecting an API, answer:

| Resource | Must Define |
|---|---|
| Connection | Max authenticated/unauthenticated connections; requests in transit per connection |
| Task | Who creates them, who waits for completion, failure propagation, cancellation and timeout limits |
| Queue | Capacity, max element size, queue full behavior, close behavior |
| Subscription | Per-connection/per-resource limit, TTL (time-to-live), slow consumer handling, reconnection strategy |
| Runtime | Async worker, blocking worker, CPU isolation strategies |
| Memory | `frame`/body limits; per-connection buffers; worst-case global values |

Using "Tokio" alone does not constitute high-concurrency design. Concurrency comes from controlled workload, short polling intervals, explicit state ownership, and recoverable tasks.

## Select Channels by Semantics

| Requirement | Common Tool | Design Requirements |
|---|---|---|
| Bounded producer request queue | `tokio::sync::mpsc` (bounded) | Use `.send().await` to induce backpressure; record capacity limits |
| Single-request/response pair | `oneshot` | Both sender and receiver drop are protocol results; do not unwrap |
| Global close or latest config | `watch` | Guarantees only the latest value; handle sender drops gracefully |
| Multi-subscription instantaneous events | `broadcast` | Receiver may be `Lagged`; events must be either discardable or reconstructible |
| Synchronous thread bridging | `std::sync::mpsc` / dedicated threads | Do not block `.recv()` on async workers in synchronous contexts |

The `rmux` SDK transport uses bounded `mpsc` channels to receive commands and creates a new `oneshot` per call. A single actor serializes socket writes, pairing responses via FIFO ordering. The benefit is that protocol state has only one writer; the cost includes head-of-line blocking on producers and strict response order guarantees required by single-connection protocols.

## Actor Supervision and Task Management

An **actor** model suits scenarios where:
- A resource requires a strict sequence (e.g., socket writes, state machines, device handles);
- Many callers exist but state changes must be serialized;
- Commands and results can be modeled as enums;
- Clear error conditions are defined for queue fullness, actor exit, and pending requests.

**Checklist:**
1. Command queues are bounded;
2. When an actor exits, all pending replies fail;
3. New commands are rejected after shutdown;
4. Reader/writer sub-tasks are saved and aborted/joined on exit;
5. Terminal failures are cached for subsequent calls to ensure fast failure paths;
6. Best-effort `Drop` cleanup does not mimic strong consistency semantics.

When managing similar tasks using `JoinSet`, results return in completion order. If the API must preserve input ordering (e.g., broadcasting like `rmux`), carry input indices through and sort during aggregation. When target counts are untrusted or potentially large, add a `Semaphore` for batching; note that `JoinSet` itself does not limit concurrency levels.

## Slow Consumers and Backpressure

Explicit strategies must be chosen before applying them safely:
- **Blocking producers**: Reliable but introduces latency; suitable when requests cannot be dropped;
- **Discard and reconstruct snapshots**: Suitable for UI/render/status states where merging is acceptable;
- **Disconnect slow consumers**: Suitable for real-time streams, returning diagnostic reasons on failure;
- **Persistent replay**: Suitable for business events that must not be lost;
- **Pause upstream then resume**: Set high/low watermarks to avoid jitter near thresholds.

The `rmux` case study combined event coalescing, subscription limits/TTLs, output age caps, and low-watermark recovery. This demonstrates that "bounded channels" alone are insufficient: item size limits, subscription base sizes, and retention times must also be constrained.

## Shared State and Locking
- Large state should be partitioned by lifecycle and contention patterns; do not bundle all responsibilities under a single global `Mutex<AppState>`.
- Pure counters or flags can use atomic types, but first clarify ordering semantics and cross-variable invariants.
- Extract owned snapshots/commands from locks before releasing guards and awaiting them: `.await` after dropping the guard.
- Background tasks hold `Weak<T>`; they naturally stop when their owner releases it, avoiding circular references with `Arc`.
- Use `std::sync::Mutex` for critical sections that are extremely short and do not span await points. Use Tokio mutexes only when cross-scheduling waits are required by the async runtime.
- Do not mechanically choose between `RwLock` based on "read-heavy, write-light"; first measure lock duration and fairness characteristics.

## Runtime Is Not Larger Than Necessary

The `rmux` daemon fixed its multi-threaded runtime's async worker count to 1 because the primary workload is readiness-driven I/O; render/status operations have been merged into a single thread. Adding more workers would increase cross-thread wakeups and resident memory usage. This is not a universal constant but an architecture decision validated through measurement.

**Checklist:**
1. Identify blocking workloads: file descriptors, DNS lookups, `stdio`, CPU-intensive tasks, FFI calls;
2. Move these to `spawn_blocking` or dedicated pools;
3. Ensure the outer bounded channel/Semaphore protects potentially unbounded blocking queues/pools;
4. Compare throughput (p95/p99), wakeups, RSS against default core counts and candidate values for worker count/stack size/blocking limits via configuration tests or benchmarks.

## Close Protocol Recommendation

Recommended sequence:
```text
Stop accepting new connections / acceptors
  -> Publish shutdown signal
  -> Allow current safe points on connections/workers to terminate gracefully
  -> Cancel unnecessary tasks
  -> Join all supervised tasks
  -> Flush and release resources
  -> Return explicit failure after timeout
```

The peer's disconnection should also participate in cancellation. Long-waiting requests must not permanently occupy server-side waiters even if the client disconnects; test coverage must include "count zero after disconnect" and "no background tasks surviving shutdown."

# Concurrency References

## Key std types
| Type | Module | Use |
|------|--------|-----|
| `thread::spawn` | std::thread | OS threads |
| `Mutex<T>` | std::sync | Mutual exclusion |
| `Arc<T>` | std::sync | Atomic ref counting |
| `mpsc` | std::sync::mpsc | Multi-producer channel |
| `AtomicU64` | std::sync::atomic | Lock-free atomics |
| `tokio` | tokio crate | Async runtime |
| `rayon` | rayon crate | CPU-bound data parallelism |
| `crossbeam` | crossbeam crates | Channels, queues, deques, scoped threads |
| `loom` | loom crate | Concurrent execution model testing |

## Further reading
- std::thread docs: https://doc.rust-lang.org/std/thread/
- std::sync docs: https://doc.rust-lang.org/std/sync/
- Tokio tutorial: https://tokio.rs/tokio/tutorial
- Rayon docs: https://docs.rs/rayon/
- Crossbeam docs: https://docs.rs/crossbeam/
- Loom docs: https://docs.rs/loom/

---
name: rust-concurrency
description: Design, implement, diagnose, and test Rust concurrency and parallelism with threads, Send and Sync, locks, atomics, channels, Tokio, Rayon, Crossbeam, bounded backpressure, actor ownership, task supervision, graceful shutdown, runtime diagnostics, and Loom model tests. Use when users ask about shared state, deadlocks, async tasks, CPU parallelism, high concurrency, daemon resource budgets, slow consumers, worker pools, lock-free structures, or concurrent correctness.
---

# Rust Concurrency

> Based on the standard library `std::thread`, `std::sync`, and `std::sync::atomic` modules, along with the Async Book. Use when designing, debugging, load-testing, or reviewing threaded and async Rust code; cancellation, task ownership, lock scope, runtime sizing, queues, overload management, message passing, hand basic ownership to rust-stable and unsafe invariants to rust-unsafe-ffi.

## Capability Boundaries

### ✅ Strengths
1. OS threads (`thread::spawn`, `Builder`, `join`, scoped threads, move closures)
2. Synchronization primitives (Mutex, RwLock, Barrier, Condvar, OnceLock, LazyLock)
3. Atomic types (AtomicBool/Isize/Usize, load/store/fetch_add/swap/compare_exchange, Ordering)
4. Channels (`mpsc`: multi-producer single-consumer, Receiver, Sender)
5. `Send` / `Sync` trait system (automatic derivation and manual implementation)
6. async/await syntax with the Future trait
7. Tokio runtime (`tokio::main`, `tokio::spawn`, select!, JoinSet)
8. Async I/O foundations (`tokio::fs`, `tokio::net`, `tokio::io`)
9. Bounded queues, backpressure, slow consumers, concurrency limits and overload strategies
10. Task supervision, connection lifecycles, cancellation safety and graceful shutdown
11. CPU-bound data parallelism and dedicated Rayon pools
12. Crossbeam channels, queues, work-stealing deques, and scoped threads
13. Read-heavy snapshots, sharded maps, caches, and alternative locks when measurements justify them
14. Loom model checking and Tokio runtime diagnostics

### ⚠️ Prerequisites
1. Understanding Rust ownership model (`rust-stable`)

### ❌ Inapplicable Scenarios
1. Unsafe code concurrent execution → use `rust-unsafe-ffi` skill
2. Basic ownership/borrowing → use `rust-stable` skill

## When to Use

- "Process data with multiple threads"
- "How to write async/await"
- "Tokio runtime usage"
- "Shared data between threads"
- "Avoid data races"
- "Rate limiting and graceful shutdown in high-concurrency services"
- "Tokio channel backlog or slow consumers"

## Data Privacy

This skill does not collect, store, or transmit any user data.

---

## I. OS Threads

```rust
use std::thread;

let handle = thread::spawn(move || {
    println!("Hello from thread!");
});
handle.join().unwrap();

// Thread with configuration
let builder = thread::Builder::new()
    .name("worker".into())
    .stack_size(1024 * 1024);
let handle = builder.spawn(move || { /* ... */ }).unwrap();

// scoped threads (1.63+)
let mut v = vec![1, 2, 3];
thread::scope(|s| {
    s.spawn(|| {
        v.push(4); // borrow, no move required
    });
});
println!("{v:?}"); // v remains usable
```

## II. Synchronization Primitives

```rust
use std::sync::{Arc, Mutex, RwLock, Barrier, OnceLock, LazyLock};

// Mutex (mutual exclusion lock)
let counter = Arc::new(Mutex::new(0));
let mut handles = vec![];

for _ in 0..10 {
    let counter = Arc::clone(&counter);
    handles.push(thread::spawn(move || {
        let mut num = counter.lock().unwrap();
        *num += 1;
    }));
}

// RwLock (read-write lock)
let data = Arc::new(RwLock::new(vec![1, 2, 3]));
{
    let read = data.read().unwrap();
    assert_eq!(read.len(), 3);
} // Drop the read guard before taking the write lock.
data.write().unwrap().push(4);

// OnceLock (thread-safe lazy initialization)
static CONFIG: OnceLock<String> = OnceLock::new();
let config = CONFIG.get_or_init(|| load_config());

// LazyLock
static CACHE: LazyLock<HashMap<String, Data>> = LazyLock::new(HashMap::new);
```

## III. Atomic Operations

```rust
use std::sync::atomic::{
    AtomicBool, AtomicU64, Ordering
};

static COUNTER: AtomicU64 = AtomicU64::new(0);
COUNTER.fetch_add(1, Ordering::SeqCst);

static READY: AtomicBool = AtomicBool::new(false);
READY.store(true, Ordering::Release);
let ready = READY.load(Ordering::Acquire);

// Ordering levels
// Relaxed — no ordering guarantees (only atomicity)
// Release — write visibility
// Acquire — read visibility
// AcqRel — both reads and writes visible
// SeqCst — global sequential order (strongest, but not automatically default; explicit Ordering required for atomic operations)
```

## IV. Channels

```rust
use std::sync::mpsc;

let (tx, rx) = mpsc::channel();
thread::spawn(move || {
    tx.send(1).unwrap();
    tx.send(2).unwrap();
});
for received in rx {
    println!("Got: {received}");
}

// Multi-producer scenario
let (tx, rx) = mpsc::channel();
let tx1 = tx.clone();
```

## V. async/await

```rust
use tokio::time;

async fn do_work(id: u32) -> &'static str {
    time::sleep(time::Duration::from_secs(1)).await;
    println!("Task {id} done");
    "ok"
}

#[tokio::main]
async fn main() {
    // Concurrent execution
    let (r1, r2) = tokio::join!(do_work(1), do_work(2));

    // select!
    tokio::select! {
        result = do_work(1) => println!("task1: {result}"),
        result = do_work(2) => println!("task2: {result}"),
    }

    // tokio::spawn
    let handle = tokio::spawn(do_work(3));
    handle.await.unwrap();
}
```

## VI. Send / Sync

```rust
// T is Send if its ownership can be transferred across threads
// &T is Sync if it can be shared references across threads

// Types that are both Send + Sync: Arc<Mutex<T>>, i32, &'static str
// !Send types: Rc<T>, *const T
// !Sync types: RefCell<T>, Cell<T>

// Manual implementations are unsafe contracts. Do not add them merely to
// satisfy a compiler error; prove aliasing, lifetime, and thread-safety first.
```

## VII. Select the Execution Model

| Workload | Default starting point | Avoid |
|---|---|---|
| Many readiness-driven network operations | Tokio tasks with bounded admission | One task or buffer per unbounded input |
| CPU-heavy independent items | Rayon parallel iterators or a dedicated pool | Running long CPU work on Tokio workers |
| Blocking filesystem, FFI, or legacy APIs | Bounded `spawn_blocking` submissions or a dedicated pool | Treating Tokio's blocking queue as backpressure |
| Synchronous MPMC messaging or work stealing | Crossbeam channels, queues, or deques | Selecting lock-free structures without measurement |
| Small shared state with short critical sections | `std::sync` locks | Holding guards across `.await` or callbacks |
| Read-mostly immutable snapshots | `ArcSwap` after profiling | A concurrent map for every read-heavy value |
| Shared keyed mutable state | Sharded ownership or `DashMap` after contention tests | Multi-key operations without an atomicity design |
| Expiring concurrent cache | Moka with explicit capacity and eviction policy | An unbounded map called a cache |

Tokio is primarily for I/O concurrency; Rayon is for CPU parallelism. Mixing them requires an explicit handoff, independent concurrency limits, and shutdown ownership. Read [Concurrency Tool Selection](references/concurrency-tool-selection.md) before introducing a third-party primitive.

## Workflow

1. **Classify the workload** — separate readiness-driven I/O, CPU parallelism, blocking calls, synchronization, and durable messaging before selecting a runtime or primitive.
2. **Write concurrency budgets** — define maximum connections, in-flight tasks, queue capacity, item size, timeouts, memory, CPU pools, and shutdown deadlines.
3. **Determine state ownership** — prefer partitioned or single-writer ownership; share state only with an explicit atomicity and lock-ordering contract.
4. **Select communication semantics** — choose bounded point-to-point, request/reply, latest-value, lossy broadcast, or durable replay deliberately; specify queue-full and receiver-lag behavior.
5. **Supervise execution** — retain task or thread handles, propagate failure, contain panic, prevent orphan work, and define caller-cancellation behavior.
6. **Design graceful shutdown** — stop admission, close producers, publish cancellation, join within a deadline, flush required state, and return unresolved failures.
7. **Measure before tuning** — record throughput, p50/p95/p99 latency, queue depth, saturation, task poll time, wakeups, lock wait, CPU, allocations, and RSS.
8. **Verify the model** — test overload and cancellation, use Loom for small synchronization state machines, and use tokio-console or tracing for runtime stalls. Read [Concurrency Testing and Diagnostics](references/concurrency-testing-and-diagnostics.md).

## Gotchas

1. Mutex::lock() returns a `MutexGuard`; do not await before dropping to avoid deadlocks
2. tokio::spawn's Future must be both Send and 'static; non-Send references will cause compilation errors
3. Async closures capture ownership differently than regular closures — use the move keyword explicitly for transfer of state
4. Cancelled Futures in select! branches do not execute cleanup logic directly before dropping
5. Atomic Ordering is not relational semantics; misuse of Relaxed can lead to unexpected memory ordering issues
6. broadcast lag is distinct from normal success paths; must choose between discarding, rebuilding snapshots, disconnecting slow consumers, or persistently replaying events
7. max_blocking_threads limits only the number of blocking threads and does not provide backpressure for submission queues; high-cost tasks require Semaphore or bounded queues
8. JoinSet returns results in completion order; if API requires input ordering, carry indices through to restore sequence during aggregation
9. `DashMap`, `parking_lot`, `ArcSwap`, and lock-free queues change semantics as well as performance; benchmarks do not replace invariant review
10. Loom sees only synchronization performed through Loom-aware types and can suffer state-space explosion; keep models small and deterministic
11. Rayon work may outlive the async caller unless cancellation and pool ownership are designed explicitly

## On-Demand Resources

- [Concurrency Examples](examples/examples.md)
- [Type & Tool Quick Reference](references/references.md)
- [Concurrency Tool Selection](references/concurrency-tool-selection.md): Read when choosing Tokio, Rayon, Crossbeam, locks, sharded maps, snapshots, or caches.
- [Concurrency Testing and Diagnostics](references/concurrency-testing-and-diagnostics.md): Read when proving synchronization correctness, diagnosing runtime stalls, or load-testing overload and shutdown.
- [Production Async Service Patterns](references/production-async-services.md): Read when designing actors, backpressure, slow consumers, task supervision, runtime configuration, and shutdown protocols.
- `examples/golden-threads/`: CI-built scoped thread examples

## Official References

- [std::thread Documentation](https://doc.rust-lang.org/std/thread/)
- [std::sync Documentation](https://doc.rust-lang.org/std/sync/)
- [std::sync::atomic Documentation](https://doc.rust-lang.org/std/sync/atomic/)
- [Async Book](https://rust-lang.github.io/async-book/)
- [Tokio Guide](https://tokio.rs/tokio/tutorial)
- [Rayon](https://docs.rs/rayon/)
- [Crossbeam](https://docs.rs/crossbeam/)
- [Loom](https://docs.rs/loom/)