Go concurrency patterns: goroutines, channels, worker pools and context

Short answer: start goroutines with go f(), pass data between them with channels, share memory only under a mutex, and give every goroutine a way to stop through a context.Context. The patterns you will use most are the worker pool, fan-out and fan-in, and errgroup for running tasks in parallel and collecting the first error.

What is a goroutine?

A lightweight thread managed by the Go runtime. Starting one costs a few kilobytes, so spawning thousands is normal. go work() returns immediately and runs work concurrently.

How do channels work?

ch := make(chan int)        // unbuffered: send blocks until someone receives
buf := make(chan int, 10)   // buffered: send blocks only when full

go func() { ch <- 42 }()
v := <-ch

Close a channel from the sender side when no more values will come. Receivers can then range over it until it is drained. Never close a channel from the receiver, and never close it twice.

Worker pool

Limit concurrency to a fixed number of workers reading jobs from a channel.

func process(ctx context.Context, jobs []Job, workers int) []Result {
    in := make(chan Job)
    out := make(chan Result)

    var wg sync.WaitGroup
    for i := 0; i < workers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for j := range in {
                out <- handle(ctx, j)
            }
        }()
    }

    go func() {
        defer close(in)
        for _, j := range jobs {
            select {
            case in <- j:
            case <-ctx.Done():
                return
            }
        }
    }()

    go func() { wg.Wait(); close(out) }()

    var results []Result
    for r := range out {
        results = append(results, r)
    }
    return results
}

Without a limit, one goroutine per item can exhaust memory, database connections or an upstream rate limit.

Fan-out and fan-in

Fan-out starts several goroutines reading from the same channel. Fan-in merges multiple channels into one. The worker pool above does both: several workers consume in (fan-out), and all write to out (fan-in).

errgroup: the practical default

For "run these N things in parallel, stop on the first failure", use golang.org/x/sync/errgroup. It is shorter and safer than hand-built version.

g, ctx := errgroup.WithContext(ctx)
g.SetLimit(8)

for _, id := range ids {
    g.Go(func() error {
        return fetch(ctx, id)
    })
}
if err := g.Wait(); err != nil {
    return err
}

The derived ctx is cancelled when any task returns an error, so the others can stop early if they respect it. In Go 1.22 and later the loop variable is new for each iteration, so capturing id in a closure is safe. In older versions you had to copy it.

Context and cancellation

Pass ctx as the first argument to anything that can block. Check ctx.Done() in loops, and pass it to database and HTTP calls.

ctx, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()

select {
case res := <-doWork(ctx):
    use(res)
case <-ctx.Done():
    return ctx.Err()
}

Mutexes: when sharing is simpler

Channels are for passing ownership and coordinating. For a shared counter or cache, a mutex is simpler.

type Counter struct {
    mu sync.Mutex
    n  map[string]int
}

func (c *Counter) Inc(k string) {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.n[k]++
}

Use sync.RWMutex when reads far outnumber writes, and sync/atomic for single numeric values.

What goes wrong?

• Goroutine leaks: a goroutine blocked forever on a channel nobody reads. Always give it an exit path through ctx or a done channel.
• Data races: two goroutines touching the same variable, one writing. Run tests with go test -race. It finds them reliably.
• Deadlocks: all goroutines blocked. The runtime reports "all goroutines are asleep" in simple cases.
• Unbounded concurrency: spawning per request without limits.
• Forgetting `wg.Add` before starting the goroutine, so Wait returns too early.
• Sending on a closed channel, which panics.

Make -race part of your CI. It costs minutes and saves days. New to the language? Start with Go for backend beginners.