Go is fast, efficient and well suited to services that need to handle thousands of requests quickly. That combination makes it popular with teams building APIs, integrations, payment platforms and other high-throughput systems.
But Go does not make a service automatically reliable.
A small mistake involving a response body, goroutine, cache or shared lock can remain hidden for weeks. Under normal traffic, the service appears healthy. Then demand increases, memory climbs, queues grow and one struggling component starts affecting everything around it.
This case study looks at real production incidents involving Go services and the lessons they offer growing businesses. It also explains how disciplined Go Development, observability and dependable IT infrastructure solutions can prevent a performance problem from becoming an expensive outage.
Case study: when a Go service keeps consuming memory
Nylas has described a real incident involving Go services running in Kubernetes. Their monitoring showed pods repeatedly restarting without an obvious application error. Memory usage continued to rise until each pod reached its configured limit.
When that limit was exceeded, Kubernetes restarted the pod. During the restart window, the API could become temporarily unavailable.
The investigation used Go’s built-in profiling tools, including pprof, to examine heap allocations, active goroutines and memory-consuming functions. The eventual cause was an HTTP response body that was not closed after a service made an outbound request.
That sounds like a minor oversight. In a long-running, high-throughput service, it is not.
Every request that leaves a response body open can retain resources longer than intended. Repeated thousands of times, the effect accumulates. The service may continue responding for a while, but its memory and connection usage gradually move in the wrong direction.
You can read the full Nylas case study on finding a memory leak in a Go service.
“The most dangerous production failures are often caused by code that is almost correct.”
Why this incident matters
The issue was not an exotic failure in the Go runtime. It was a lifecycle problem: a resource was created but not reliably released.
That pattern appears in many forms:
- HTTP response bodies that are not closed
- Files or database rows left open
- Goroutines waiting indefinitely on a channel
- Tickers and background workers without cancellation
- Global slices or maps that grow without a limit
- Caches without expiry or eviction
- Large subslices that retain an entire underlying buffer
Garbage collection helps Go reclaim memory, but it cannot reclaim objects that are still reachable. If a goroutine, global variable or open resource retains a reference, the garbage collector cannot safely remove it.
A second lesson: not every memory problem is a memory leak
Detectify also published a useful investigation into a Go microservice that appeared to have a memory leak. Its memory usage gradually increased until the team experienced out-of-memory errors or needed to restart the service.
The investigation used pprof to compare heap behaviour with the memory reported by the container. The team found that a function was creating significant memory pressure by fetching, transforming and appending large amounts of data.
However, the heap profile did not show a conventional leak. The Go runtime had allocated memory during a peak workload and was retaining some of it for future use rather than immediately returning it to the operating system.
The practical result still mattered: the container appeared to consume far more memory than expected.
The team improved the code by reducing the amount of data held in memory and investigated Go runtime behaviour, including how unused memory was returned to the operating system. Their experience is a reminder that diagnosing production memory issues requires more than looking at a single dashboard.
Read the full Detectify investigation into a Go microservice memory problem.
How a memory problem becomes an outage
Memory pressure is only one part of the failure. The more serious risk comes from how other components respond.
Imagine a service that handles customer requests and calls two downstream APIs. A small leak causes garbage collection to run more often. As CPU usage rises, response times increase. The upstream service sees timeouts and retries the requests.
Those retries add more work to an already stressed service.
Soon, the system may experience:
- Higher memory use
- Longer garbage collection pauses
- Slower responses
- Increased client retries
- Growing request queues
- More goroutines waiting for work
- Container restarts or out-of-memory kills
- Errors passed back to customers and connected services
This is a feedback loop. Scaling out may provide temporary relief, but if every new instance contains the same defect, the additional capacity only delays the failure.
Google’s Shakespeare Search postmortem documents a similar pattern at a different scale: an unusual surge in demand exposed a latent resource leak, leading to a cascading failure. The service was not simply “too small”. A hidden defect became dangerous when traffic changed.
The concurrency bottleneck hiding inside a fast service
Memory leaks are not the only scaling vulnerability in Go.
Go makes it easy to start concurrent work with goroutines. That is powerful, but concurrency still needs boundaries. A service that starts one goroutine per request without a clear cancellation path can eventually create thousands of unnecessary workers.
Likewise, a single shared mutex may be harmless at low traffic but become a serious bottleneck at high throughput.
A common failure pattern looks like this:
- Requests arrive concurrently.
- Each request needs access to a shared cache or map.
- A global mutex protects the entire structure.
- Reads and writes queue behind one lock.
- Latency increases as contention grows.
- Clients retry slow requests.
- More requests compete for the same lock.
At that point, the service may have plenty of CPU available, yet still perform poorly because work is being serialised around one shared resource.
What robust Go Development does differently
Preventing these incidents begins during architecture and code review, not when the first alert fires.
1. Define ownership and lifecycle
Every goroutine should have a reason to exist and a clear way to stop. Use context.Context for cancellation and ensure background workers respond to shutdown signals.
Every external resource should have an explicit lifecycle:
- Close HTTP response bodies
- Set network and database timeouts
- Release files and streams
- Stop tickers
- Drain or cancel worker queues
- Handle error paths as carefully as success paths
2. Put limits around everything that can grow
A cache without a maximum size is not a performance feature. It is a future memory incident.
Use appropriate controls such as:
- Maximum cache size
- TTL or LRU eviction
- Bounded channels
- Worker pools
- Request body limits
- Pagination and streaming
- Connection pool limits
- Maximum queue depth
The objective is not to eliminate growth. It is to make growth predictable.
3. Avoid unnecessary shared state
Shared state creates both memory retention and concurrency risk. Where possible, keep data local to a request or worker. Where sharing is necessary, consider sharded locks, read/write locks or data structures suited to the access pattern.
The right choice depends on the workload. An expert Go developer measures contention rather than assuming that a familiar primitive will scale.
4. Build observability into the service
CPU and HTTP latency are useful, but they are not enough. A production Go service should also expose and monitor:
- Heap allocation and resident memory
- Garbage collection duration
- Goroutine count
- Open connections
- Queue depth
- Mutex and blocking profiles
- Request rate and error rate
- Restart and out-of-memory events
Go’s pprof is particularly valuable for investigating heap, goroutine, mutex and blocking behaviour. It must be protected behind authentication or an internal network boundary because profiling endpoints can reveal sensitive service information.
5. Test failure modes, not just successful requests
A load test that confirms response times under ideal conditions is incomplete.
Resilient Go Development should test:
- Downstream timeouts
- Partial dependency failure
- Slow database responses
- Retry storms
- Large payloads
- Long-running connections
- Sustained traffic over several hours
- Container memory limits
- Graceful shutdown
- Deployment rollback
This is where many scaling vulnerabilities are discovered before customers experience them.
Key Insights
- Go’s garbage collector does not prevent retained objects, open resources or goroutine leaks.
- A service can suffer serious memory pressure without having a conventional heap leak.
- Global locks and unbounded concurrency can turn high throughput into high contention.
- Retries can amplify a local slowdown into a wider service outage.
- Memory, goroutines, queues and dependency health should be monitored together.
- Load testing must include failure conditions and sustained traffic, not only normal requests.
- Good architecture, profiling and operational ownership are more valuable than simply adding servers.
Why infrastructure and development should work together
Application code does not operate in isolation. Memory limits, deployment strategy, monitoring, backups, security controls and incident response all influence the outcome of a software failure.
That is why growing businesses benefit from combining expert development with dependable managed IT services. A capable partner can help review the application, harden the hosting environment, monitor service behaviour and coordinate a response when something changes.
At Picnic IT, our development service covers websites, business applications, integrations and ongoing care. Our managed IT services provide proactive monitoring and practical technology support, while our cybersecurity services help protect the infrastructure around the application.
The result is a more joined-up approach to IT infrastructure solutions: code, hosting, security and support considered as parts of the same business-critical system.
Conclusion: performance is designed before it is measured
High-throughput Go microservices rarely fail because Go is inherently unreliable. They fail when resource lifecycles are unclear, concurrency is left unbounded, dependencies are allowed to retry without control or monitoring only looks at the surface.
The encouraging part is that these risks are manageable.
With disciplined Go Development, bounded architecture, profiling, realistic load testing and proactive infrastructure support, businesses can identify weaknesses while they are still engineering problems rather than customer-facing outages.
If your Go service is approaching a major growth milestone, now is the time to review its memory behaviour, concurrency model and failure paths. Contact Picnic IT to discuss how your development and IT infrastructure can be made more resilient before the next traffic spike tests it for you.