Memory Usage Discrepancy Between Go Tool Pprof and Docker Stats

While using Go 1.11, you may encounter a discrepancy in memory usage reported by Go tool pprof (runtime.MemStats.sys) and docker stats. Docker stats relies on cgroups to obtain memory usage information, which includes Page Cache and Resident Set Size (RES). On the other hand, pprof displays a fuzz value for memory usage, making it difficult to directly compare between the two.

Understanding cgroups Memory Usage

cgroups provide a way to limit and track resource usage within a container. The usage_in_bytes metric in cgroups includes both memory used by the process and cached memory. If a container reads files from the host system, the memory cached by the kernel will also be counted in usage_in_bytes.

Reclaiming Memory in Docker

If a container reaches its memory limit, Docker attempts to reclaim unused memory. This is unlike an OOM error, which occurs when all memory has been used.

Ways to Limit Memory Usage

To control the memory usage of a docker container, you can specify a memory limit in the docker run command or in the docker-compose.yml file using the mem_limit parameter.

Conclusion

The discrepancy in memory usage reporting between pprof and docker stats arises from the different ways they measure memory utilization. cgroups, as used by docker stats, include file cache memory in its calculation, while pprof presents a fuzz value without accounting for cache. By managing memory limits through cgroups, you can prevent uncontrolled memory growth in your containers.