Go's http.Get, Concurrency, and Connection Reset by Peer

Question:

While utilizing Go routines and channels for efficient web page downloading, why do some requests fail with the "connection reset by peer" error when running from a server on a different continent?

Answer:

The "connection reset by peer" error indicates that the remote server has intentionally or involuntarily terminated the connection due to resource limitations or excessive connection initiation. To address this issue:

1. Optimize Concurrency Level:

Instead of initiating numerous parallel connections (1000-2000), experiment with different concurrency levels to determine the most efficient number. Excessively high concurrency can lead to connection resets.

2. Set Transport.MaxIdleConnsPerHost:

Configure the Transport.MaxIdleConnsPerHost value to correspond with your concurrency level. When this value is lower than the number of concurrent connections, server connections may be closed after each request and need to be re-opened. This recurrent opening and closing process can slow down the progress and potentially exceed server connection limits.

3. Delay Connection Initiations:

If the issue persists despite setting MaxIdleConnsPerHost appropriately, consider introducing a slight delay between connection initiations to avoid triggering connection resets from the server.