We noticed random 5xx occurring. Fortunately, we wrote defensive code that asserts on the correct invariant. This time we had a good Stacktrace and noticed the errors:
tcp XXXX.X.X.X:XXXX->YYYY.Y.Y.Y:YYYY: read: connection reset by peer
Where XXXX is the the IP address of our machine and YYYY is the IP address of CloudSql. We discovered this thread of other people experiencing this same issue with GCP CloudSQL. https://github.com/GoogleCloudPlatform/cloud-sql-proxy/issues/343
One particular comment on the thread stood out and rubbed me the wrong way: "Hey folks - commenting on a closed/stale issue is generally considered a bad practice." As indicated by the downvotes, I'm not alone. Having had to interact with GCP DevRel in the past, I immediately knew this was someone from GCP DevRel, and I was not disappointed.They followed up a few comments later to defend themselves and shut the thread down. The bug is still live 2 years later and unrecognized.