Charlie Wang

CBC.ca blocks the default Go HTTP user agent

For some reason, CBC's website blocks the default Go HTTP client user agent string Go-http-client/1.1.

Compare

curl 'https://www.cbc.ca' \
    --silent \
    --show-error \
    -o /dev/null \
    -H "User-Agent: Go-http-client/1.1"
# curl: (92) HTTP/2 stream 1 was not closed cleanly: INTERNAL_ERROR (err 2)

to the default cURL user agent:

curl 'https://www.cbc.ca' \
    --silent \
    --show-error \
    -o /dev/null \
    -w "%{http_code}"
# 200

This happened sometime in August or September of this year, when I noticed a Go RSS client I was using stopped working for CBC. The error there was

Get "https://www.cbc.ca/webfeed/rss/rss-canada-toronto": stream error: stream ID 1; INTERNAL_ERROR; received from peer

which, along with the cURL error, suggests the connection was terminated rather crudely.

More posts