I too can't seem to get Google to accept it...
I've constructed this address to use:
I do NOT have it shared as public, nor do I believe I should have to as I am authenticating and all.
When I enter this in my browser, a proper calendar is downloaded as it should, so I'm convinced that the URL is appropriate.
However, google, after awhile, says: "Could not fetch the url."
I can't see how this is a firewall issue, as I can access my Zimbra server from other places without issue.