I have had the same problem with the 553 error after upgrading from 4.5.7 to 5.0.0. In fact I had it some months ago too after upgrading between versions, I also started getting "Undelivered Mail Returned to Sender" mails back. I could not find any errors in the logs, only bounce responses from the ISP's mail server resulting in the "Undelivered Mail Returned to Sender" mails.
Just to round off the picture a bit, I am running the zimbra server on an ADSL line with a static IP and using my ISP's mail server as a smart host (relay server) to send out going mails.
<email@example.com>: host smtprelay.isp.com[XXX.XXX.XXX.XXX] said: 553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1) (in reply to RCPT TO command)
I recall, as above, at some stage during Troubleshooting turning on and off DNS lookups in the MTA tab in the server administration console, This did not really better anything for me. So I had a close look at what the 553 error message was all about and this is where I found the answer. In my case the second point applied "retrieve (POP) your mail first before sending your messages through SMTP." i.e. you have to authenticate with the smart host before sending mail!!
The following error will be bounced back to a sender if the email address the sender is trying to send to is not fully activated:
"Reason: 553 sorry, that domain isn't in my list of allowed rcpthosts (#5.7.1)"
1. The domain name contained in the email address (being sent to) has not fully propagated yet.
Domain propagation is the process where Internet Service Providers (ISPs) around the world automatically
update their records (DNS tables) to reflect the new domain or to reflect a change in the hosting provider for the domain name. Once the changes have been made, the domain's website and email services will be activated. Please note that full domain propagation can take between 24-72 hours. To verify when your domain was last updated.
2. Your POP mail client is not configured properly. We suggest the following: Use port 587 with SMTP authentication or if using port 25, retrieve (POP) your mail first before sending your messages through SMTP.
So I had a look at the wiki article I used back then to configure the smart host checking that /opt/zimbra/conf/relay_password was still correct, I ran the following as in the wiki article to fix the problem:
postconf -e smtp_sasl_password_maps=hash:/opt/zimbra/conf/relay_password
postconf -e smtp_sasl_auth_enable=yes
postconf -e smtp_cname_overrides_servername=no
postconf -e smtp_sasl_security_options=noanonymous
Funny is that there were no auth errors from the smart host at all or other flags that would point one in the direction that authenticate before send was not working....
I hope this helps someone out there some time.