Another Letsencrypt method

Discuss your pilot or production implementation with other Zimbra admins or our engineers.
phoenix
Ambassador
Ambassador
Posts: 26063
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Another Letsencrypt method

Postby phoenix » Wed Apr 03, 2019 10:58 am

JDunphy wrote:Hi Bill,

I am making it worse the more I add to that wiki article. Yikes! ;-)
Hi Jim

Just a quick note to let you know I'll give this a shot later today. The wiki article is great and that's not the problem, it's most likely the way my brain works and don't forget I don't do this for a living. :)

I'll give a full reply hopefully later today.


Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
phoenix
Ambassador
Ambassador
Posts: 26063
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Another Letsencrypt method

Postby phoenix » Wed Apr 03, 2019 7:20 pm

Hi Jim

Success is in the air, with a minor hiccup. :)

Just to clarify, I did follow the instructions to get the manual process configured for a normal user and that went well for a few renewals. I thought recently that I should employ the deploy script to make it easier, that's where my problem started by installing it as the zimbra user. Running the exact commands you mentioned in your last post:

Code: Select all

% su -
# cd /opt/zimbra/
# mkdir .acme.sh
# chown zimbra:zimbra .acme.sh
# su - zimbra
% wget -O -  https://get.acme.sh | sh
In that example it still fails with the permissions error I mention in my earlier posts.

Next was the install as a normal user then copy the directory to the zimbra directories, once installed I ran:

Code: Select all

cp -r /home/acme/.acme.sh /opt/zimbra/.acme.sh
chown -R zimbra:zimbra /opt/zimbra/.acme.sh
OK, so far but running the commands to issue the certificate failed with a permissions error for the log file. It appears the log file location is created during the acme script install and therefore points to a directory that was created in the original normal user directory. That needed to be changed:

Code: Select all

vi /opt/zimbra/.acme.sh/account.conf
Then modify the log file entry to point to the new zimbra user directory and include my CloudFlare API:

Code: Select all

LOG_FILE="/opt/zimbra/.acme.sh/acme.sh.log"
Once that was done the issue and deployment and restart of ZCS went well, certificates installed and ZCS up with new certs.

Thanks for all your help with this problem. :)
:
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 415
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 8.7.11_P12 RHEL6 Network Edition
Contact:

Re: Another Letsencrypt method

Postby JDunphy » Wed Apr 03, 2019 8:11 pm

That's great Bill...

Looks like it needed write permission in the local directory so simple fix is to change directory to make the first way work. Could also be 'cd /tmp'. Here is that additional step with the addition of that 'cd' command if /opt/zimbra is owned by root.

Code: Select all

% su -
# cd /opt/zimbra/
# mv .acme.sh .acme.sh-
# mkdir .acme.sh
# chown zimbra:zimbra .acme.sh
# su - zimbra
% cd /opt/zimbra/.acme.sh
% wget -O -  https://get.acme.sh | sh

Here is the entire fail and success that you mentioned. First the fail without switching to a directory where the zimbra user has write permission.

Code: Select all

[zimbra@tmail ~]$ ls -ald .
drwxr-xr-x 54 root root 4096 Apr  3 12:29 .
[zimbra@tmail ~]$ pwd
/opt/zimbra
[zimbra@tmail ~]$ wget -O -  https://get.acme.sh | sh
--2019-04-03 12:30:15--  https://get.acme.sh/
Resolving get.acme.sh... 2607:5300:201:3100::5663, 144.217.161.63
Connecting to get.acme.sh|2607:5300:201:3100::5663|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 705 [text/plain]
Saving to: `STDOUT'

100%[=====================================================================================================================================================>] 705         --.-K/s   in 0s     

2019-04-03 12:30:15 (119 MB/s) - written to stdout [705/705]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  174k  100  174k    0     0   619k      0 --:--:-- --:--:-- --:--:--  658k
[Wed Apr  3 12:30:16 PDT 2019] Installing from online archive.
[Wed Apr  3 12:30:16 PDT 2019] Downloading https://github.com/Neilpang/acme.sh/archive/master.tar.gz
sh: line 5827: master.tar.gz: Permission denied
[Wed Apr  3 12:30:16 PDT 2019] Download error.

Followed by the success... switching to directory where zimbra has write permission

Code: Select all

[zimbra@tmail ~]$ cd .acme.sh
[zimbra@tmail .acme.sh]$ wget -O -  https://get.acme.sh | sh
--2019-04-03 12:30:38--  https://get.acme.sh/
Resolving get.acme.sh... 2607:5300:201:3100::5663, 144.217.161.63
Connecting to get.acme.sh|2607:5300:201:3100::5663|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 705 [text/plain]
Saving to: `STDOUT'

100%[=====================================================================================================================================================>] 705         --.-K/s   in 0s     

2019-04-03 12:30:38 (103 MB/s) - written to stdout [705/705]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  174k  100  174k    0     0  1383k      0 --:--:-- --:--:-- --:--:-- 1430k
[Wed Apr  3 12:30:38 PDT 2019] Installing from online archive.
[Wed Apr  3 12:30:38 PDT 2019] Downloading https://github.com/Neilpang/acme.sh/archive/master.tar.gz
[Wed Apr  3 12:30:39 PDT 2019] Extracting master.tar.gz
[Wed Apr  3 12:30:39 PDT 2019] Installing to /opt/zimbra/.acme.sh
[Wed Apr  3 12:30:39 PDT 2019] Installed to /opt/zimbra/.acme.sh/acme.sh
[Wed Apr  3 12:30:39 PDT 2019] Installing alias to '/opt/zimbra/.bashrc'
[Wed Apr  3 12:30:39 PDT 2019] OK, Close and reopen your terminal to start using acme.sh
[Wed Apr  3 12:30:39 PDT 2019] Installing cron job
[Wed Apr  3 12:30:39 PDT 2019] Good, bash is found, so change the shebang to use bash as preferred.
[Wed Apr  3 12:30:39 PDT 2019] OK
[Wed Apr  3 12:30:39 PDT 2019] Install success!
[zimbra@tmail .acme.sh]$

I have updated the wiki article.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 415
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 8.7.11_P12 RHEL6 Network Edition
Contact:

Re: Another Letsencrypt method

Postby JDunphy » Sun Apr 07, 2019 4:24 pm

I have had a few people surprised they have automatically renewed and loaded letsencrypt certificates without intervention. Here is how and why:

This only happens if you chose the automatic DNS validation method with the zimbra deploy method and installed acme.sh using the zimbra user. If you never commented out that cron entry, you will find your zimbra servers with new certificates every 60 days installed and zimbra restarted to load your new certificate. During the acme.sh install, this entry is created and you will find this at the bottom of zimbra's crontab.

Code: Select all

18 0 * * * "/opt/zimbra/.acme.sh"/acme.sh --cron --home "/opt/zimbra/.acme.sh" > /dev/null

acme.sh runs every night but won't pull a new certificate unless it matches a threshold or is forced from the command line. So how does it know what to do using just the --cron and --home command line arguments? If you initially did this:

Code: Select all

% acme.sh --issue --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org
% acme.sh --issue --deploy --deploy-hook zimbra --dns dns_cf -d mail.example.com -d mail.example.net -d mail.example.org

You will find that /opt/zimbra/.acme.sh/mail.example.com directory has a file: mail.example.com.conf which contains everything necessary to first renew your certificates and what DeployHook you used.

That means that if you run the above 2 commands to initially install a certificate and then deploy it with the Zimbra deploy hook you would never have to run any commands manually again and your certificates will be renewed without intervention.

This was one of the advantages of moving to the zimbra user and using the automatic DNS method with a deploy hook since you don't have to stop the proxy while attempting to validate your new certificates like with other methods. Warning: the zmcontrol restart has bugs that may not restart some daemons.. postfix is one example that I outlined that hopefully Zimbra is in the process of fixing.
Bug: https://forums.zimbra.org/viewtopic.php?f=15&t=65332&p=288882&hilit=potential+bug+mta#p288882 I have my systems patched with my proposed fix since this happened to me and have not had a problem since. I ran renews every day for 3 months on a test server so the process seems sound at this point.
phoenix
Ambassador
Ambassador
Posts: 26063
Joined: Fri Sep 12, 2014 9:56 pm
Location: Liverpool, England

Re: Another Letsencrypt method

Postby phoenix » Wed Apr 17, 2019 9:13 am

Hi Jim

Just a quick update.

I've recently had to rebuild my production server and, obviously, new certificates had to be installed. Thanks to your comments about about the permissions 'problem' it all worked a treat. The install of the acme client worked OK after 'fixing' the account.conf file, then a 'test' issue and then your Deploy script.. :)

The only thing worth a mention is that a copy/paste of your deploy script from the wiki gives rather strange formatting when pasted into a konsole editor, I'm assuming that's caused by the 'wiki'?

Thanks again for all your help, I can now be satisfied that they're easy to install (but still a pain to completely understand for me).
Regards

Bill

Rspamd: A high performance spamassassin replacement

If you'd like to see this implemented in a future version of ZCS then please vote on Bugzilla entries 97706 & 108168
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 415
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 8.7.11_P12 RHEL6 Network Edition
Contact:

Re: Another Letsencrypt method

Postby JDunphy » Wed Apr 17, 2019 2:27 pm

Thanks for the feedback Bill.

I updated the wiki with this link: https://github.com/JimDunphy/acme.sh/blob/master/deploy/zimbra.sh which may help with the cut/paste. I am becoming more confident to having this added to acme.sh mainline but wanted to make sure we had enough testing with it. It has become incredibly simple to add certificates and renew them automatically with this automatic DNS method. I use them for all our certs across any platform and any type of server... as of 6 months ago I now use them inside RFC1918 address space by using the --challenge-alias in conjunction with this DNS method. Sometimes, I think its too automated. I recently added that logger statement so I can have swatch look for it and notify me that my certificates have been swapped out because they were just happening without my knowledge and I still need to verify that zmcontrol restart actually restarted everything and not just say it did. :-)

Hint: To see when they will be updated by that built-in cronjob entry created by acme.sh during its install.

Code: Select all

# su - zimbra
% cd .acme.sh
./acme.sh --list
Main_Domain     KeyLength  SAN_Domains                              Created                       Renew
...
...


Note: Letsencrypt is moving on July 8 to signing these using their own ISRG Root X1 key (Internet Security Research Group) .... https://letsencrypt.org/2019/04/15/transitioning-to-isrg-root.html ... This means that the cross-signature from IdentTrust won't be needed since the browsers have long had this ISRG Root X1 public key included in their trusted CA's. The problem is zimbra and its verify option with the java keystore. You can see what they have listed with this command:

Code: Select all

# su - zimbra
% keytool -list -v  -keystore /opt/zimbra/common/lib/jvm/java/jre/lib/security/cacerts -storepass changeit |grep -i owner

It appears they have not updated that keystore in ages so I'll be doing a little testing and perhaps one solution might be to add the new ISRG Root X1 public key directly via the zimbra deploy script with acme.sh ... In any event, I will keep everyone posted if we can expect problems but still too early to tell. This problem will affect all letsencrypt methods for Zimbra since we currently just chain the IdentTrust key with our signed key. I always knew that we would eventually have to update the IdenTrust key we are using in 2021 so we have some options to weigh what is best. Still lots of time as this is very fluid. The acme protocol is also tracking to become a standard so that bodes well for all commercial CA's - https://tools.ietf.org/html/rfc8555 ... The future might be using acme clients like acme.sh for any commercial or free Certificate. Who knew? That could significantly make certificate creation and renewals vastly easier through the zimbra admin console.

If anyone from Zimbra/Syncor is reading this. You could do the community a big favour by sneaking in that ISRG Root X1 during one of your patch updates. It could save your tech support a lot of man hours should they not cross sign by default after July 8 for certificates.
User avatar
JDunphy
Outstanding Member
Outstanding Member
Posts: 415
Joined: Fri Sep 12, 2014 11:18 pm
Location: Victoria, BC
ZCS/ZD Version: 8.7.11_P12 RHEL6 Network Edition
Contact:

Re: Another Letsencrypt method

Postby JDunphy » Thu Jun 13, 2019 4:19 pm

Another note on this... It's been a few years without problems but last night's renewal showed me something new. zmcontrol restart had a few problems.
Nothing cert related but related to how zimbra determines when a process is running to know when to restart it. I had already patched and reported the MTA bug
https://forums.zimbra.org/viewtopic.php?f=15&t=65332&hilit=potential+bug+mta

so postfix will always restart properly but it appears there are a lot more places. The pattern we are looking for is 'kill -0' in their startup scripts.

Given the recent security threats, I thought I would share what changes when you replace a certificate... Courtesy of a morning tripwire report:

Code: Select all

added: /opt/zimbra/.acme.sh/ca/acme-v02.api.letsencrypt.org
added: /opt/zimbra/.acme.sh/ca/acme-v02.api.letsencrypt.org/ca.conf
added: /opt/zimbra/.acme.sh/ca/acme-v02.api.letsencrypt.org/account.json
added: /opt/zimbra/.acme.sh/ca/acme-v02.api.letsencrypt.org/account.key
changed: /opt/zimbra/ssl/.rnd
changed: /opt/zimbra/ssl/zimbra/jetty.pkcs12
changed: /opt/zimbra/ssl/zimbra/commercial/commercial.crt
changed: /opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt
changed: /opt/zimbra/conf/slapd.crt
changed: /opt/zimbra/conf/smtpd.crt
changed: /opt/zimbra/conf/ca/commercial_ca_1.crt
changed: /opt/zimbra/conf/nginx.crt
changed: /opt/zimbra/common/etc/java/cacerts
changed: /opt/zimbra/.acme.sh/http.header
changed: /opt/zimbra/.acme.sh/mail.example.com/mail.example.com.cer
changed: /opt/zimbra/.acme.sh/mail.example.com/ca.cer.real
changed: /opt/zimbra/.acme.sh/mail.example.com/fullchain.cer
changed: /opt/zimbra/.acme.sh/mail.example.com/mail.example.com.conf
changed: /opt/zimbra/.acme.sh/mail.example.com/ca.cer
changed: /opt/zimbra/.acme.sh/ca

Note: the changing to the latest acme protocol version 2 is not normal but you can expect it for acme.sh v2.8.2
Now what didn't start?

Code: Select all

$ zmcontrol status
Host mail.example.com
   amavis                  Running
   antispam                Running
   antivirus               Running
   convertd                Running
   ldap                    Running
   logger                  Stopped
      zmlogswatchctl is not running
   mailbox                 Running
   memcached               Running
   mta                     Running
   opendkim                Running
   proxy                   Running
   service webapp          Running
   snmp                    Stopped
      zmswatch is not running.
   spell                   Running
   stats                   Running
   zimbra webapp           Running
   zimbraAdmin webapp      Running
   zimlet webapp           Running
   zmconfigd               Running

The solution was simple enough:

Code: Select all

# su - zimbra
% zmlogswatchctl start
% zmswatchctl start

This is hardly enterprise software.

Code: Select all

grep 'kill -0' zmlogswatchctl zmswatchctl
zmlogswatchctl:    kill -0 $pid 2> /dev/null
zmlogswatchctl:          kill -0 $zmrrdfetchpid 2> /dev/null
zmlogswatchctl:          kill -0 $zmrrdfetchpid 2> /dev/null
zmlogswatchctl:        kill -0 $pid 2> /dev/null
zmswatchctl:    kill -0 $pid 2> /dev/null
zmswatchctl:        kill -0 $pid 2> /dev/null

Looks like I have more patching to do around here. BTW, I don't see the point of reporting and showing them bug fixes if I can't get zmmtastatus patched on 8.7.11. Those that rely on zmcontrol restart to work reliably are on borrowed time if you do unattended automatic restarts for things like certificate renewal or backups. I guess I might go back to doing only ldap,postfix,nginx reloads and mailboxd restarting given that reality. The deploy hook I am using with acme.sh has them present so its easy enough to make that change and comment out the zmcontrol restart.

Hint: My biggest issues is that certificates get replaced without me noticing so I have a swatch monitor which runs on our central syslog machine notify me after the acme.sh Zimbra hook issues a logger command that it was renewed... I also have a script that does acme.sh --list and parses the renewal date to send me an email 1 day before I can expect to see a new certificate which is currently every 60 days. That keeps me in the loop. I can post this if there is interest. With a single zimbra instance this isn't a problem but if you do it for everything that isn't zimbra related that can be a lot of certs to keep track off. Fortunately, I have never had a problem with web farms, grafana, plex ... just the commercial stuff we pay for it would seem. The irony eh? :-)

Return to “Administrators”

Who is online

Users browsing this forum: Bing [Bot] and 14 guests