Page 1 of 3 123 LastLast
Results 1 to 10 of 26

Thread: Zimbra 5.0.18 Upgrade to 6.0.4 - Some Issues

  1. #1
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default Zimbra 5.0.18 Upgrade to 6.0.4 - Some Issues

    Last night/this morning we upgraded our production Zimbra system from 5.0.18 to 6.0.4, unfortunately not without some drama. So first, I have to give extraordinary kudos to Irfan at Zimbra who worked with us through the wee hours to get our system functional for end users before the sun came up this morning.

    To be fair, our system has been in production since 4.0.3, and though we have been doing a lot of testing prior to this upgrade, this was probably the least smooth major version upgrade we have had with Zimbra. Not complaining mind you; having headed a multi-million dollar software development project I am of the strong opinion that Zimbra's QA is very, very good. These things happen, so I thought it would be helpful to post here some of what we are seeing. We have a support ticket open with Zimbra on these issues.

    First, the upgrade installer destroyed the GoDaddy commercial certs, which needed to get reinstalled. But since the certificate directories have changed a little versus the documentation we were using, we started getting a little nervous following the documentation until we sorted that out. This seems to be a common problem, so we were prepared.

    Second, we use syslog-ng on SLES10 SP3, and the upgrade installer broke that (thanks Irfan for fixing that one).

    Third, even after syslogging was fixed, the Admin Console functionality remains diminished. We have no "Certificates" tab in the left-hand nav bar, and the Server Status has all red "X"s. The License tab under Servers > Global Settings has also gone missing, so we'll need to use the CLI to replace our soon-to-be-expiring ZCS license. I expect we will find a few more broken bits in the coming days.

    Fourth, LDAP replication was totally broken by this upgrade (and at this point remains broken). Our additional MTA and mailbox servers, which were originally LDAP replicas for performance, now talk to the LDAP master directly. Zimbra support has escalated this issue for us. This error seems to have impacted one other poster in the forums too:
    http://www.zimbra.com/forums/install...x-upgrade.html

    Fifth, we are discovering as the day unfolds a few non-mission critical differences between 5.0 and 6.0 that cause confusion for end users. For example, it used to be that if you had a COS on a domain and an account which had the Documents feature enabled but where domain-level Documents had not been created, then when users logged in, the Documents tab simply would not display. Under 6.0.4, the Documents tab does display, and when users click on the Documents tab they get a jetty 500 error -- and call us. It's as if the COS settings now have new preferences over the domain settings, which now requires us to add a few more COSs. Not a big deal for sure, but some end users get very nervous over errors like these--more so the ones who have Apache experience!

    Overall, nothing really show-stopping here; we keep enough spare horsepower on our servers so that our LDAP master is perfectly capable of handling the load until Zimbra figures out the issue. And again, Zimbra support has been really, really terrific.

    But if you are considering upgrading a ZCS system which has been in production as long as ours, at least from our experience it appears that the upgrade to 6.0.x may not be without some bumps in the road.

    Hope that helps,
    Mark

  2. #2
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    Quote Originally Posted by vavai View Post
    Hi LMStone,

    Thanks for sharing your upgrading experience. Just a quick question : So now you are currently running ZCS 6.0.4 on SLES 10 SP3, aren't you ?

    I found some issue installing (clean install) ZCS 6.0.4 on SLES 11 64 bit regarding syslog-ng. The problem also hit me on clean install ZCS 6.0.4 on SLES 10. Would you like to share how to fix the syslog-ng problem ?

    Thread regarding my experience installing ZCS on SLES 11.

    Yes, we are now running ZCS-NE 6.0.4 on SLES10-SP3 (I'll update my profile shortly!)

    Here is the /etc/syslog-ng/syslog-ng.conf file as tweaked by Zimbra for us earlier today:

    Code:
    #
    # /etc/syslog-ng/syslog-ng.conf
    #
    # Automatically generated by SuSEconfig on Sat Apr  5 16:28:11 EDT 2008.
    #
    # PLEASE DO NOT EDIT THIS FILE!
    #
    # you can modify /etc/syslog-ng/syslog-ng.conf.in instead
    #
    #
    #
    # File format description can be found in syslog-ng.conf(5)
    # and /usr/share/doc/packages/syslog-ng/syslog-ng.txt.
    #
    
    #
    # Global options.
    #
    options { long_hostnames(off); sync(0); perm(0640); stats(3600); };
    
    #
    # 'src' is our main source definition. you can add
    # more sources driver definitions to it, or define
    # your own sources, i.e.:
    #
    #source my_src { .... };
    #
    source src {
    	#
    	# include internal syslog-ng messages
    	# note: the internal() soure is required!
    	#
    	internal();
    
    	#
    	# the following line will be replaced by the
    	# socket list generated by SuSEconfig using
    	# variables from /etc/sysconfig/syslog:
    	#
    	unix-dgram("/dev/log");
    	unix-dgram("/var/lib/named/dev/log");
    
    	#
    	# uncomment to process log messages from network:
    	#
    	udp(ip("0.0.0.0") port(514));
    };
    
    
    #
    # Filter definitions
    #
    filter f_iptables   { facility(kern) and match("IN=") and match("OUT="); };
    
    filter f_console    { level(warn) and facility(kern) and not filter(f_iptables)
                          or level(err) and not facility(authpriv); };
    
    filter f_newsnotice { level(notice) and facility(news); };
    filter f_newscrit   { level(crit)   and facility(news); };
    filter f_newserr    { level(err)    and facility(news); };
    filter f_news       { facility(news); };
    
    filter f_mailinfo   { level(info)      and facility(mail); };
    filter f_mailwarn   { level(warn)      and facility(mail); };
    filter f_mailerr    { level(err, crit) and facility(mail); };
    filter f_mail       { facility(mail); };
    
    filter f_cron       { facility(cron); };
    
    filter f_local      { facility(local0, local1, local2, local3,
                                   local4, local5, local6, local7); };
    
    filter f_acpid      { match('^\[acpid\]:'); };
    filter f_netmgm     { match('^NetworkManager:'); };
    
    filter f_messages   { not facility(news, mail) and not filter(f_iptables); };
    filter f_warn       { level(warn, err, crit) and not filter(f_iptables); };
    filter f_alert      { level(alert); };
    
    
    #
    # Most warning and errors on tty10 and on the xconsole pipe:
    #
    destination console  { pipe("/dev/tty10"    group(tty) perm(0620)); };
    log { source(src); filter(f_console); destination(console); };
    
    destination xconsole { pipe("/dev/xconsole" group(tty) perm(0400)); };
    log { source(src); filter(f_console); destination(xconsole); };
    
    # Enable this, if you want that root is informed immediately,
    # e.g. of logins:
    #
    #destination root { usertty("root"); };
    #log { source(src); filter(f_alert); destination(root); };
    
    
    #
    # News-messages in separate files:
    #
    destination newscrit   { file("/var/log/news/news.crit"
                                  owner(news) group(news)); };
    log { source(src); filter(f_newscrit); destination(newscrit); };
    
    destination newserr    { file("/var/log/news/news.err"
                                  owner(news) group(news)); };
    log { source(src); filter(f_newserr); destination(newserr); };
    
    destination newsnotice { file("/var/log/news/news.notice"
                                  owner(news) group(news)); };
    log { source(src); filter(f_newsnotice); destination(newsnotice); };
    
    #
    # and optionally also all in one file:
    # (don't forget to provide logrotation config)
    #
    #destination news { file("/var/log/news.all"); };
    #log { source(src); filter(f_news); destination(news); };
    
    
    #
    # Mail-messages in separate files:
    #
    destination mailinfo { file("/var/log/mail.info"); };
    log { source(src); filter(f_mailinfo); destination(mailinfo); };
    
    destination mailwarn { file("/var/log/mail.warn"); };
    log { source(src); filter(f_mailwarn); destination(mailwarn); };
    
    destination mailerr  { file("/var/log/mail.err" fsync(yes)); };
    log { source(src); filter(f_mailerr);  destination(mailerr); };
    
    #
    # and also all in one file:
    #
    destination mail { file("/var/log/mail"); };
    log { source(src); filter(f_mail); destination(mail); };
    
     
    #
    # acpid messages in one file:
    #
    destination acpid { file("/var/log/acpid"); };
    log { source(src); filter(f_acpid); destination(acpid); flags(final); };
    
    #
    # NetworkManager messages in one file:
    #
    destination netmgm { file("/var/log/NetworkManager"); };
    log { source(src); filter(f_netmgm); destination(netmgm); flags(final); };
    
    
    #
    # Cron-messages in one file:
    # (don't forget to provide logrotation config)
    #
    #destination cron { file("/var/log/cron"); };
    #log { source(src); filter(f_cron); destination(cron); };
    
    
    #
    # Some boot scripts use/require local[1-7]:
    #
    destination localmessages { file("/var/log/localmessages"); };
    log { source(src); filter(f_local); destination(localmessages); };
    
    
    #
    # All messages except iptables and the facilities news and mail:
    #
    destination messages { file("/var/log/messages"); };
    log { source(src); filter(f_messages); destination(messages); };
    
    
    #
    # Firewall (iptables) messages in one file:
    #
    destination firewall { file("/var/log/firewall"); };
    log { source(src); filter(f_iptables); destination(firewall); };
    
    
    #
    # Warnings (except iptables) in one file:
    #
    destination warn { file("/var/log/warn" fsync(yes)); };
    log { source(src); filter(f_warn); destination(warn); };
    
    #
    # Enable this, if you want to keep all messages in one file:
    # (don't forget to provide logrotation config)
    #
    #destination allmessages { file("/var/log/allmessages"); };
    #log { source(src); destination(allmessages); };
    
    
    filter f_local0       { facility(local0); }; # zimbra
    destination zmail { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra 
    log { source(src); filter(f_mail); destination(zmail); }; # zimbra
    destination local0 { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
    log { source(src); filter(f_local0); destination(local0); }; # zimbra
    filter f_auth       { facility(auth); }; # zimbra
    destination zmauth { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
    log { source(src); filter(f_auth); destination(zmauth); }; # zimbra
    The "destination local1" and related parameters for the second zimbra log file did not work; syslog refused to start.

    Hope that helps!
    Mark

  3. #3
    Join Date
    Mar 2007
    Location
    Chicago, IL
    Posts
    1
    Rep Power
    8

    Default LDAP replication

    Quote Originally Posted by LMStone View Post
    Fourth, LDAP replication was totally broken by this upgrade (and at this point remains broken). Our additional MTA and mailbox servers, which were originally LDAP replicas for performance, now talk to the LDAP master directly. Zimbra support has escalated this issue for us.
    Mark,

    Thanks for the rundown. If you can, any status updates on resolving the LDAP replication issue would be much appreciated. We have production servers running since 4.x as well, and plan to move from 5.x to 6.x soon.

  4. #4
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    Quote Originally Posted by JBerbaum View Post
    Mark,

    Thanks for the rundown. If you can, any status updates on resolving the LDAP replication issue would be much appreciated. We have production servers running since 4.x as well, and plan to move from 5.x to 6.x soon.
    No news from Zimbra so far today, but, yes, of course, I will post updates and the solutions here when we have them, no problem.

    All the best,
    Mark

  5. #5
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default Status Update

    I thought it would be helpful to post a status update on things that have changed today...

    First, we determined we had a failed zimlet upgrade issue during the install, so from the cli we undeployed and (re)deployed several zimlets to match the zimlets installed in our test box, a ZCS 6.0.4 system installed fresh at 6.0.2. We are having a Jetty problem with the Social zimlet, but everything else end-user facing is OK. (Only one of our hosted clients is clamoring for the Social zimlet, and several do not want it).

    Second, we are missing Server Statistics and Certificates in the left-vertical nav bar of the Admin Console, and there is no License tab in Global Settings. On initial login to the Admin Console we are greeted with a full set of red X's, even though Zimbra on all of the servers is running fine. This thread shows something similar: http://www.zimbra.com/forums/install...es-broken.html

    Third, the whole logger/stats thing is, we think, a combination of several things (at least for us). See http://www.zimbra.com/forums/install...r-working.html for example.

    After reading that thread carefully, I think our case is related, but there are at least two things going on here: 1) we use syslog-ng not syslogd, and 2) I don't think the updated syslogging portion of the installer completed successfully, though there is nothing in the logs we can see to confirm that.

    Syslogging is not fully functional on our system just yet in that we can't get syslog-ng (remember, we are a SuSE shop and SuSE installs syslog-ng by default) to start with the Zimbra-supplied new 6.0 settings. So for the moment we are using the 5.0 settings in /etc/syslog-ng/syslog-ng.conf:

    Code:
    #ZCS 5.0 Syslogging code
    filter f_local0       { facility(local0); }; # zimbra
    destination zmail { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra 
    log { source(src); filter(f_mail); destination(zmail); }; # zimbra
    destination local0 { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
    log { source(src); filter(f_local0); destination(local0); }; # zimbra
    filter f_auth       { facility(auth); }; # zimbra
    destination zmauth { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
    log { source(src); filter(f_auth); destination(zmauth); }; # zimbra
    
    #ZCS 6.0 Syslogging code
    #Syslog won't start with this Zimbra-supplied line below; syslog-ng complains of a syntax error.:
    #source zimbra_src { unix-stream("/dev/log"; keep-alive(yes); max-connections(20););}; # zimbra
    #Syslog will start with this modified line below
    #source zimbra_src { unix-stream("/dev/log"); }; # zimbra
    #filter zimbra_local0 { facility(local0); }; # zimbra
    #filter zimbra_local1 { facility(local1); }; # zimbra
    #filter zimbra_auth { facility(auth); }; # zimbra
    #filter zimbra_mail { facility(mail); }; # zimbra
    #destination zimbra_mail { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra
    #destination zimbra_local1 { file("/var/log/zimbra-stats.log" owner("zimbra")); }; # zimbra
    #destination zimbra_local0 { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra
    #destination zimbra_auth { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra
    #log { source(zimbra_src); filter(zimbra_mail); destination(zimbra_mail); }; # zimbra
    #log { source(zimbra_src); filter(zimbra_local0); destination(zimbra_local0); }; # zimbra
    #log { source(zimbra_src); filter(zimbra_local1); destination(zimbra_local1); }; # zimbra
    #log { source(zimbra_src); filter(zimbra_auth); destination(zimbra_auth); }; # zimbra
    Even when syslog-ng was started successfully with the amended ZCS 6.0 code, /var/log/zimbra-stats.log failed to populate.

    Further, we have no /opt/zimbra/logger/db/data directory; I am going to try to create the directory and initialize the sqllite db as per this thread and see what happens: http://www.zimbra.com/forums/install...r-working.html

    More to come!

    Any tips/suggestions appreciated!

    With best regards to all,
    Mark

  6. #6
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    News from the front:

    The missing bits in the Admin Console are "fixed". Essentially, the upgrade installer (at least in our case) does not apply the correct ZCS 6.0 Global Administrator rights to such accounts existing before the upgrade.

    We created a new ZCS account, clicked the "Global Administrator" tick box, and now this user account when logged in to the Admin Console sees everything that should be seen by a Global Admin.

    So, now we are down to:

    1) broken stats, which we know in our case is related to syslog-ng configuration issues and possibly also to the sqlite database failed initialization during the upgrade, and

    2) broken LDAP replication, which is in Zimbra's court at the moment.

    3) broken Social zimlet throwing multiple Jetty 403 errors on service/proxy "forbidden" when navigating to the Social tab.

    Happy New Year to all,
    Mark

  7. #7
    Join Date
    Oct 2005
    Location
    USA, Canada and India
    Posts
    777
    Rep Power
    11

    Default

    Mark do you think these problems were specific to ZCS + SLES10 and/or to your setup only ?


    Thanks
    Raj
    i2k2 Networks
    Dedicated & Shared Zimbra Hosting Provider

  8. #8
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    Quote Originally Posted by raj View Post
    Mark do you think these problems were specific to ZCS + SLES10 and/or to your setup only ?


    Thanks
    Raj
    Hi Raj!

    Our stats issue is partly specific to any distro using syslog-ng instead of syslogd IMHO. Since I don't know why the sqlite data directory didn't get created during the upgrade, I can't say if that part is a Zimbra-SLES problem or not.

    The admin console issues I think are due to our hosting farm having been up since 4.0.3; so much has changed in the intervening versions that these kinds of upgrade issues with user accounts' rights do not surprise me.

    The LDAP replication issue has me stumped (and Zimbra too at this point).

    The one area in Zimbra where we have done nothing creative is LDAP. I really expect we will find something silly as the root cause here of replication no longer working.

    Our hosting business is growing but we are not going to add any new servers to this farm and will instead start a new, totally separate Zimbra farm with a fresh install of ZCS 6.0.4/5.

    Hope that helps,
    Mark

  9. #9
    Join Date
    Mar 2006
    Location
    Beaucaire, France
    Posts
    2,322
    Rep Power
    13

    Default

    Quote Originally Posted by LMStone View Post
    First, the upgrade installer destroyed the GoDaddy commercial certs, which needed to get reinstalled.
    As I said somewhere else, I had the same issue while upgrading from 5.0.18 to 5.0.21...

  10. #10
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    Quote Originally Posted by vavai View Post
    Hi Mark,

    According to syslog-ng problem, the same problem also hit me with ZCS 6.0.4 with SLES 11 64 bit but what make me confused, I'm trying ZCS 6.0.3 clean install with SLES 11 64 bit and it has no problem with syslog-ng. Trying to upgrade into 6.0.4 and similar problem hit me again.

    Sadly, I haven't compare the syslog-ng configuration between two version
    It would be helpful if you wouldn't mind posting the syslog-ng.conf file from your working SLES/ZCS 6.0.3 system please.

    Can do?

    Thanks!
    Mark

Similar Threads

  1. [SOLVED] Important Mta Issue!!!!!!!!
    By borngunners in forum Migration
    Replies: 2
    Last Post: 01-05-2010, 06:44 AM
  2. Replies: 15
    Last Post: 11-24-2009, 08:46 AM
  3. [SOLVED] Zimbra logwatch.
    By nishith in forum Administrators
    Replies: 5
    Last Post: 06-10-2009, 05:42 PM
  4. Zimbra server crashed
    By goetzi in forum Administrators
    Replies: 6
    Last Post: 03-25-2006, 01:00 PM
  5. Mail logs
    By Rick Baker in forum Installation
    Replies: 8
    Last Post: 01-17-2006, 04:33 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •