Results 1 to 4 of 4

Thread: [SOLVED] mysql/innodb vserver hashify conflict produces data loss

  1. #1
    Join Date
    May 2009
    Posts
    134
    Rep Power
    6

    Angry [SOLVED] mysql/innodb vserver hashify conflict produces data loss

    Hello, all. After weeks of troubleshooting, we have discovered a conflict between the zimbra mysql/innodb process and the vserver hashify process. We are running zimbra on CentOS 5.3 on VServer with reasonable success other than this major issue.

    Whenever we run vserver <zimbra server name> hashify, we see odd
    behaviors:

    * The mysql and zimbra database directories are time stamped with
    the time hashify ran as if mysql was restarted but there is no
    record of mysql being restarted.
    * MOST IMPORTANTLY, the innodb log stops recording but it appears
    that mysql is still functioning. Mail is displayed, events
    (e.g., calendar, tasks) are recorded and are fully functional

    The disaster occurs when mysql is restarted, e.g., when restarting the
    zimbra service. Since the innodb log has recorded no data since hashify
    ran, mysql thinks the system has crashed and backs out all data since
    the last hashify! At least, I believe that's what is happening in my
    MySQL ignorance.

    As some background on hashify, the hashify process is not an essential process but it is the single VServer feature which tipped us to choose VServer over OpenVZ and is a life saver in the managed service provider market.

    There may be hundreds of vservers on a host. This density is a result
    of the excellent resource utilization of VServer. Since many if not most of the systems may duplicate file, e.g., the majority of the distribution binaries, hashify is used to eliminate the redundancy both on disk and in RAM.

    The vserver files are hashed. Any files with the same hash are replaced
    with immutable but unlinkable hard links except for one base copy. As a result, one hundred instances of the file will take little more space than a single instance on both disk and in RAM. Explanations can be found here util-vserver:Vhashify - Linux-VServer and here Frequently Asked Questions - Linux-VServer

    Any ideas on why this scanning/linking process would cause this behavior in innodb? Any suggestions on where to start troubleshooting? Thanks - John
    www.spiritualoutreach.com
    Making Christianity intelligible to secular society

  2. #2
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    As you were told a month ago, you can get a Linux environment suitable for running Zimbra from VMWare, Xen, or raw hardware. If you need to share hardware, you ought to be able to run your other legacy vservers, unmodified, on top of VMWare or Xen... though if your resources are that constrained, you should also consider outsourcing.

    How much would you value a month of your time and 30 days' worth of crashes and lost mail?

  3. #3
    Join Date
    May 2009
    Posts
    134
    Rep Power
    6

    Default

    Sorry, I still vehemently disagree. This has been very costly to us but the world is not just about me. I have gladly wrestled this to the ground and now we have solved it so that others may benefit from our work. I believe that is one of the core definitions of community and open source contribution - my and my company's way of contributing rather than just leaching. Yes, this one cost us a lot more than we would have liked and we sure wish someone else had paid the tuition before us but our contribution pales in light of what it has cost others to give us the wealth of open source products that power our company.

    Along the way, we have enriched our engineering knowledge base so we can move beyond taking the defaults and being glad it works. We now know much more about how Zimbra works. We have resolved many other issues along the way and can now bend the product to our environment so that our environment can better serve our clients rather than having to bend our environment to the product. We did not find it particularly helpful to be told to disable most of our security, violate best practice security models, change our entire infrastructure and all of our other IT products just so that this one product would work. Although it is understandable from the point of view of containing the cost of support, it is a product centric approach rather than a customer centric one. We have now preserved our security model, our infrastructure, our other product choices, can use Zimbra in that environment and will share that knowledge with others.

    We will post the resolution shortly once we have worked out the final details.
    www.spiritualoutreach.com
    Making Christianity intelligible to secular society

  4. #4
    Join Date
    May 2009
    Posts
    134
    Rep Power
    6

    Default

    Since we are not running multiple zimbra instances on the same vserver host, it made sense to simply exclude the entire /opt/zimbra directory from hashify. We could have excluded /opt/zimbra/db/data (and I suppose /opt/zimbra/logger/db/data) only but this seemed to kill two birds with one stone (eliminate useless hashification and solve our problem).

    To do this, we did:
    Code:
    mkdir /etc/vservers/zimbra/apps/vunify
    cp /usr/lib64/util-vserver/defaults/vunify-exclude /etc/vservers/zimbra/apps/vunify/exclude
    echo /opt/zimbra >> /etc/vservers/zimbra/apps/vunify/exclude
    vserver zimbra hashify
    This assumes a 64 bit system and a vserver named zimbra.

    There were several other instances we encountered to enable zimbra to run with all features enabled on a vserver although this was certainly the most serious. We also encountered some small issues with CentOS 5.3 We will pull all those together into a how-to for both the Zimbra and VServer communities.

    We have elected to not find why hashify and mysql/innodb do not get along. This does not normally manifest itself because MySQL defaults its data directory to /var and /var is excluded from hashify by default. Zimbra's use of /opt/zimbra to store the databases unearthed the problem.

    Hope this helps someone else! - John
    www.spiritualoutreach.com
    Making Christianity intelligible to secular society

Similar Threads

  1. Zimbra Desktop Stopped Syncing with Zimbra Online
    By dasofsky in forum Error Reports
    Replies: 0
    Last Post: 01-19-2009, 03:57 PM
  2. Replies: 2
    Last Post: 02-12-2008, 11:55 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •