Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: [SOLVED] mysql innodb corrupt, innodb_force_recovery 6 won't work

  1. #1
    Join Date
    Dec 2009
    Posts
    14
    Rep Power
    5

    Default [SOLVED] mysql innodb corrupt, innodb_force_recovery 6 won't work

    Hi,

    this morning our ZCS server discovered not work, after few reviewing I discover the problem is mysql unable to start. with some test I and google I realize is a db corrupt but not so much info I can find on how to recover in this unlucky situation, what I've been already tried:

    1. I stop the whole zcs, only su zimbra and test mysql.server start, it won't work
    2. I read the log/mysql_error.log realize the problem is some inconsistency in log and innodb start to recovery but failed at error: "Page directory corruption: supremum not pointed to"
    3. I read the zimbra mysql recovery guide, but even I set innodb_force_recovery = 6 I do not have any luck to start mysql.server start with user zimbra

    my ZCS is version zcs-6.0.0_GA_1802.RHEL5.20090830140212.
    in CentOS in a VM, was running perfect.

    so what I have to do next?
    Last edited by wolvesled; 03-22-2010 at 04:16 AM.

  2. #2
    Join Date
    Dec 2009
    Posts
    14
    Rep Power
    5

    Default the error log of mysql

    Code:
    100322 16:57:52  mysqld started
    100322 16:57:52 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
    100322 16:57:52 [Warning] option 'max_join_size': unsigned value 18446744073709551615 adjusted to 4294967295
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    100322 16:57:53  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Page directory corruption: supremum not pointed to
    100322 16:57:53  InnoDB: Page dump in ascii and hex (16384 bytes):
     len 16384; hex <0000 many of "0"s>; asc
    
    
    
    
    
    
    
    
                                                  ;InnoDB: End of page dump
    100322 16:57:53  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
    InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
    InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
    InnoDB: Page number (if stored to page already) 0,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
    InnoDB: Page directory corruption: supremum not pointed to
    100322 16:57:53  InnoDB: Page dump in ascii and hex (16384 bytes):
     len 16384; hex <0000 many of "0"s>; asc
    
    
    
    
    
    
    
    
                                                  ;InnoDB: End of page dump
    100322 16:57:54  InnoDB: Page checksum 1575996416, prior-to-4.0.14-form checksum 1371122432
    InnoDB: stored checksum 0, prior-to-4.0.14-form stored checksum 0
    InnoDB: Page lsn 0 0, low 4 bytes of lsn at page end 0
    InnoDB: Page number (if stored to page already) 0,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
    100322 16:57:54InnoDB: Error: trying to access a stray pointer 0xd5a23ff8
    InnoDB: buf pool start is at 0x55a08000, end at 0xb6908000
    InnoDB: Probable reason is database corruption or memory
    InnoDB: corruption. If this happens in an InnoDB database recovery, see
    InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
    InnoDB: how to force recovery.
    100322 16:57:54InnoDB: Assertion failure in thread 3083704000 in file ./../include/buf0buf.ic line 268
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery.html
    InnoDB: about forcing recovery.
    100322 16:57:54 - mysqld got signal 11 ;
    This could be because you hit a bug. It is also possible that this binary
    or one of the libraries it was linked against is corrupt, improperly built,
    or misconfigured. This error can also be caused by malfunctioning hardware.
    We will try our best to scrape up some info that will hopefully help diagnose
    the problem, but since we have already crashed, something is definitely wrong
    and this may fail.
    
    key_buffer_size=0
    read_buffer_size=1048576
    max_used_connections=0
    max_connections=110
    threads_connected=0
    It is possible that mysqld could use up to
    key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 225280 K
    bytes of memory
    Hope that's ok; if not, decrease some variables in the equation.
    
    thd=(nil)
    Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    Cannot determine thread, fp=0xbfc667b8, backtrace may not be correct.
    Stack range sanity check OK, backtrace follows:
    0x8192065
    0x83c91fb
    0x8342596
    0x834db2b
    0x82f40fd
    0x82f58e4
    0x82e50fe
    0x82e3f0e
    0x8270c86
    0x8265662
    0x8191298
    0x8196645
    0xb7cf0e8c
    0x80edf51
    New value of fp=(nil) failed sanity check, terminating stack trace!
    Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
    stack trace is much more helpful in diagnosing the problem, so please do
    resolve it
    The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
    information that should help you find out what is causing the crash.
    100322 16:57:54  mysqld ended

  3. #3
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,017
    Rep Power
    25

    Default

    Do you have a backup ?

  4. #4
    Join Date
    Dec 2009
    Posts
    14
    Rep Power
    5

    Default

    not luckly, was planning to have a backup server, but happens before we get there.

  5. #5
    Join Date
    Dec 2009
    Posts
    14
    Rep Power
    5

    Default

    still looking for chance of non-or-minium lost, let me know if impossible.

    P.S. was was running fine and no power failure, can be some software issues?
    P.S.2 tested in the case if remove the ibdata and iblogs, that way mysql start, so means the library has no problem.

  6. #6
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,374
    Rep Power
    11

    Default

    I would open a support ticket with Zimbra on this.

    Hope that helps,
    Mark

  7. #7
    Join Date
    Dec 2009
    Posts
    14
    Rep Power
    5

    Default thanks!

    Thank you all very much in advance!

    I was just wondering the ZCS is storing the emails in database or independently in filesystem organized by postfix? maybe any chance to start imap and pull out all emails at least.

    I'm taking searches.

  8. #8
    Join Date
    Sep 2007
    Posts
    186
    Rep Power
    8

    Default

    If you are running NE, this may be of interest to you.

    King0770-Notes-When innodb force recovery Fails - Zimbra :: Wiki

  9. #9
    Join Date
    Dec 2009
    Posts
    14
    Rep Power
    5

    Default not luck

    Quote Originally Posted by king0770 View Post
    If you are running NE, this may be of interest to you.

    King0770-Notes-When innodb force recovery Fails - Zimbra :: Wiki
    Thanks for the info, but I look into my redolog folder there is only redo.log of 60MB, and the archive folder is empty, seems there isn't any incremental logs.

    I tried to find the binlog of mysql, but not luck, anyone every seen it?

    I'm looking this article about recovery of the innodb data: How to Recover Data using the InnoDB Recovery Tool Chris on MySQL
    the last option is to restore the meta data and use the "store" folder together to reinstall zimbra.

    hope not too stupid going this way...

  10. #10
    Join Date
    Oct 2005
    Location
    USA, Canada and India
    Posts
    777
    Rep Power
    11

    Default

    did you try innodb_force_recovery = 1 to 6 all six options in an increasing order of 1,2...6 or just tried 6 directly.
    if you have not tried all 6 then i suggest you do that in increasing order and see if you can start mysqld service

    Raj
    i2k2 Networks
    Dedicated & Shared Zimbra Hosting Provider

Similar Threads

  1. Corrupt database
    By menno.pieters in forum Administrators
    Replies: 2
    Last Post: 02-02-2009, 06:01 AM
  2. [SOLVED] MySQL not starting
    By Deeeep in forum Administrators
    Replies: 2
    Last Post: 11-30-2008, 01:23 PM
  3. Zimbra fails after working for 2 weeks
    By Linsys in forum Administrators
    Replies: 10
    Last Post: 10-07-2008, 01:42 AM
  4. Replies: 7
    Last Post: 02-27-2008, 05:17 PM
  5. Upgrade 4.5.1 -> 4.5.2 Network Edition on RH Failed
    By CJ.deb in forum Installation
    Replies: 7
    Last Post: 03-01-2007, 07:05 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •