Results 1 to 2 of 2

Thread: System crash leads to major MySQL corruption

Hybrid View

  1. #1
    Join Date
    Aug 2011
    Posts
    1
    Rep Power
    4

    Default System crash leads to major MySQL corruption

    I had a system crash this morning (due to overheating due to a failed fan). Unfortunately several key files in /opt/zimbra were corrupted/truncated by the ensuing mess. The biggest problem is MySQL won't start.

    Here is the version: Release 7.1.1_GA_3196.RHEL5_64_20110527011124 CentOS5_64 FOSS edition.


    I've tried the various steps for using innodb_force_recovery without success...it just won't start due to the corruptions.

    No, I don't have a good mysqldump, unfortunately.

    Are there any hopes of recovering? I've seen that the NE has zmrestore, etc...anybody create something like this for the FOSS?

    Or am I just 100% lost. The sad part is the drives I intend to use for backups are in the mail as we speak...one week too late.

    John

  2. #2
    Join Date
    Aug 2007
    Location
    Buena Park, CA
    Posts
    5
    Rep Power
    8

    Default

    Here is the output in mysql_error.log. I removed the hex dumps for clarity. If someone needs the full info just PM me and I'll email it along.:

    110801 04:05:58 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
    InnoDB: The InnoDB memory heap is disabled
    InnoDB: Mutexes and rw_locks use GCC atomic builtins
    InnoDB: Compressed tables use zlib 1.2.3
    110801 4:05:58 InnoDB: Initializing buffer pool, size = 603.0M
    110801 4:05:58 InnoDB: Completed initialization of buffer pool
    110801 4:05:58 InnoDB: highest supported file format is Barracuda.
    InnoDB: The log sequence number in ibdata files does not match
    InnoDB: the log sequence number in the ib_logfiles!
    110801 4:05:58 InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Error: space id in fsp header 1224737009, but in the page header 25639
    InnoDB: Error: tablespace id 18446744073709551615 in file ./mboxgroup9/open_conversation.ibd is not sensible
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Warning: a page in the doublewrite buffer is not within space
    InnoDB: bounds; space id 0 page number 1936, page 2 in doublewrite buf.
    InnoDB: Warning: a page in the doublewrite buffer is not within space
    InnoDB: bounds; space id 0 page number 2058, page 15 in doublewrite buf.
    InnoDB: Warning: a page in the doublewrite buffer is not within space
    InnoDB: bounds; space id 204 page number 2863, page 16 in doublewrite buf.
    InnoDB: Warning: a page in the doublewrite buffer is not within space
    InnoDB: bounds; space id 266 page number 368, page 31 in doublewrite buf.
    InnoDB: Warning: a page in the doublewrite buffer is not within space
    InnoDB: bounds; space id 266 page number 370, page 32 in doublewrite buf.
    InnoDB: Warning: a page in the doublewrite buffer is not within space
    InnoDB: bounds; space id 223 page number 358, page 33 in doublewrite buf.
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 1.
    InnoDB: You may have to recover from a backup.
    110801 4:05:58 InnoDB: Page dump in ascii and hex (16384 bytes):
    len 16384; hex 12231a <EDITED FOR LENGTH>
    InnoDB: End of page dump
    110801 4:05:58 InnoDB: Page checksum 128194496, prior-to-4.0.14-form checksum 893462745
    InnoDB: stored checksum 304290357, prior-to-4.0.14-form stored checksum 893462745
    InnoDB: Page lsn 0 2766573610, low 4 bytes of lsn at page end 2766573610
    InnoDB: Page number (if stored to page already) 1,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
    InnoDB: Page may be a freshly allocated page
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 1.
    InnoDB: You may have to recover from a backup.
    InnoDB: It is also possible that your operating
    InnoDB: system has corrupted its own file cache
    InnoDB: and rebooting your computer removes the
    InnoDB: error.
    InnoDB: If the corrupt page is an index page
    InnoDB: you can also try to fix the corruption
    InnoDB: by dumping, dropping, and reimporting
    InnoDB: the corrupt table. You can use CHECK
    InnoDB: TABLE to scan your table for corruption.
    InnoDB: See also MySQL :: MySQL 5.1 Reference Manual :: 13.6.6.2 Forcing InnoDB Recovery
    InnoDB: about forcing recovery.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/appointment'
    InnoDB: in InnoDB data dictionary has tablespace id 270,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/appointment_dumpster'
    InnoDB: in InnoDB data dictionary has tablespace id 171,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/data_source_item'
    InnoDB: in InnoDB data dictionary has tablespace id 221,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/imap_folder'
    InnoDB: in InnoDB data dictionary has tablespace id 275,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/imap_message'
    InnoDB: in InnoDB data dictionary has tablespace id 281,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/mail_item'
    InnoDB: in InnoDB data dictionary has tablespace id 234,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/mail_item_dumpster'
    InnoDB: in InnoDB data dictionary has tablespace id 169,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/open_conversation'
    InnoDB: in InnoDB data dictionary has tablespace id 283,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/pop3_message'
    InnoDB: in InnoDB data dictionary has tablespace id 190,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/revision'
    InnoDB: in InnoDB data dictionary has tablespace id 198,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/revision_dumpster'
    InnoDB: in InnoDB data dictionary has tablespace id 170,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup5/tombstone'
    InnoDB: in InnoDB data dictionary has tablespace id 212,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    110801 4:05:58 InnoDB: Error: table 'mboxgroup9/open_conversation'
    InnoDB: in InnoDB data dictionary has tablespace id 241,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.13.4 Troubleshooting InnoDB Data Dictionary Operations
    InnoDB: for how to resolve the issue.
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 47.
    InnoDB: You may have to recover from a backup.
    110801 4:05:58 InnoDB: Page dump in ascii and hex (16384 bytes):
    len 16384; hex 7fdd8caf0000002 <EDITED FOR LENGTH>
    InnoDB: End of page dump
    110801 4:05:58 InnoDB: Page checksum 1456109135, prior-to-4.0.14-form checksum 1801731356
    InnoDB: stored checksum 2145225903, prior-to-4.0.14-form stored checksum 160692190
    InnoDB: Page lsn 0 2387046152, low 4 bytes of lsn at page end 2387046152
    InnoDB: Page number (if stored to page already) 47,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 0
    InnoDB: Page may be an update undo log page
    InnoDB: Page may be an index page where index id is 0 12
    InnoDB: (index "FOR_IND" of table "SYS_FOREIGN")
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 47.
    InnoDB: You may have to recover from a backup.
    InnoDB: It is also possible that your operating
    InnoDB: system has corrupted its own file cache
    InnoDB: and rebooting your computer removes the
    InnoDB: error.
    InnoDB: If the corrupt page is an index page
    InnoDB: you can also try to fix the corruption
    InnoDB: by dumping, dropping, and reimporting
    InnoDB: the corrupt table. You can use CHECK
    InnoDB: TABLE to scan your table for corruption.
    InnoDB: See also MySQL :: MySQL 5.1 Reference Manual :: 13.6.6.2 Forcing InnoDB Recovery
    InnoDB: about forcing recovery.
    InnoDB: Error: trying to access page number 3228459753 in space 0,
    InnoDB: space name ./ibdata1,
    InnoDB: which is outside the tablespace bounds.
    InnoDB: Byte offset 0, len 16384, i/o type 10.
    InnoDB: If you get this error at mysqld startup, please check that
    InnoDB: your my.cnf matches the ibdata files that you have in the
    InnoDB: MySQL server.
    110801 4:05:58 InnoDB: Assertion failure in thread 47476784580688 in file fil/fil0fil.c line 4352
    InnoDB: We intentionally generate a memory trap.
    InnoDB: Submit a detailed bug report to MySQL Bugs.
    InnoDB: If you get repeated assertion failures or crashes, even
    InnoDB: immediately after the mysqld startup, there may be
    InnoDB: corruption in the InnoDB tablespace. Please refer to
    InnoDB: MySQL :: MySQL 5.1 Reference Manual :: 13.6.6.2 Forcing InnoDB Recovery
    InnoDB: about forcing recovery.
    110801 4:05:58 - mysqld got signal 6 ;
    This could be because you hit a bug. It is also possible that this binary
    or one of the libraries it was linked against is corrupt, improperly built,
    or misconfigured. This error can also be caused by malfunctioning hardware.
    We will try our best to scrape up some info that will hopefully help diagnose
    the problem, but since we have already crashed, something is definitely wrong
    and this may fail.

    key_buffer_size=8384512
    read_buffer_size=1048576
    max_used_connections=0
    max_threads=110
    threads_connected=0
    It is possible that mysqld could use up to
    key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 234605 K
    bytes of memory
    Hope that's ok; if not, decrease some variables in the equation.

    Thread pointer: 0x0
    Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    stack_bottom = (nil) thread_stack 0x40000
    /opt/zimbra/mysql/libexec/mysqld(my_print_stacktrace+0x2e)[0x77384e]
    /opt/zimbra/mysql/libexec/mysqld(handle_segfault+0x351)[0x594f71]
    /lib64/libpthread.so.0[0x3f2ac0eb10]
    /lib64/libc.so.6(gsignal+0x35)[0x3f2a030265]
    /lib64/libc.so.6(abort+0x110)[0x3f2a031d10]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd3343e]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd187f1]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd18a6c]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd106a3]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabcfdb8f]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd04a19]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd2b1a5]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd28aad]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd1c5cf]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabdc1f1a]
    /opt/zimbra/mysql-standard-5.1.55-pc-linux-gnu-i686-glibc23/lib/mysql/plugin/ha_innodb_plugin.so[0x2aaaabd4a810]
    /opt/zimbra/mysql/libexec/mysqld(_Z24ha_initialize_handlertonP13st_plugin_in t+0x31)[0x68b0e1]
    /opt/zimbra/mysql/libexec/mysqld[0x70707d]
    /opt/zimbra/mysql/libexec/mysqld(_Z11plugin_initPiPPci+0x81b)[0x70a6db]
    /opt/zimbra/mysql/libexec/mysqld[0x595b6f]
    /opt/zimbra/mysql/libexec/mysqld(main+0x6b5)[0x59a185]
    /lib64/libc.so.6(__libc_start_main+0xf4)[0x3f2a01d994]
    /opt/zimbra/mysql/libexec/mysqld[0x4cfb49]
    The manual page at MySQL :: MySQL 5.5 Reference Manual :: C.5.4.2 What to Do If MySQL Keeps Crashing contains
    information that should help you find out what is causing the crash.
    110801 04:05:58 mysqld_safe mysqld from pid file /opt/zimbra/db/mysql.pid ended

Similar Threads

  1. [SOLVED] Mailserver down when send file attach of 50Mb
    By ZMilton in forum Administrators
    Replies: 20
    Last Post: 04-10-2008, 12:44 PM
  2. Replies: 6
    Last Post: 11-23-2005, 08:26 AM
  3. system failure: getDirectContext
    By avisser in forum Installation
    Replies: 3
    Last Post: 10-12-2005, 06:32 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •