Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: [SOLVED] Zimbra SQL is Dead - Cannot Start ./mysql.server

  1. #1
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Default [SOLVED] Zimbra SQL is Dead - Cannot Start ./mysql.server

    I am having a major problem with Zimbra...there isn't a mysql.sock file in the ~/db directory (as user zimbra). Therefore, I cannot access mysql at all.

    This all started from someone pulling the power plug to the server from the wall. Once the server rebooted, it had a corrupted reiserfs. I was able to recover that by running the repair tree option. But, now I am stuck with a non-functioning Zimbra server.

    I have to leave right now, but I'll provide more details later and if asked.
    cyberdeath

  2. #2
    Join Date
    Jul 2007
    Location
    Baltimore
    Posts
    1,649
    Rep Power
    11

    Default

    mysql.sock is a socket created when the daemon starts. when mysql isn't running, there is no mysql.sock file. so your problem isn't that theres no mysql.sock, it's probably that mysql isn't running. for that you should maybe check your log files and find out why that is.

  3. #3
    Join Date
    Jan 2008
    Posts
    658
    Rep Power
    8

    Default

    How exactly are you trying to start mysql?

    This may or may not help you, but worth a quick look:
    http://www.zimbra.com/forums/adminis...er-outage.html

  4. #4
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Default

    I am trying the /opt/zimbra/bin/mysql.server start command.

    Code:
    mail:/opt/zimbra/bin # ./mysql.server start
    mail:/opt/zimbra/bin # ./mysql.server status
    mysql is not running
    Of course, when I try the ./mysql command to access the cli, it gives me the socket error:

    Code:
    ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/zimbra/db/mysql.sock' (2)
    Is there something with Zimbra that I need to use to fix the sql db error? Or should I be able to use standard MySQL tools? If so, what would you recommend?

    Here is the mail.err file that was mentioned by y@w in the link.

    Code:
    080716 21:21:19  mysqld started
    InnoDB: Log scan progressed past the checkpoint lsn 0 1783981612
    080716 21:21:20  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Doing recovery: scanned up to log sequence number 0 1784059982
    InnoDB: 2 transaction(s) which must be rolled back or cleaned up
    InnoDB: in total 3 row operations to undo
    InnoDB: Trx id counter is 0 2706432
    080716 21:21:39  InnoDB: Error: table 'mboxgroup67/appointment'
    InnoDB: in InnoDB data dictionary has tablespace id 921,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
    InnoDB: for how to resolve the issue.
    080716 21:21:39  InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 080716 21:21:40 - mysqld got signal 11;
    This could be because you hit a bug. It is also possible that this binary
    or one of the libraries it was linked against is corrupt, improperly built,
    or misconfigured. This error can also be caused by malfunctioning hardware.
    We will try our best to scrape up some info that will hopefully help diagnose
    the problem, but since we have already crashed, something is definitely wrong
    and this may fail.
    key_buffer_size=0
    read_buffer_size=1044480
    max_used_connections=0
    max_connections=110
    threads_connected=0
    It is possible that mysqld could use up to
    key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 224839 K
    bytes of memory
    Hope that's ok; if not, decrease some variables in the equation.
    thd=(nil)
    Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    Cannot determine thread, fp=0x526a1458, backtrace may not be correct.
    Stack range sanity check OK, backtrace follows:
    0x818bf60
    0x83b6e59
    0x83b7a47
    0x837a1e6
    0x837c2c3
    0x8368849
    0x839271e
    0x82d5968
    0xb7e252ab
    0xb7d5058e
    New value of fp=(nil) failed sanity check, terminating stack trace!
    Please read http://dev.mysql.com/doc/mysql/en/using-stack-trace.html and follow instructions on how to resolve the stack trace. Resolved
    stack trace is much more helpful in diagnosing the problem, so please do
    resolve it
    The manual page at http://www.mysql.com/doc/en/Crashing.html contains
    information that should help you find out what is causing the crash.
    080716 21:21:40  mysqld ended
    I apologize for the earlier short post as I had to step out unexpectedly (but I still wanted to get something into the forums).

    If you need any more information, please let me know.

    Thanks!
    cyberdeath

  5. #5
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    21

    Default

    We use innodb, which makes repair very difficult. This is the best route for recovery:
    Mysql Crash Recovery - Zimbra :: Wiki

  6. #6
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Default

    Well, I got to step 4 of the guide you mentioned, jholder....and then it went downhill:
    Code:
    mysqldump: Got error: 1033: Incorrect information in file: './zimbra/config.frm' when using LOCK TABLES
    Suggestions?
    cyberdeath

  7. #7
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Default

    Okay so I fiddled around with it a bit and was able to get this error message when trying #4 on the page jholder provided:

    Code:
    080717  1:32:38  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Doing recovery: scanned up to log sequence number 0 1784059982
    InnoDB: 2 transaction(s) which must be rolled back or cleaned up
    InnoDB: in total 3 row operations to undo
    InnoDB: Trx id counter is 0 2706432
    080717  1:32:38  InnoDB: Error: table 'mboxgroup67/appointment'
    InnoDB: in InnoDB data dictionary has tablespace id 921,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
    InnoDB: for how to resolve the issue.
    080717  1:32:38  InnoDB: Error: table 'mboxgroup67/imap_folder'
    InnoDB: in InnoDB data dictionary has tablespace id 927,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
    InnoDB: for how to resolve the issue.
    080717  1:32:38  InnoDB: Error: table 'mboxgroup67/imap_message'
    InnoDB: in InnoDB data dictionary has tablespace id 929,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
    InnoDB: for how to resolve the issue.
    080717  1:32:38  InnoDB: Error: table 'mboxgroup67/pop3_message'
    InnoDB: in InnoDB data dictionary has tablespace id 924,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
    InnoDB: for how to resolve the issue.
    080717  1:32:38  InnoDB: Error: table 'mboxgroup67/tombstone'
    InnoDB: in InnoDB data dictionary has tablespace id 922,
    InnoDB: but tablespace with that id or name does not exist. Have
    InnoDB: you deleted or moved .ibd files?
    InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
    InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
    InnoDB: table still exists in the InnoDB internal data dictionary.
    InnoDB: Please refer to
    InnoDB: http://dev.mysql.com/doc/refman/5.0/en/innodb-troubleshooting.html
    InnoDB: for how to resolve the issue.
    080717  1:32:38  InnoDB: Starting an apply batch of log records to the database.
    After that, it just dumps the data.

    Code:
    InnoDB: End of page dump
    080717  1:32:39  InnoDB: Page checksum 809267069, prior-to-4.0.14-form checksum                                                                              1330400654
    InnoDB: stored checksum 1895350281, prior-to-4.0.14-form stored checksum 1488708                                                                             828
    InnoDB: Page lsn 0 1783877752, low 4 bytes of lsn at page end 1783925701
    InnoDB: Page number (if stored to page already) 61,
    InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 504
    InnoDB: Page may be an index page where index id is 0 1365
    InnoDB: Database page corruption on disk or a failed
    InnoDB: file read of page 61.
    InnoDB: You may have to recover from a backup.
    InnoDB: It is also possible that your operating
    InnoDB: system has corrupted its own file cache
    InnoDB: and rebooting your computer removes the
    InnoDB: error.
    InnoDB: If the corrupt page is an index page
    InnoDB: you can also try to fix the corruption
    InnoDB: by dumping, dropping, and reimporting
    InnoDB: the corrupt table. You can use CHECK
    InnoDB: TABLE to scan your table for corruption.
    InnoDB: See also InnoDB: http://dev.mysql.com/doc/refman/5.0/en/forcing-recovery                                                                             .html
    InnoDB: about forcing recovery.
    080717  1:32:39 - mysqld got signal 11;
    This could be because you hit a bug. It is also possible that this binary
    or one of the libraries it was linked against is corrupt, improperly built,
    or misconfigured. This error can also be caused by malfunctioning hardware.
    We will try our best to scrape up some info that will hopefully help diagnose
    the problem, but since we have already crashed, something is definitely wrong
    and this may fail.
    Any suggestions for this one?

    Thanks for the help thus far =).
    cyberdeath

  8. #8
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    21

    Default

    I'm gonna be honest with you. This looks bad, and without a backup, you're probably hosed.

    Did you start mysql in recovery mode? Also, can you cat that frm file? Do you have a backup of that file?

  9. #9
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Default

    I had to set innodb_force_recovery to 6 to get it to boot again. Once in, I noticed that it errors at mboxgroup67 and specifically with the open_conversation table.

    Code:
    mysql> check table open_conversation;
    ERROR 2013 (HY000): Lost connection to MySQL server during query
    And when I analyze table:

    Code:
    mysql> analyze table open_conversation;
    ERROR 2006 (HY000): MySQL server has gone away
    No connection. Trying to reconnect...
    Connection id:    2
    Current database: mboxgroup67
    
    +-------------------------------+---------+----------+----------+
    | Table                         | Op      | Msg_type | Msg_text |
    +-------------------------------+---------+----------+----------+
    | mboxgroup67.open_conversation | analyze | status   | OK       |
    +-------------------------------+---------+----------+----------+
    1 row in set (0.01 sec)
    As far as backups are concerned...we don't have any backups of the server...yes, horrible practice...but it wasn't my decision.

    Further suggestions would be greatly appreciated...I *really* need to be able to recover this server...since it holds a lot of important information.

    Thanks again

    *EDIT*

    I tried to dump the information yet again....

    Code:
    for db in `< /tmp/mysql.db.list`;
    do
        /opt/zimbra/mysql/bin/mysqldump $db -S /opt/zimbra/db/mysql.sock -u root --password=pwhere > /tmp/mysql.sql/$db.sql
        echo -e "Dumped $db\n"
    done
    ...and it worked!

    So, here's where I run into another problem.....Step 7:

    Code:
    mail:/opt/zimbra/bin # ./mysql -e "drop database zimbra"
    ERROR 1217 (23000) at line 1: Cannot delete or update a parent row: a foreign key constraint fails
    Last edited by cyberdeath; 07-16-2008 at 11:01 PM.
    cyberdeath

  10. #10
    Join Date
    Jan 2008
    Location
    127.0.0.1, Virginia, USA
    Posts
    42
    Rep Power
    7

    Default

    Turns out, I forgot a step...which I quickly found out by using this command while in ./mysql (and looking under foreign key constraints):

    Code:
    SHOW INNODB STATUS
    Once I did the command to drop the mboxes...that solved the problem. Server is running nicely now.

    Thanks to everyone who lent a hand and pointed me in the right direction.

    Hopefully this information will be useful to someone else in the future =).
    cyberdeath

Similar Threads

  1. slapd message error
    By smoke in forum Administrators
    Replies: 7
    Last Post: 04-27-2008, 03:23 PM
  2. Replies: 9
    Last Post: 03-01-2008, 07:21 PM
  3. Replies: 22
    Last Post: 12-02-2007, 04:05 PM
  4. Unable to start tomcat
    By chanck in forum Administrators
    Replies: 11
    Last Post: 06-11-2006, 12:58 AM
  5. 3.1 on FC4 problems
    By cohnhead in forum Installation
    Replies: 8
    Last Post: 05-26-2006, 11:16 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •