Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: raw_logs mysql table corruption - how to recover?

  1. #1
    Join Date
    Mar 2007
    Posts
    33
    Rep Power
    8

    Default raw_logs mysql table corruption - how to recover?

    Hello all,

    Recently, I noticed I was having issues with the Zimbra logger. As a result, I found this wiki page.

    So I issued a REPAIR TABLE statement, and after 5+ hours wait, I decided to investigate, and found I'd run into the exact same issue as outlined in this MySQL bug report. My du/df returns exactly the same results as outlined in that bug (an empty and yet full /tmp/ directory)

    Doing a tail servername.com.err returns several lines similar to the below..
    Code:
    071219 15:56:04 [ERROR] /opt/zimbra/logger/mysql/libexec/mysqld: Disk is full writing '/tmp/STYAnvMQ' (Errcode: 28). Waiting for someone to free space... Retry in 60 secs
    Any thoughts? Right now, I'm EXTREMELY loath to abort the "REPAIR TABLE" process, yet I am pretty sure it's stuck...

    TIA!

  2. #2
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    Are you out of space in /tmp?

  3. #3
    Join Date
    Mar 2007
    Posts
    33
    Rep Power
    8

    Default

    Quote Originally Posted by jholder View Post
    Are you out of space in /tmp?
    No.

    Code:
    # df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/sda1             51625340   2993668  46010232   7% /
    none                   1046016         0   1046016   0% /dev/shm
    /dev/sda3             51614140  25925372  23067328  53% /opt
    /dev/sda4              2015824   2004140         0 100% /tmp
    # du -s /tmp
    52      /tmp

  4. #4
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    Look closer.

    Yes you are.

  5. #5
    Join Date
    Mar 2007
    Posts
    33
    Rep Power
    8

    Default

    Quote Originally Posted by jholder View Post
    Look closer.

    Yes you are.
    So how come du -s (at the bottom of my paste in my previous post) reports only 52K used up?

    In any case, if I AM out of space, how should I progress? Kill "REPAIR TABLE"?

  6. #6
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    Not sure on the -s switch. Man indicates:
    "-s Display an entry for each specified file. (Equivalent to -d 0)"

    /tmp isn't a file, it's a directory. It's possible that i'm wrong on that (actually it's likely), but your df clearly indicates that /tmp is 100% full.

    Can you free up space in /tmp? If you do, it should redo the operation, fixing it.

  7. #7
    Join Date
    Mar 2007
    Posts
    33
    Rep Power
    8

    Default

    Quote Originally Posted by jholder View Post
    Not sure on the -s switch. Man indicates:
    "-s Display an entry for each specified file. (Equivalent to -d 0)"

    /tmp isn't a file, it's a directory. It's possible that i'm wrong on that (actually it's likely), but your df clearly indicates that /tmp is 100% full.

    Can you free up space in /tmp? If you do, it should redo the operation, fixing it.
    du -s = summarise du.

    Let me run it without du -s & you'll see it's not full
    Code:
    # du /tmp/
    4       /tmp/.ICE-unix
    4       /tmp/hsperfdata_zimbra
    20      /tmp/zimbra/afb99765-2950-47e2-9ae7-a95cc575e374-deferred
    24      /tmp/zimbra
    4       /tmp/hsperfdata_root
    52      /tmp/
    Convinced yet?

  8. #8
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

  9. #9
    Join Date
    Oct 2005
    Location
    Thatcher, AZ
    Posts
    5,606
    Rep Power
    20

    Default

    Nope.

    Let's think through this logically:
    1) You try to repair a table and get an error: "Disk is full writing '/tmp/STYAnvMQ' (Errcode: 28). Waiting for someone to free space..."
    2) I ask if you're out of space on /tmp
    3) You say no, but display a df which says that /tmp is 100% full
    4) df also says that there is zero k left on that mount

    Which is more likely:
    You're out of space on /tmp which is indicated by mysql and df?
    or
    You're misinterpreting the results of du -s ?

  10. #10
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    For instance:
    Quote Originally Posted by Centurion View Post
    According to the standard system tools:
    Code:
    [root@mail tmp]# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    ...snipped...
    /dev/md3              3.0G  1.8G  1.1G  63% /tmp
    But looking at the /tmp directory with "du" reveals....
    Code:
    [root@mail tmp]# du -hs /tmp
    820K    /tmp
    Digging deeper with "lsof" we see the following:
    Code:
    [root@mail tmp]# lsof | grep "/tmp"
    bash       1970    root  cwd       DIR        9,3       4096          2 /tmp
    screen    13507   jgray    3r     FIFO        9,3                 15747 /tmp/uscreens/S-jgray/13507.pts-0.mail
    perl      19932  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    perl      19932  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    zmlogger  19937  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    zmlogger  19937  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    zmlogger  19937  zimbra    4w      REG        9,3 1884374605         26 /tmp/zmlogger.out (deleted)
    mysqld_sa 21321  zimbra    1w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
    mysqld_sa 21321  zimbra    2w      REG        9,3         70         15 /tmp/zmcontrol.out.20738 (deleted)
    logswatch 21391  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
    logswatch 21391  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
    mysqld    21402  zimbra    5u      REG        9,3          0         20 /tmp/ibD4jO0l (deleted)
    mysqld    21402  zimbra    6u      REG        9,3          0         21 /tmp/ibzy1fOB (deleted)
    mysqld    21402  zimbra    7u      REG        9,3          0         22 /tmp/ibXukIBR (deleted)
    mysqld    21402  zimbra    8u      REG        9,3          0         23 /tmp/ibJj1dq7 (deleted)
    mysqld    21402  zimbra   12u      REG        9,3          0         25 /tmp/iblBE0Vn (deleted)
    perl      21423  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
    perl      21423  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
    zmlogger  21446  zimbra    1w      REG        9,3         82         19 /tmp/logswatch.out
    zmlogger  21446  zimbra    2w      REG        9,3         82         19 /tmp/logswatch.out
    zmlogger  21446  zimbra    4w      REG        9,3     373318         27 /tmp/zmlogger.out
    mysqld_sa 21877  zimbra    1w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
    mysqld_sa 21877  zimbra    2w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
    mysqld    21972  zimbra    5u      REG        9,3       3075         28 /tmp/ibagshl0 (deleted)
    mysqld    21972  zimbra    6u      REG        9,3          0         29 /tmp/ibY1LZ2w (deleted)
    mysqld    21972  zimbra    7u      REG        9,3          0         30 /tmp/ibakKIK3 (deleted)
    mysqld    21972  zimbra    8u      REG        9,3          0         31 /tmp/ib4yAKFB (deleted)
    mysqld    21972  zimbra   12u      REG        9,3          0         32 /tmp/ibk4EXla (deleted)
    java      22007  zimbra  mem       REG        9,3      32768     299138 /tmp/hsperfdata_zimbra/22007
    zmtomcatm 22790    root    1w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
    zmtomcatm 22790    root    2w      REG        9,3     105248         17 /tmp/zmcontrol.out.20738 (deleted)
    java      22791  zimbra  mem       REG        9,3      32768     330626 /tmp/hsperfdata_root/22791
    swatch    23099  zimbra    1w      REG        9,3         79         34 /tmp/swatch.out
    swatch    23099  zimbra    2w      REG        9,3         79         34 /tmp/swatch.out
    perl      23117  zimbra    1w      REG        9,3         79         34 /tmp/swatch.out
    perl      23117  zimbra    2w      REG        9,3         79         34 /tmp/swatch.out
    lsof      27703    root  cwd       DIR        9,3       4096          2 /tmp
    bash      27704    root  cwd       DIR        9,3       4096          2 /tmp
    lsof      27705    root  cwd       DIR        9,3       4096          2 /tmp
    logswatch 28631  zimbra    1w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    logswatch 28631  zimbra    2w      REG        9,3      71523         18 /tmp/logswatch.out (deleted)
    sshd      32234   jgray    7u     unix 0xc67d0380              98791141 /tmp/ssh-JGggs32234/agent.32234
    BTW, don't know if your profile is current (4.5.6_GA_1044.RHEL4_64_20070706142644 CentOS4_64) but 4.5.10 is out
    Last edited by mmorse; 12-19-2007 at 10:36 AM.

Similar Threads

  1. Zimbra fails after working for 2 weeks
    By Linsys in forum Administrators
    Replies: 10
    Last Post: 10-07-2008, 12:42 AM
  2. Replies: 7
    Last Post: 10-05-2007, 03:14 PM
  3. Replies: 18
    Last Post: 03-20-2006, 01:22 PM
  4. Installation on FC3 under Xen
    By andreground in forum Installation
    Replies: 9
    Last Post: 11-14-2005, 09:29 AM
  5. Can't start MySQL to init db - Gentoo
    By fak3r in forum Installation
    Replies: 3
    Last Post: 10-10-2005, 06:59 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •