Results 1 to 5 of 5

Thread: Halloween - IT Admin Horror Stories

  1. #1
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default Halloween - IT Admin Horror Stories

    Post your favorite 'oh crap' moments here!

    (Whether they be 100% true. or an exaggerated IT horror story that's taken on additional mythical proportions every time it's told.)

    *OS/network/disk failure, virus/spyware/rootkit, even ID10T user error/configuration moments - not a 'my favorite bug' thread.
    Props if the situation was solved by deploying Zimbra though

    Last edited by mmorse; 10-31-2008 at 12:43 PM.

  2. #2
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,017
    Rep Power
    25

    Default

    Well ... I was working for a large multi-national running HP-UX systems and Oracle/SAP, and one day the clock struck twelve and the OS just started to disappear Down went SAP and Oracle like a sack of spuds!

    Mayhem broke with the IT manager standing over my shoulder wanting to know what had happened ... I did not have a clue, and I could not even get onto the system as it was completely hosed! So the task of restoring the server began and after 30 minutes I had everything backup and running again. phewww

    Until 1pm! The system disappearing again What the hell is going off, panic set it, this time I managed to keep a couple of sessions open to allow me to check the system.

    And then it clicked .... I wonder .... Yep indeed, somebody had setup a cronjob AS ROOT, that attempted to 'cd' to a directory which then proceeded with a 'rm -rf *' Though the ******* other admin did not verify that the directory existed before performing the remove! Well once we had restored the system again, the cronjob was removed, and we were all running fine again.

    Morale of the story is to always protect root access and ensure you have adequate backups!!!

  3. #3
    Join Date
    Oct 2005
    Location
    USA, Canada and India
    Posts
    777
    Rep Power
    11

    Default look daddy..i can delete :(

    My 9 months old son always lies down next to me when I am working. Other day I was logged in using WinSCP (root) to one of our client’s zimbra server (more than 700 users on it)
    I got distracted doing something and suddenly I hear big thump and look at my screen, It said "Are you sure you want to delete file ‘/’ ?"
    My son woke up from his sleep and hit the delete key on my laptop using his leg..and thank god he did not hit twice ..lol
    So now i keep him at-least 2 feet away from my laptop..but i see it him deleting one of my servers soon..just question of time.

    Raj
    i2k2 Networks
    Dedicated & Shared Zimbra Hosting Provider

  4. #4
    Join Date
    Mar 2007
    Location
    Austin
    Posts
    441
    Rep Power
    8

    Default

    One time, I was working at a large hosting place here in Texas. We had a customer who was doing online pharma sales, and the DEA wanted to investigate them. We had to make up some excuse as to why they were going to be down, don't for the life of me remember what it was.

    So we've got feds wandering around the data center wanting to poke at servers. Their main system (db server probably) is running on a nice Dell 2650, with a RAID 5 in it. The DEA folks decide they want the data from this system.

    So one of our techs goes out to the system. The live, running system. And he pulls out one of the drives.

    Okay, but this is a RAID. Got that nice R at the beginning for Redundant. Pulling a drive out of a RAID 5 set won't really hurt anything too much.

    Unless you then decide to put it back in. When you put the drive back in, the card now has to figure out what is going on. It's going to have to determine whether that is a good drive, a new drive, if it has to completely restripe, or if it can recover, etc.

    Which is what they do, you pay for expensive RAID cards because they can handle that.

    Unless, while the system is trying to figure out WTF you're doing to it...you pull the power.

    Yeah, the DEA wasn't happy.

  5. #5
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,017
    Rep Power
    25

    Default

    roflao Had a user once who made a mistake within a client/server application and decided to run to the computer room and switch off the server He thought he could get there before the change actually took place

Similar Threads

  1. [SOLVED] admin account email issues
    By dazza_act in forum Installation
    Replies: 2
    Last Post: 10-06-2008, 07:26 PM
  2. Disable SSL on the Admin Port 7071
    By rasputin in forum Installation
    Replies: 2
    Last Post: 04-06-2008, 04:29 AM
  3. Silly mistake -- now cant log into admin console
    By animasana in forum Administrators
    Replies: 10
    Last Post: 07-05-2007, 05:00 AM
  4. How do I set Admin password?
    By sjames in forum Installation
    Replies: 4
    Last Post: 07-24-2006, 05:01 PM
  5. Zimbra refuses to auth admin.
    By alexbsa in forum Administrators
    Replies: 1
    Last Post: 12-05-2005, 12:23 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •