sql server nightmare

B

bogle300

New Member
Joined
Jan 21, 2010
Messages
64
Reaction score
0
agh...gonna have a good weekend...my sql database blew up tuesday night and I got absolutely no sleep working on it. Luckly I had hardware support for my server and called dell to provide software support for $600 bucks...they helped me from 3am to 4pm...it was the financial database and I had no quality backup and got sooooo stressed out. After working from 10pm to 3am I knew I had to find some tech support...the transaction log somehow copied itself on the C drive and i didn't have enough hard drive space to hold the 2nd transaction log so sql blew the database up and deleted it!

After 12 hours of work, it finally worked but it was a nightmare because I had no simple backup to restore it! I have moved database files around a hundread times and never had a problem...never had such a wierd problem...the query anayler command didn't work have the time...finally opened the master table and manually edited some fields to put the database in emergency mode to finally get it back up!!!

Always make a backup even if you have do it a thousand times but it will blow up the one time you don't make a backup!!!
 
Understood perfectly clearly, that is why IT jobs pays well, IT is fully responsible for backup and if they screwed up, their job is smoked just like that.

Wish you best of luck get these restored. Then afterwards, you know what you ought to do...
 
Yeah, I had memorial experience that I save the company headache. To let you know that I work with computer industry for 30 years. Back in 1995 I was a contractor for Marriot Hotel and I'm responsible for over 26 hotels in western NY and liason between Marriot HQ in Dc and francise HQ in NY. The francise HQ name Del Monte which owned many of Marriot Hotels. One day Del Monte got into crisis with financial post transaction. So, when I got there and check to see what happening. I find out that they ran out of space on 4 old SCSI hard drives mirrored (total 8 hard drives). They was using Novell Netware OS. So, I told them that the need additional hard drives and will take between 3 to 4 days to get it. Financial dept got freaked out that they CANNOT cancel the year end post transactions. If they do, they will lose week worth of data input. It's more of hanging on the string and waiting to complete the proccess. So, I sat there and thinking if there's a way to expand the hard drive without interruption. Then I got a brain storm. So, I explain to financial dept what I can do. They think that I'm crazy and risky. At the same time I didn't asked my outsourcing boss a permission to do what I'm going to do. I know the outsourcing policy and I know my boss behavior. I know he will refused to let me what I'm gonig to do, but I know it will work. So I went ahead do it.
What I did? Like what I said earlier that there are 8 hard drives, 4 hard drives mirrored each other. I disable the mirroring system and now only 4 main hard drives running together. I powered down the unmirrored drives and I took out one unmirrored hard drive and added to the 5th slot of the main hard drives. Then I powered up the 5th hard drive and set the command to Novell to rescan to find any new hardware. Novell found it and connected. Then I mounted to expand the volume, increasing the 65 MB available to 265 MB space available. Yeah, think of the old hey day of MB size. It worked and a big sigh. So I told the financial dept to go ahead and do re-post. 1 and half hour later, post transaction was complete. Leaving only about 185 MB space available. Everyone in the finance dept was very happy. Then I told the company to go ahead and get two new SCSI hard drives and added back to the unmirrored drives and turn on the mirroring system. Need to get that done asap. So that what I mean about "risky" part. So, I told the Del Monte CEO that I may not come back again to finish up the work because I know I will get fired soon after I told my boss. So, I decided to go ahead and do resignation letter instead of having him to fire me bcause it will look bad on resume. Next day, I was back in my office and I explained to my boss of what happen and what need to be done. He screamed at me, pissed off big time. Then I handed him my resignation letter. Then days later, CEO of Del Monte want to hire me to work full time as IT. I told him no thanks and I decided to move to Maryland. I know working for Del Monte, I will make more than what I work for outsourcing company. So, it's interesting week for me that I save the Del Monte cost of having financial dept working overtime to re-do the week worth of data input. It only took me 20 minutes to move the hard drive to give room for re-posting. I felt good about it and I did it.
 
Let me explain you little more of why it's risky. I have to be real careful about moving the hard drive, before I added to 5th slot, I have to be real careful to be sure I got the right SCSI ID number and also the terminator. If I set the wrong ID number, like two same ID numbers, the whole server will crashed and I'll be fucked. You know what I mean? Since I'm expert on Novell and been working with Novell since 1986 or 87 when Novell first release 2.16b version. So, I know inside and out about Novell and the hardware. I love Novell. Sadly, now I no longer work with Novell since I left the outsourcing company. Got more involved with M$ and Linux.
 
bogle300 said:
agh...gonna have a good weekend...my sql database blew up tuesday night and I got absolutely no sleep working on it. Luckly I had hardware support for my server and called dell to provide software support for $600 bucks...they helped me from 3am to 4pm...it was the financial database and I had no quality backup and got sooooo stressed out. After working from 10pm to 3am I knew I had to find some tech support...the transaction log somehow copied itself on the C drive and i didn't have enough hard drive space to hold the 2nd transaction log so sql blew the database up and deleted it!

After 12 hours of work, it finally worked but it was a nightmare because I had no simple backup to restore it! I have moved database files around a hundread times and never had a problem...never had such a wierd problem...the query anayler command didn't work have the time...finally opened the master table and manually edited some fields to put the database in emergency mode to finally get it back up!!!

Always make a backup even if you have do it a thousand times but it will blow up the one time you don't make a backup!!!
Click to expand...
This is EXACTLY why you need a regular quality backup. I have two words for you. Disaster Recovery Plan. OK that was three but you get the point.
 
SneakerNet said:
Let me explain you little more of why it's risky. I have to be real careful about moving the hard drive, before I added to 5th slot, I have to be real careful to be sure I got the right SCSI ID number and also the terminator. If I set the wrong ID number, like two same ID numbers, the whole server will crashed and I'll be fucked. You know what I mean? Since I'm expert on Novell and been working with Novell since 1986 or 87 when Novell first release 2.16b version. So, I know inside and out about Novell and the hardware. I love Novell. Sadly, now I no longer work with Novell since I left the outsourcing company. Got more involved with M$ and Linux.
Click to expand...
WOW.. terminator. That is some old school SCSI.
Newer SCSI busses are self terminating.
 
And top of that why I went ahead to do that is because Del Monte didn't do a good job on doing tape backup, I tested the tape backup to see how clean it is. It was horrible, lot of read errors. This is the reason I made a decision to expand the hard drive space and save time. I'm on a thin line of either getting the server crashed and getting fired. I had so much fun.
 
SneakerNet said:
And top of that why I went ahead to do that is because Del Monte didn't do a good job on doing tape backup, I tested the tape backup to see how clean it is. It was horrible, lot of read errors. This is the reason I made a decision to expand the hard drive space and save time. I'm on a thin line of either getting the server crashed and getting fired. I had so much fun.
Click to expand...

Remember if I screwed up big time, Del Monte can sue Outsourcing company and it will go out of business. All that because of me. Scary thought but I did it. It went smoothly.
 
bogle300 said:
agh...gonna have a good weekend...my sql database blew up tuesday night and I got absolutely no sleep working on it. Luckly I had hardware support for my server and called dell to provide software support for $600 bucks...they helped me from 3am to 4pm...it was the financial database and I had no quality backup and got sooooo stressed out. After working from 10pm to 3am I knew I had to find some tech support...the transaction log somehow copied itself on the C drive and i didn't have enough hard drive space to hold the 2nd transaction log so sql blew the database up and deleted it!

After 12 hours of work, it finally worked but it was a nightmare because I had no simple backup to restore it! I have moved database files around a hundread times and never had a problem...never had such a wierd problem...the query anayler command didn't work have the time...finally opened the master table and manually edited some fields to put the database in emergency mode to finally get it back up!!!

Always make a backup even if you have do it a thousand times but it will blow up the one time you don't make a backup!!!
Click to expand...


Good advice,
Being in the IT field I have also experienced stuff like this, knock on wood, so far i always had a good backup to fall back on. Today, I work for the third largest bank in Canada and I'm responsible for making sure 140 servers are backed up everynight not to include supporting them-what a pain in the butt, however, stories like this make it all work worth it.
 
cdaigle430 said:
Good advice,
Being in the IT field I have also experienced stuff like this, knock on wood, so far i always had a good backup to fall back on. Today, I work for the third largest bank in Canada and I'm responsible for making sure 140 servers are backed up everynight not to include supporting them-what a pain in the butt, however, stories like this make it all work worth it.
Click to expand...

Yes, that is right. Here where I worked now, we have over 200 servers and mutilple UPS'es and mutliple tape backup system and multiple NAS for backup purpose. All well worth over 50 Terabytes.
 
Back
Top