Bored out of my freaking skull [Archive] - Glock Talk

PDA

View Full Version : Bored out of my freaking skull


Zak3056
08-02-2005, 20:10
I'm currently sitting in my office (14 hours after I came to work) babysitting a server migration, anticipating being here at least another 4 hours.

Why am I doing this on a tuesday evening, you ask? Because said file server had the poor manners to begin its death spiral earlier this week... so instead of a nice, leasiurely weekend to move data over, all of it has to be done by 8AM tomorrow morning. This would not normally be too bad, but the hot backup machine I'm pulling data off of is a 150MHz Pentium with 64 megs of RAM. Compounding this technical hurdle is the poor management of files by the departments involved. There are two directories that I've found so far that have over 20,000 files each (no subdirectories, no sir!)

It's quite saddening to sit here and watch a 3KB file take about 750 miliseconds to copy over a 100MB ethernet segment on its own VLAN. It's amusing to note that mutli-megabyte files take about the same amount of time. It's downright terrifying that there are over 200,000 discrete files on this box.

Once the files are copied, I get to reset all the permissions, verify the RAID array is good, make sure the UPS is working, provision a backup device and make sure I can make a good backup. I'm wondering if I'm going to be done by 8AM.

At least I get to skip work tomorrow. ;g

Console me! ;f

Zak3056
08-02-2005, 23:54
1:54AM, still here. Estimate two hours to go...

Zak3056
08-03-2005, 02:09
4:08AM, still here. Sleep beckons.

gwalchmai
08-03-2005, 06:29
Originally posted by Zak3056
Compounding this technical hurdle is the poor management of files by the departments involved. There are two directories that I've found so far that have over 20,000 files each (no subdirectories, no sir!) And I thought my users were the only ones who believed in the "toss it in the closet" school of file management... ;)

funbob
08-03-2005, 07:52
I knew there was I reason I decided not to go into IT... ;f Not that management is so much better, I just get to babysit people instead of servers ;f .

vanRichten
08-03-2005, 08:02
Ack... that's never fun. I feel your pain.

I had a new hire wonder why everything she put in deleted items in Notes was removed when she closed the program. Evidently she was using it as a storage directory at her old job. ;g

Zak3056
08-03-2005, 10:22
I finally left work at 5:10AM. I didn't bother to set the building alarm, because the first shift fabrication guys had already started to come in. ;g

Originally posted by vanRichten
I had a new hire wonder why everything she put in deleted items in Notes was removed when she closed the program. Evidently she was using it as a storage directory at her old job. ;g

I had two users who did the same thing. One of them had an entire folder tree below his deleted items in outlook, then wanted me to recover it all when he accidentially emptied the folder!

ngray
08-09-2005, 11:02
feel your pain, I was up til 6am saturday morning migrating an apache box and another mysql box...

Cryptoboy
08-09-2005, 11:20
Been there, done that, feel your pain! I work for an IT outsourcing company, so I deal with stuff like that on a semi-frequent basis (almost even the oldest servers we have here aren't near as bad as that one you described!)

frefoo
08-09-2005, 20:59
I always say "I am in the wrong job if I want to work 9 to 5".

I once worked 36 hours straight until backup arrived. I was back in 6 hours.

That is part of the job get used to it.

If you cant then find a different job.

Sorry to sound so mean but the simple fact is thats how it goes in IT from time to time. It is never fun but it is required from time to time.

Zoolander
08-10-2005, 00:24
Baby sitting a server is a snooze job.

Meman5150
08-10-2005, 00:27
IT pays good so I'll stick with it. ;c goodluck

Zak3056
08-10-2005, 18:18
Originally posted by frefoo
That is part of the job get used to it.

If you cant then find a different job.

Sorry to sound so mean but the simple fact is thats how it goes in IT from time to time. It is never fun but it is required from time to time.

You misunderstand. I enjoy my job and don't really mind when it requires long hours (though I mind it a hell of alot more since I became an exempt employee than when I got paid for it.) That said, I'm still human and do get to vent when the day goes particularly poorly.

I'm not a wet behind the years rookie who is mystified as to the ways of the IT world, and shocked, shocked, I say, to have had to work a 21 hour day. I've been doing this professionally (in one way or another) for fifteen years.

Egyas
08-10-2005, 18:46
Originally posted by Zak3056
[BCompounding this technical hurdle is the poor management of files by the departments involved. There are two directories that I've found so far that have over 20,000 files each (no subdirectories, no sir!)[/B]


I have a very simple solution for this. I have several departments that do this kind of crap. On a regular basis. Especially the reporting team where I work. The ONLY department I let slide is the HR department, because they are REQUIRED by company policy to keep everything for three years.

Anyway, When I have to migrate data, home directories, departmental shares, etc, etc, I simply delete ANYTHING that hasn't been modified in the last 6 months. If you haven't used it in 6 months, you don;t need it. I have a 6 month tape rotation, so that means I can hold your crap for another 6 months. If, after 1 year you haven't needed it, remake it and go cry in your Cheerios. lol

Zoolander
08-11-2005, 00:11
Holy control freaks.

Egyas
08-11-2005, 05:32
Originally posted by Zoolander
Holy control freaks.

I kow I sould a bit like a BOFH, but there is a method to my madness. We have limited drive space available for departmental shares, home directories, etc. Even running with full inline compression I am constently having to beg and plead the departments to clear space. Purchasing additional storage space is not an option right now, so we're stuck with the space we have. When the oppertunity happens to "help myself" to free-up some space, I take it.

Zak3056
08-11-2005, 17:08
Originally posted by Egyas
Anyway, When I have to migrate data, home directories, departmental shares, etc, etc, I simply delete ANYTHING that hasn't been modified in the last 6 months. If you haven't used it in 6 months, you don;t need it. I have a 6 month tape rotation, so that means I can hold your crap for another 6 months. If, after 1 year you haven't needed it, remake it and go cry in your Cheerios. lol

We're an engineering/manufacturing company, and a large percentage of our business is nuke work. That attitude isn't going to fly at my place, and rightly so. It happens fairly often that we'll have to dig back and find something we did 10 years ago for a customer to either refabricate a part as a replacement, or give them data that THEY have lost along the way.

The directories with 20k files were DXFs of every possible iteration of one of our products for use in manufacturing. Also, the amount of space used was pretty much background noise (about 80MB) so aside from the time it took to migrate (and the time it takes to FIND anything) there's no REAL issue here. I'm talking to them about building a logical heirarchy to make it easier to work with, but they're resistant to change.

We have limited drive space available for departmental shares, home directories, etc.

You know, the big problem I always have is backup capacity. I have about 100GB of unused storage laying around (all of it RAID1 or RAID5) and can always buy more disk cheaply (73GB of 10kRPM SCSI disk is going for about $250) but we're talking $2-3k minimum for more tape to back it all up on (unless you want to go VXA or DAT) or someone has to come in at night and swap tapes, your backup window gets longer, and your restore times start to become extremely scary.

Egyas
08-11-2005, 20:17
Originally posted by Zak3056
We're an engineering/manufacturing company, and a large percentage of our business is nuke work. That attitude isn't going to fly at my place, and rightly so. It happens fairly often that we'll have to dig back and find something we did 10 years ago for a customer to either refabricate a part as a replacement, or give them data that THEY have lost along the way.

The directories with 20k files were DXFs of every possible iteration of one of our products for use in manufacturing. Also, the amount of space used was pretty much background noise (about 80MB) so aside from the time it took to migrate (and the time it takes to FIND anything) there's no REAL issue here. I'm talking to them about building a logical heirarchy to make it easier to work with, but they're resistant to change.
Obviously, in your situation my policy would be quite probably drastically bad. I work for a telecommunications company, and the majority offenders deal with order entry, former telemarketing people, Business markets, and VoIP order entry. In these specific areas for our company, there is ZERO need for long-term "static" document storage.


Originally posted by Zak3056
You know, the big problem I always have is backup capacity. I have about 100GB of unused storage laying around (all of it RAID1 or RAID5) and can always buy more disk cheaply (73GB of 10kRPM SCSI disk is going for about $250) but we're talking $2-3k minimum for more tape to back it all up on (unless you want to go VXA or DAT) or someone has to come in at night and swap tapes, your backup window gets longer, and your restore times start to become extremely scary.

Oddly, we have the EXACT opposite problem. We have massive backup potential (like 400GB), but little storage space. The servers currently being used have only 50GB or so for Departmental shares, home directories, etc. With compression, we stretch that to almost 100GB. And there is zero capital available for expansion of these servers after commiting to a MASSIVE (as in for 40,000 employees) PC and laptop refresh. Now, we are in the process of migrating these legacy servers, domain controlers, etc, over to new servers with larger capacity, running Windows 2003 advanced server, etc.

My, will THAT be nice when it's done in 6 months or so. :cool:

frefoo
08-11-2005, 21:16
Originally posted by Zak3056
You misunderstand. I enjoy my job and don't really mind when it requires long hours (though I mind it a hell of alot more since I became an exempt employee than when I got paid for it.) That said, I'm still human and do get to vent when the day goes particularly poorly.

I'm not a wet behind the years rookie who is mystified as to the ways of the IT world, and shocked, shocked, I say, to have had to work a 21 hour day. I've been doing this professionally (in one way or another) for fifteen years.

I did misunderstand for that I apologize


Originally posted by Zak3056
Once the files are copied, I get to reset all the permissions, verify the RAID array is good, make sure the UPS is working, provision a backup device and make sure I can make a good backup. I'm wondering if I'm going to be done by 8AM.


Why do you need to reset the permissions? Look into a product like Secure Copy from Small Wonders Software.

Link (http://www.functional-it.com/migratie_smallwonders.htm)

I assume Windows based on your other posts.

Also from your other post regarding backup/restore times, perhaps an enterprise storage platform (EMC/HDS/IBM) can help improve that.

While the costs for EMC/HDS/IBM etc are high to start, in the long run it may help solve some of your problems.


DISCLAIMER: I am a consultant that works with EMC/HDS/IBM gear at different clients. At one client, I support over 2 petabytes of storage and more then 5000 hosts. I am not an employee of EMC/HDS/IBM or Small Wonders.

gwalchmai
08-12-2005, 04:56
Originally posted by Egyas
Oddly, we have the EXACT opposite problem. We have massive backup potential (like 400GB), but little storage space. The servers currently being used have only 50GB or so for Departmental shares, home directories, etc. With compression, we stretch that to almost 100GB. And there is zero capital available for expansion of these servers after commiting to a MASSIVE (as in for 40,000 employees) PC and laptop refresh. Now, we are in the process of migrating these legacy servers, domain controlers, etc, over to new servers with larger capacity, running Windows 2003 advanced server, etc. 40,000 employees and 400 GB of backup potential - I commend your frugality, sir! ;f

Egyas
08-12-2005, 06:30
Originally posted by gwalchmai
40,000 employees and 400 GB of backup potential - I commend your frugality, sir! ;f

LOL. Perhaps I should have been more clear. 40,000 employees enter-prise wide, 600 or so here in my facility. :)

The reason I mentioned the 40,000 is because of them, roughly 32,000 or so received a new Desktop or laptop, thus eating up all capital that wasn;t earmarked for the new server or other "strategic projects". No $$$ for the last 2 years fo upgrading the legacy windows NT servers because they're phazing them out.

David_G17
08-12-2005, 20:59
http://www.nonstop.lv/files/bubblewrap.swf

^ next time you're bored.

Zak3056
08-13-2005, 20:40
Originally posted by frefoo
Why do you need to reset the permissions? Look into a product like Secure Copy from Small Wonders Software.

The old machine was running Samba 2.0 on Redhat 7.1, while the new (actually not new, but space available and able to be dedicated to my eng department) machine is a Dell NAS running "Windows Powered" (aka Windows 2000 embedded.) UNIX file permissions and samba ACLs don't translate well to NTFS permissions, requiring everything to be reset once the copy job finished.

Also from your other post regarding backup/restore times, perhaps an enterprise storage platform (EMC/HDS/IBM) can help improve that.

While the costs for EMC/HDS/IBM etc are high to start, in the long run it may help solve some of your problems.

The money's just not there for that now. We've been cutting staff for the last year trying to keep going until the work comes back and spending lots of bucks on a SAN and everything that goes with it isn't going to help. ;f

Long term, I'd like to move in that direction, but really my data storage requirements aren't that high at present... I'm only backing up ~160GB, and I only have about 80 users accessing it, so there's no performance gain there for me, either. We've talked about going paperless on all the engineering/QA documents in the future (we have entire buildings dedicated to housing paper, believe it or not--for example, there are over 250,000 pages of documentation we have to retain for one Taiwanese nuke plant alone...and all we do is pipe supports!) and that would REALLY require something along the lines of what you suggest, if only for the warm fuzzy feeling that kind of gear gives you. But until then, DAS, VS80 tapes, and some hot backup is all I really need.

(edited to fix annyoing smiley that appeared instead of something else I posted.)

frefoo
08-14-2005, 12:49
Originally posted by David_G17
http://www.nonstop.lv/files/bubblewrap.swf

^ next time you're bored.

~1

I have to remember that

NetNinja
08-18-2005, 23:08
I am way late on this thread but to see an engineering/manufacturing firm using a 150mhz machine with 64mb of ram for important files?

I know that probably was not on a raid but on a Red Headed step child desktop machine.

I know file servers don't have to be beefy but holly smokes!

I walked into a job that was paying me 65k.
It was one of those rip off schools that load you up with student loans and you hopefully get a job being a court reporter or a medical paperwork assistant. Trade school

After talking to the other Network admin and him telling me the company not having any money for better equipment or the jurry rigged software they were using to rely on for critical bussiness needs I turned in my resignation after a week.

I don't do crisis management. it's stuff like you mentioned about getting a server that's critical to your company up in 24 hours with no resources is why I don't work for company's like that anymore.