Jump to content
Brian Enos's Forums... Maku mozo!

What Not To Do In A Data Center -


Recommended Posts

So we are upgrading our phone system software at the office today and have a couple of Telco techs in working on it in the cold room of our data center. I thought we had someone in there working with them but apparently they went to lunch or something leaving these two guys in there by themselves (yeah, I'll deal with that later).

A little while ago I get a call from a user in our Michigan office reporting "system overheating" messages appearing while they are using an application on one of our servers so I ask one of the guys I work with to check it out. Turns out that the telco guys had a sheet of paper with some notes they were folowing and rather than just holding it in their hands they found they could use the suction from the intake fan of the $400k server in the rack next to them to hold it right at eye level. Isn't that convenient? Other than it completely cut off the air circulation to the box it worked great.

It looks like the server's self protection code caused it to shut itself down before any real damage was done fortunately.

Link to comment
Share on other sites

That sucks! :D

A long, long time ago at an insurance company far, far away, a manager used DBase III to manage a lot of business. An enterprising secretary asked to borrow the disks (remember 5.25" copy protected disks?) to learn the program. A few days later they were returned, topped with a thank you note, all neatly stuck to the file cabinet with a magnet....

Link to comment
Share on other sites

  • 2 weeks later...

years ago I managed an IT shop which, because of org structure, also owned the internal telecom organization.

A young lady that worked for me had a propensity for metallic nail-polish (hey, this was the 80s), and one day while giving a tour of the switch room, she ran her fingers down a row of punch-block and... shorted out an entire bank of phone lines with her fingernails.

We cut out the section (including the charred backer-board) and presented it to her. It became the annual "golden fingernail" award in our department, awarded each year for the dumbest thing that happened in the data center...

Link to comment
Share on other sites

Eons ago before Halon systems in server rooms were common place and vacuum tubes were common in computers I saw a guy hang a cable up on a convenient "hook" in the ceiling tiles. Of course this hook was to the sprinkler system.

Luckily they were in the process of purchasing a system to update the vacuum tube system and replacing it with a "modern" computer.

I was about 7 at the time if memory serves. I still remember the "hisssssss....whooooooosssshhhhhhhh....sizzzleeeee". Ah, good times, good times.

I often wonder if they "accidentally" did that for the insurance.

Link to comment
Share on other sites

:D The best I ever saw was when I started in a new shop in my early days as a programmer and took a tour of the data center. Walked thru and noticed what look like a metal saw blade stuck in the ceiling and asked why it was there.....Evidently a 'bright' electrician came in to cut thru some wires and neglected to check if the wires were hot or not. They took him out on a stretcher and left the blade in the ceiling as a warning for future stupidity!
Link to comment
Share on other sites

During my "office space" days, I worked for about 5 different companies (mind you I actually sat in the same cube during all the outsourcing/buy outs over the four years). The last outsourcing deal was between WorldCom and EDS. To try and impress us new comers, EDS took us on a tour of their "campus" in Plano. This is a very large corporate campus with about a half dozen multi-story buildings and manicured grounds. The most opulent thing was a enclosed glass bridge 8 stories up that connected the average joe to the "god pod" where the executive officers kept their quarters. Being a very nuts and bolts kind of group, we peppered the tour guide with lots of questions.

One question centered around their "raised floor space" and the capability of their on site diesel generators. The reply was that there was a weeks worth of fuel that would keep the entire building humming. When asked whether they had included the building's cooling system onto the backup power source, the manager responsible got kind of a funny look and said that "no, we didn't wire in the A/C." Six stories full of servers, midrange, and mainframe machines and no way to cool the space should the power go out....Don't know you'll be needing all that fuel...Did you get the memo?

On this same tour I kept noticing some bright orange signs pasted to the doors of the breakrooms as we passed by. I finally went up close and looked at one. The sign said "DHCP coming this weekend." OMG! This was in August 1999! Just to put things in perspective, Microsoft introduced DHCP in 1994 with NT version 3.5. I went to one of my co-workers and pointed out the sign. We quickly decided that EDS stood for "eventually does something." I bet the next week, there was an army of people laid off whose sole purpose had been to assign/maintain all the IP addresses for every machine living at the campus. And I bet the next quarter, the stock price went up since they cut costs!

The final thing happened right before I quit. I had already done all the Y2k patches on the Sun boxes I was responsible for. When we went to migrate the servers into the "raised floor space," the manager in charge of the mid tier machines gave me the OS and monitoring software levels that were standard for EDS. If it wasn't "running those releases, it wasn't getting put" in his building. Those releases were about a year out of date. I have no idea what ever happened because I left the next day and never gave him the root passwords.

That place was so unreal...can you believe in 1999 they had just amended corporate policy to allow women to wear pants and no panty hose?!

Link to comment
Share on other sites

Eons ago before Halon systems in server rooms were common place and vacuum tubes were common in computers I saw a guy hang a cable up on a convenient "hook" in the ceiling tiles. Of course this hook was to the sprinkler system.

To this very day, sprinkler heads in hotel rooms are often accompanied by a little sign with the image of a coathanger behind a circle with a slash through it.

Link to comment
Share on other sites

We had an issue here, on a bi weekly basis we had a contractor come in and clean our data center (usually a wednsday) and on the weekend we would get water alarms from our underfloor moisture detection system...turned out one of their number thougth it would be OK to carry a bucket with water into the DC....

Now we run 1/2 dozen roombas :0

"That place was so unreal...can you believe in 1999 they had just amended corporate policy to allow women to wear pants and no panty hose?!"

I'd be more infavor of a policy that forbids women to wear pants or panty hose <_<

Sorry Carina I couldn't help myself...

Link to comment
Share on other sites

That place was so unreal...can you believe in 1999 they had just amended corporate policy to allow women to wear pants and no panty hose?!

EDS? Hell yes, I can believe that. And remember the level of arrogance. When EDS was bought out by GM, the EDS'ers when they got merged into the GM datacenters continued to refer to those systems as "the GM account". On the other hand, even through Ross Perot is single-handedly responsible for Clinton winning in 1992, "On Wings of Eagles" is still a good read.

Link to comment
Share on other sites

I don't have too many stupid human tricks stories to tell, but there were times when I was a lowly tech support geek for Diamond Multimedia.

I had a call where this gentleman was having problems with his newly purchased PCI sound board. During the course of troubleshooting it, I asked him to shut down his PC, remove the board, and try it in a different PCI slot. His next response was "Is it supposed to make sparks like that?" He apparently pulled the board without shutting it down first.

There was another call where a customer asked to get an RMA for his video card that was broken. I asked him what he meant by broken. Eventually it boiled down to me asking if by broken, did he mean physically, into more than one piece....

Apparently he had the case laying on its side installing the board, and a sleeping cat was disturbed by a ringing telephone and it launched itself vertically, landing in the computer case, snapping his video card in two at the PCI slot, leaving the portion of the PCB with the gold contacts stuck in the slot.

My last job before I came to where I am now, I was auditing the User Rights of the organization, and found that the domain users group was nested in the global administrators group. I also found out that my predecessor had shared the root of every drive with Full Control, Everyone. This I found out after a user in the main office got a virus that was network aware.... :o

The first IT Manager I worked with there was afraid to let us do any real work, and always contracted out stuff. His $300 an hour Vegas Pimp Daddy consultant was wanting to upgrade the Checkpoint IP110 firewall (Nokia appliance) we had in our branch office here. They decided to do this in the middle of the day without notifying anyone. Needless to say, the remote upgrade failed. This led to me getting sent out to purchase some RAM for the Firewall. I asked if he was sure that the unit could be upgraded, and what type of memory to get. Since I was relatively new there, and didn't have a corporate visa card yet, I had to drag the VP in the office down the street to the local overpriced computer store and buy a stick of SDRAM. I get back, hunt down the proper Torx wrench to open the case up, and find out the unit can't be upgraded. No DIMM slot, and the existing memory is soldered to the PCB in the unit.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...