A lesson learned on yesterday's job !

Andy's thread about the BA server has reminded me to post this.

In between certifying my companies' installers work I do EICRs. 

Since April (on and off) I have been doing an EICR on a medium sized hospital with 4 floors and several hundred circuits. The Estates Dept gave me the circuit schedules from the last EICR which was done 10 years ago.

They have just one server room within which sits a large server backed up by a 2 x 16A UPSes. 

On my circuit charts from the previous report it stated that circuit 9 fed the sockets for the server room which also supplied the telephones.

Having been impressed by the accuracy of these circuit schedules up to now I thought I was taking a safe risk.

I decided to inspect circuit 2 -  a ring main apparently feeding 2 admin offices. So I tripped off the MCB for circuit 2.

As I opened a socket to split the ring legs for r1, rn, r2 I could hear in the distance the sound of a UPS bleeping. For some reason I dismissed the noise as probably being something else - hospitals always have bleeping machines. After a few minutes the bleeping changed to a more urgent tone - then silence - then crowds of people began running around shouting "the internet and the phones have gone off - what have you done ?".

Anyway to cut the story short, the circuit charts lulled me into a false sense of security. Perhaps this particular board was tested on a Friday afternoon 10 years ago. Or someone changed the circuits and did not update the circuit schedule.

What have I learned from this experience ? With hindsight I think I should have asked for a scheduled shutdown of the server at the beginning of the job in order to correctly identify the circuit feeding the server room.

Lesson 2:  the UPS is NOT big enough, or it's batteries have failed (and have not been tested)  It should have held up and kept t'internet going a bit lot longer

Did you get a telling off over this?

Sometimes I turn the power off in the managers office so he knows I'm still about lol.

I remember one in a hospital years ago, old board with loads of rewireable fuses in it, non of them marked up and needed to isolate an outside light circuit, it was in an area of the hospital that didn't contain any critical gear. Anyway hospital spark looks at it, did a bit of swearing, opened a fitting on the offending circuit and shorted the cables with a screwdriver, there was a bit of a bang and the lights went out, "sorted, there you go" says the spark, "how does that help finding the fuse"? I asked, "it doesn't" he replied, "but I'm off for a fortnights holiday tomorrow, let some other ****** find it, I'm sick of boards not being labelled", He put his screwdriver in his pocket and disappeared. 

ProDave said:

Lesson 2:  the UPS is NOT big enough, or it's batteries have failed (and have not been tested)  It should have held up and kept t'internet going a bit lot longer

Click to expand...

No. The UPS only really needs to keep the servers up long enough to perform a shutdown. The UPS isn't to keep the servers sunning for a y length of time, that is what generators are for.

The IT guy who was called out said that the UPSes are supposed to last for 20 mins but I did'nt think I'd had the circuit off that long . The generators would have kicked in had there been a proper power cut. My understanding was that the UPSes are just designed to keep the server going until changeover to generator power.

When I reset the 32A MCB it had to cope with the flat UPSes and gave a loud crack and that was the end of it- the MCB was finished. The ring main had 2 x 16A UPSes on it along with 2 large offices (plus the main intruder alarm for the hospital) AND all this was also protected by a 100ma RCD !. I will recommend 2 new circuits be run, one for the server room and another for the intruder alarm- both on non - RCD circuits.

Like anything UPS’s batteries have a finite life. Based on what the IT guys told me I was confident I’d got half an hour plus to carry out a feeder swap over at two of our substations. What I didn’t know I’d shutdown the radio system, I carried on totally oblivious to the ongoing chaos I’d caused. The UPS packed up after fifteen minutes, and as per usual I got the blame.

rapparee said:

Did you get a telling off over this?

Click to expand...

Not yet, it happened at 5.00pm on Friday. I have emailed my boss telling him that if he gets a callout bill from the IT engineer then don't pay it as I acted on circuit information supplied to me from the Estates department which turned out to be incorrect. However with hindsight I now believe that this incident could have been avoided if I had arranged for a scheduled shutdown of the server room at the beginning of the job and correctly identified the circuit feeding that server room.

Tony I'm with you on this. It seems to me that most UPSes I come across, ESPECIALLY in hospitals, have been neglected even though they are often supplying very expensive equipment !

just remind them that if it had been a power cut then they would have had the same problem

Andy™ said:

just remind them that if it had been a power cut then they would have had the same problem

Click to expand...

Thats my usual thoughts in such a situation, however it sounds like this site is backup by a generator, and the system had a UPS that was good for perhaps long enough for it to come online, just not long enough for the time the OP had it off.

Best one (in laugh-able terms) was testing on an indoor market, the stalls/kiosks had small single phase boards in each one, was going on all right untill I got to a largish stall that that sepaarte stall with more displays and stock next to it whcih was a second stall he ahd rented, now this on a bit of a corner as you come into the market (picture a entrance corridor going into the market hall and opening up to the market hall an external corner right as you emerge) anyway, knocked the board off in this second area, and lade from the stall round teh corner comes in shouting moaning, apparrently her power to the mobile accessories stall had gone off, Oh dear says I, we will have it back on in just two ticks, sorry about that, I didn't know your shop was off this board. I stick my head into her stall to see if all her circuits have gone off (as opposed to any being from the stall the other side), and then she starts on about, what happens if she was upgrading the firmware on a telephone for a client when it goes off and leaves the telephone with partial firmware and breaks it and goes on about trying to hold the landlord of teh building (our clinet responsible). I say, its not likely to be an issue here it, becuase you are using a laptop computer, so you've got redundancy covered. "Oh no", she says, "The battery is completely dead, so it only runs off the mains". I shook my head, and as she wasn't claiming that anything like that had happened, I thought it wasn't worth my time trying to point out the bleeding obvious that if a loss off power would cause damage to what she was working on, then the least she should do is ensure that there was a functional battery in the laptop!

well here's mine from the other year. row of units. 1x DNO feed. i need to re-terminate & reposition the SWA that comes into the unit im in and the one that then heads off to the next unit. arrange with next unit for when he's out for a few hours. 15 mins in all going well, both SWA removed from old bodge, one partly re-done about to push into new box. goes outside to van for something and the lass at the unit 2 down the other way asks if my power is off too...

I was doing an eicr in a shop (estate agent) and there was one socket I just could not isolate. In the end, with the whole CU turned off, that one socket was still live.

In the end we concluded it must be fed from the shop next door, but the next shop owner would not allow me access to investigate or do anything about it.  They just agreed not to use that particular socket.

ProDave said:

In the end we concluded it must be fed from the shop next door, but the next shop owner would not allow me access to investigate or do anything about it.  They just agreed not to use that particular socket.

Click to expand...

Short it out, and pop both L-N into a connector block while its dead, someone will find it and disconnect it at source then  :innocent

or just keep using it

ProDave said:

I was doing an eicr in a shop (estate agent) and there was one socket I just could not isolate. In the end, with the whole CU turned off, that one socket was still live.

In the end we concluded it must be fed from the shop next door, but the next shop owner would not allow me access to investigate or do anything about it.  They just agreed not to use that particular socket.

Click to expand...

Free electric I'd have it overloaded lol

what a waste...

if you swapped that black plug with small flex from rhs to 4 way adapter with the white plug on the left of the first adapter, you could then fit something into the socket on the left of the 2nd adapter...

Where I did my apprenticeship there was a safety bulletin about overloading a socket.

The picture showed something like that with stacked 3 way adaptors. Each plug had two flexes going into it. But that still wasn't enough so the last bit of kit, they had stripped the flex back, L into one plug, N into another and E into a third plug as none of the plugs would fit a third complete flex.

I currently have my office set up in an un finished room in the new house.  A single extension lead ending with a 3 way adaptor. Each of the 3 outlets from the 3 way adaptor feeds a 6 way socket strip.  What could possibly be wrong with that?

Phoenix said:

Short it out, and pop both L-N into a connector block while its dead, someone will find it and disconnect it at source then  :innocent

Click to expand...

That sounds familiar, switch off a wait for someone to start shouting. I never did find out what a 800A fuse switch fed, I turned it off and nobody noticed.

Andy™ said:

if you swapped that black plug with small flex from rhs to 4 way adapter with the white plug on the left of the first adapter, you could then fit something into the socket on the left of the 2nd adapter...

Click to expand...

All good apart from the fact there is no socket on the left of that second adapter.

I think a lot of health services know their UPS are knackered and do nothing about it, at the old hospital my wife worked in they tested the generator every Thursday at 11 o'clock, they'd ring round and tell everyone to shut down their computers while they did the change over, so they must have known something was wrong.