[Cadre-politics] power supply to CSL room (downtime 08/08 11:45pm to 10:00am)

[Dan MacNeil]

I've replied list as I suspect as Stephane is more modest
than private and these issues are of interest to us all.

Dan wrote
>> We've been trying to get on the circuits that the university has backed
>> with diesel generators for a while now and we'll keep trying.

Stephane Alnet wrote:
> It's only worthwhile if the University's network was accessible from
> the Internet end-to-end (up to CSL's jack in CSL's room) during the
> outage.

yep, it is. Because the night watchman had a tendency to turn off
breakers that powered the network, all the university network gear in
our building is now generator backed. --I count this as a rare incident 
of political smoothness on my part.

The university has a $300,000 (spent twice to redo hvac) server room
with a few racks of space free, its own generator and a large group of
powerful people who freak out if things in that room go down.

I very much suspect that the equipment closet 30 feet from our door is
generator backed.

I've asked about the server room a couple times, once I got a "yes" and
then let a year go by without follow-up. The second time (when we got
mostapha) I asked the answer was "no".

One price for my recent rant is that asking a third time (perhaps for 
space in the closet next to us),  maybe will not go very well.

Still asking works better than not. I'll ask again.

> 
>> If we manage to accumulate a bit more money, we'll look @ moving email
>> servers to a collocation facility with multiple redundant power supplies
>> backed by generators.
> 
> Other ideas:
> - Amazon EC2[1] and similar forms of virtual hosting if you get money
> to pay for service (rather than purchasing hardware);

Using their calculator, it looks that disc and storage would cost us $60
per month and at 10 cents per hour for runtime, $720 per month for CPU

Right now we're using about 200G of disc.

For most vservers that might be too much.

We can get a nice colo from a good company (machine included) for
$2400 to $4800 a year. --We're close to being able to afford this.

> - Assuming the network staid up: Build a large hardware machine and a
> large UPS, and virtualize the services using Xen (assuming it's
> cheaper to buy or get donated a large UPS rather than a slew of
> smaller ones).

This is doable, and the direction I'm moving in for sanity anyway.

We've got two beefy boxes that could run everything except incoming spam 
filtering on xen

Enough APC UPS power to run both boxes for 10 hours is very, very 
roughly $1200, for 20 hours $2000

> 
> But these three ideas (colo, buying or building Xen) kinda kill the
> tinkering spirit. Here's my preferred one (although probably not
> within building code):
> - Convert all the servers to 12V power supplies, buy a bunch of car
> batteries, put the batteries in parallel and charge them via regular
> power supply, solar panel, wind mill, bicycle power (gives a new
> meaning to CPU cycles), jumper cables connected to a car in the
> parking lot, manually adding fresh batteries through the night, etc.
> Take advantage of the opportunity to make the servers more power
> efficient at the same time.
> If there's an electrical engineering at UMass Lowell (or near), I'm
> sure they'd love that kind of project.

Riding a bicyle for 10-12 hours at a time might be kinda fun, but our 
office floor is nasty enough with a thin film of battery acid.

> 
> 
> Of course there's also the solution to get backup MX servers from a
> friendly institution, but that's a political solution, not a technical
> one. :)

People seem ok with not getting email for a while.

It really bugs 'em when they can't read previous emails containing 
instructions for applying for a grant due the next morning @ 6:30am.

They also start to freak out about sending email...

We've had backup MX at our California data center ( a vserver at 
johnscompanies.com ) but we took it off.

We could run a backup MX at our Market Street data center (my apartment) 
but there are some time-consuming technical issues:

	1) We need to synchronize ldap to accept emails only for users
	   on our systems

	2) We need to have spam filtering at least as strong as
	   on the current primary to avoid being corn-holed by
            our friends in the canned meat industry.

FYI right now our California data center is just running csl-dns-02 and 
our market st data center is running remote-backup.thecsl.org and 
csl-dns-03.thecsl.org



> 
> OK, enough silliness for now, back to work.
> S.
> 
> 
> 
> [1] http://www.amazon.com/b/?node=201590011
>