This document is incomplete (there's 33% more in my head) and in very
raw form - e.g., the item are not "sorted" sensibly - but it's a start.
I believe that all the data we absolutely need at this point are present.
I'll send E-mail whenever there are significant updates...

- TLT


==============================================================================

GENERAL PARAMETERS / REQUIREMENTS
---------------------------------

General reliability level
  --> Power, security, environment, fire-suppression

Disaster recovery issues; single points of failure analysis, etc.

Insurance issues?

QUANTITIES PER SQUARE FOOT...

Bob Meyers:  TYPICAL numbers (new UNM Architecure building conforms)...
  . Mechanical:  $45/feet^2
  . Electrical:  $20/feet^2

Dan Dion:
  . 50-200, the blade-center being close to the latter.
  . 1400 ft^2 / $400K =  $285/ft^2... "on the high side"

Lighting "figure of merit" from PPD:  "2 watts/feet^2"

From the Liebert white paper:

  Under-floor A.C., as a sole method of cooling, is expected to become
  problematic at the 100 W/ft^2 level!


PHYSICAL SECURITY ISSUES
------------------------

--> Insurance implications

FERPA reqiured
HIPPA required
Sarbanes-Oxley [financial] desired? -- http://www.sarbanes-oxley.com/

HIPPA requires access control logs, correlated with system logs.
Swipe card system?
Cypherlocks?  (Apparently, standard (UNM?) cypherlocks are NOT HIPPA-complient.)

Cameras?
Motion sensors?

...coupled to Police and/or monitoring firm?

Motion sensors that VETO door-based alarm sensors, so you can
hold a door open for machine movement.

Appropriate TRAINING (e.g., fire response, crash-button avoidance(!), etc.

Speakers to amplify alarms.


ELEVATION-VIEW ISSUES
---------------------

Floor-tile to ceiling-tile space may need to accomodate
advanced air-movement technologies, such as Liebert
XV components.

Crane access requirement?  (Probably not.)


FLOOR
-----

Floor must support...

Dan Dion:  subfloor:  concrete slab-on-grade should support 300 lbs/ft^2;
cheapest office buildling's rate at 50 lbs/ft^2.

Floor height:  18 inches?

Old-standard floor tiles:  1000 lbs/in^2 "point load concentrated" weight limit;
nowadays: 1250;  available:  1500, 2000, 3000, 3500...
--> Based on NCAR's experience with row after row of fully-loaded
    IBM p650 Rigattas, I'm guessing we desire 2000 tiles.

Extra pedestals:  O($10) per.

AIR FLOW ISSUES seem to suggest that we require CONCRETEd floor tiles.

Red Storm experience:
 . Under-floor pressure:  CRAY wanted 20 lbs/feet^2 == 3.5 inches of water
 . 3000 CFM ==> concrete floor tiles.

STATIC-elimination flooring...?
--> This would probably eliminate the following...

Lucite (?; translucent) floor tiles, where loading is not
at the time an issue.


CEILING
-------

PPD had quoted "R-38 roof insulation"

P.H.:  data-center quality acoustical tiles:  vinyl back, painted...
no flaking, dropped the room temp 2 degrees!


ELECTRICAL POWER ISSUES
-----------------------

Barbie dolls ==> 40 to 50+ racks...

PNM:  present transformer capacity is 300 KVA.

# racks  avg KW/rack    KW   Tons AC (75% eff)  "A.C. AC" (EER=10)  Total KW
-------  -----------   ----  -----------------  ------------------  --------
  40           5        200         76                 20              220
  50           5        250         95                 25              275
  60           5        300        114                 30              330
  40          10        400        152                 40              440
  50          10        500        190                 50              550
  60          10        600        228                 60              660
  40          20        800        304                 80              880
  40          27       1080        410                108             1188
  50          20       1000        380                100             1100
  42          27       1134        430                113             1247

Reliability factor... "east/west" power (a PPD term?)

Power distribution panel with per-phase breakers, a la CIRT,
or mounted IN THE WALLS?

Crash button(s)

Power conditioner / UPS / (natural gas) generator ?
--> What about the MAINTENANCE COSTS?

NOTE:  The batteries in UPS systems induce significantly different
temperature and humidity requirements.  One may NOT want to put
the UPS in the computer room proper, but one would then have cool
it separately, etc.  Unlike computer rooms, which are vapor-barrier-
sealed enclosures, rooms for UPS systems could be fed by comfort-grade
A/C units.
 
Grounding scheme very important...! [see PSC notes]

Need to expand existing system?
  Capacity of existing transformer:  300 KVA
  Capacity of existing switchboard:  250 KVA  (~= 1200 A * 208 V)

Under-floor power distribution

Symmetra model power conditioner / UPS, from APC corp.


TAPE LIBRARY ISSUES
-------------------

Slightly lower humidity requirements, relative to computing machine.

Don't want cold air blowing directly on tape units.


AIR CONDITIONING / HEATING ISSUES / ENVIRONMENTAL ISSUE
-------------------------------------------------------

--> See "general parameters", above.

What is occupancy level?
--> To what extent does fresh air get mixed in?
    ==> Humidity control, dust control issues...

Dust control requirements -- what level of cleanliness?


*** Liebert White Paper:  under-floor A.C., as a sole method of cooling,
    is expected to become problematic at the 100 W/ft^2 level!

Think over-capacity and spreading cooling load over multiple systems
--> Backups, in event of failures.  (But units typically already have
    multiple compressors, etc. built in.)

Assume a 70% "efficiency factor" for AC
==> ??? w.r.t. EEG / EEC??

("Diversity factor":  "90%" means the spec is quite accurate.)

NOTE!  20-25% lower [cooling efficiency](?) Albuquerque's altitude!

Quote from Liebert web page: "New server designs are projected to
increase data center power consumption up to 300 watts/ft2."

!!! Staged air conditioning deployment...?

CHILLED WATER:  per-machine, or equip the room??

  Chilled water rate approx. equivalent to "ordinary" machine-room A.C.?
  --> Machines using water cooling:  IBM 9021, 3090; CRAY MTA-2...
  --> CSI knows how to do chilled water

  From Red Storm tour:  for water cooling, need to think about...
   . multiple levels of heat-exchangers; local loops, etc.
   . chemically conditioned water (e.g., heat-exchange properties...
      e.g., ethylene glycol mix;  anti-corrosives;  anti-algae;  etc.)
   . water sensors under the floor
   . DRAINS under the floor
  ==> Starting to sound seriously expensive

  More water considerations (Dan Dion):
    Modern thinking:  1-inch radius pipes (surrounded by 3 inches, radially,
    of insulation) in trenches, as headers to flexible hoses (which are no
    longer taboo).  One has to worry about:
     . where the stands (to support the raised floor) land, relative to where
        such trenches would go;
     . interruption of air flow if you can't put the 8-inch pipe diameter
        into a trench.

Rack-based air movement/cooling supplementation?
  Rack door-based
  Rack-roof-mounted

Ceiling based air movement/cooling supplementation?

--> These may mitigate need for a.c. acoustic oscillation issues!

Airflow issues:

Dan Dion recommends 400 CFM/ton; 400 CFM == 1 fully perforated floor tile
==> 1 fully perforated floor tile / ton of A.C.

==> For one JS20 cabinet (4.3 tons):  4.3 floor tiles
Cf... IBM-specified ventillated floors tiles per cabinet, minimum (JS20):
                                      2.5 (=5/2)

==> (?) Rough #:  4 ventillated floors tiles per cabinet

The CRAY at Sandia is getting 3000 CFM per cabinet...
==> 750 CFM per ventillated floors tile (assuming 4 per cabinet)
...requiring CONCRETEd floor tiles.

Sandia Red Storm:
  3000 feet^3/minute (@ 50 deg. F) for
  "2800 BTUs/hour/cabinet" == 0.8 KW, 0.23 tons (no eff. factor)
                              ==> 800 Watts / 96 CPUs ==> 8.5 W/cpu!??
  Dan's # would imply 3000/400= 7.5 tons/cabinet == 26.4 KW (no eff. factor)
                                                    ==> 274 watts/CPU (~= 32*8.5)
  ==> Tossing in a factor of 2... I bet Len meant BTUs/MINUTE/cabinet!!
      (Also:  7.5 tons * 96 cabinets = 720 tons)
      ==> 168K BTUs/hour/cabinet  (cf. a home unit: 10K...)

(...cf. a home:  5 tons for 2000 ft^2, at 10K BTU/hr (0.8 tons, no eff.) / unit.)


FIRE SAFETY / FIRE SUPPRESSION
------------------------------

Water-based OK; human-dangerous (O2 deficiancy) chemicals OK.
Human-safe non-water chemicals too expensive.

2-hour fire-rated walls; 1-hour ceiling ...?

Supression options:
 . Water
 . Halon
 . SM200 / FM200 (?)
 . FE227
 . What else?

Caution:  some systems are much more expensive than others to REFILL!

Insurance implications...?


WATER ISSUES
------------

--> See CHILLED WATER, under AIR CONDITIONING, etc.

There are water pipes running over the location of the machine room!

There are roof leak issues!
==> Build "subroof" on machine room?

May need water sensors in ceiling (if AC there) and/or
floor (if chiller systems installed.)


INTERLOCKED MONITORING
----------------------

Fire detection / supression system
Environmental systems (air, heat, etc.)
Physical security

...coupled to Police and/or monitoring firm?

Dan Dion mentioned:
"Data Trax", an SNMP-based application giving network-accessible
environmental control


OPERATOR AREA / STAGING AREA / MEETING ROOM / WHEELCHAIR ACCESS...
---------------------------------------------------------------

Ramps (possibly temporary) versus lifts.
--> Think about rack-tilt limitations -- a safety issue.

I'm told it is difficult to repair lifts when they break;
parts, etc.

Don't forget shelving!


WALLS
-----

Glass walls...  facing which way...?  Most of the way around?
(Heavy curtains for noise isolation?)
--> Sound issues?

Dan Dion:  vapor-sealed walls?

PPD had quoted "R-11 wall insulation."


DOORS
-----

--> ADA requirements?

8 foot access door required.

...


ACOUSTICAL / SOUND / NOISE ABATEMENT / VIBRATION ISSUES
-------------------------------------------------------

Noise on the floor, at head level, depends on where the air is
flowing...

Red Storm racks have SOLID DOORS (quieter) because air is moving
vertically through the rack.

--> See sections on ceiling and walls...


ADA ISSUES
----------

...(need data from experts)...


AESTHETICS
----------

See walls (glass)...

Lighting...

Lucite (?; translucent) floor tiles, where loading is not
at the time an issue.


LIGHTING
--------

Avant guard-style (incandescent?) track/spot lighting
(or florescent minimize heating?)

Typical lighting systems are not uniform across cabinates, from TOP TO BOTTOM.

Consider lights mounted INSIDE cabinates?

EMERGENCY LIGHTING?  (Not needed if regular lighting is on generator.)

DON'T WANT TO BE CARRYING AROUND A FLASHLIGHT!

"Figure of merit" from PPD:  2 watts/feet^2

Lumens, etc...?

Code restrictions?  ADA requirements??


NETWORK-RELATED
---------------

P.H.:  PLAN YOUR INFRASTRUCTURE... think ahead as much as possible about
outlets, routes, etc.

What is needed?

Don't forget the conduits!

Dial-in modems?
--> See 'phone lines' under 'telecommunications'...

Conduit for fiber cables, coming in under floor...
from which direction(s)?

Cable-length issues...

Keep cables short and linear

Wireless coverage?


TELECOMMUNICATIONS
------------------

Telephones - need some analog circuits for guarenteed operation
in an emergency power outtage; e.g., to try to communicate remotely
with a UPS, alarm, generator systems, etc.

Intercoms?
Loud speakers?


CABLING ISSUES
--------------

Overhead and/or under-floor cable trays...

If overhead, could they be "attractive" somehow...?  Wood-look!?  (Probably
a massive firecode issue.)


==============================================================================

EXAMPLE MACHINE PARAMETERS
--------------------------

Approximate / absolute minimum required inter-row spacing:

  IBM JS20:   Front 36"
              Rear  30"
  SGI ALTIX:  ???
  CRAY XT3:   36"
  CRAY XD1:   Front 53"
              Rear  36"

Weight per cabinet:

  IBM JS20:   ~1386 lbs. -- NetBAY42 Enterprise racks, max loaded:
                                                        2045 lbs. (928 kg)
  SGI ALTIX:  1547 lbs.
  CRAY XT3:   1529 lbs. (694 kg)
  CRAY XD1:   1800 lbs. (816 kg) 

Dan Dion:  IBM 3584 LT0 tape library unit, maxed out:  1000 lbs


More weight comments, quoting IBM's Dan Dion...

Floor Loading, using IBM Corporate Standard C-S 1-3705-001:

The weight distribution area is the area around the machine, starting
from the frame line, caused by the need to distribute weight beyond
the machine.  This area may not extend beyond 30 in. from the machine.

Given that service clearance areas can overlap but weight distribution
areas can not, when two machines are installed next to each other,
only half the area between the machines can be used for weight
distribution for either machine.

If there are any concerns about sub floor strength, consult a building
engineer. The distributed load depends upon equipment layout and
clearance around the equipment.  Calculated distributed floor load for
128 nodes; 118 to 151 Lb/SqFt.

Placing extra sub floor support pedestals under the raised floor tiles
are recommended.  If the two 126 node frames are placed side-by-side so
the frames share a common tile (two casters on one tile) the
concentrated load on that tile calculates to 1302 Lb which exceeds the
strength of common 1000 Lb and 1250 Lb floor tiles.  The common
practice is to add extra sub floor pedestals to strengthen tiles.


Temperature and humidity ranges:

  IBM JS20 recommended operating values:
    ambient temp:          70-74 F
    ambient rel humid:     40-50%

  SGI ALTIX recommended operating values:
    ambient temp:          5-30 C
    ambient rel humid:     10-90% noncondensing

  CRAY XT3 and XD1 recommended operating values:
    ambient temp:          64-77 F (18-25 C)
    under-floor temp:      50-60 F (10-16 C);  nominal: 55 F
    ambient rel humid:     30-50% non-condensing
    under-floor rel humid: 60-90% non-condensing

Ventillated floors tiles per cabinet, minimum:

  IBM JS20:    2.5 (=5/2)

Cabinet areas (ft^2):

  IBM JS20:    26" x 44" ==> 7.9
  SGI ALTIX:   24" x 41" ==> 6.8
  CRAY XT3:    23" x 57" ==> 9.1
  CRAY XD1:    22" x 36" ==> 5.5

Power requirement per cabinet, maximum:

  IBM JS20:         26.1 kVA / 25.5 kW
  SGI ALTIX350:                 7.2 kW  (sep. TP9300 disk cabinet:     2.2 kW)
  SGI ALTIX3700BX2:            12.4 kW  (sep. TP9300 disk cabinet:     2.2 kW)
  CRAY XT3:         14.8 kVA / 14.5 kW  (sep. disk cabinet:  6.5 kVA / 6.2 kW)
  CRAY XD1:         30.0 kVA / 27.0 kW

    NOTES:  JS20:  4.353 KVA & 4.343 KW / chassis * 6 chassis/cabinet
                   ==> 25.5 KVA & 26.1 KW / cabinet

Power (watts) per ft^2:   (???????)

  IBM JS20:    3228
  SGI ALTIX:   1824
  CRAY XT3:    1593
  CRAY XD1:    4909

Typical required power hold-up time:  16 ms

Per-disk power requirement estimates:
  Intel SPKA4 Server Platform: 14 watts/drive, maximum.
  SGI TP9300S Storage array:  250 V * 16 A = 4 kW for 154 drives
  ==> 26 watts/drive

Heat dissipation to air, per cabinet, maximum (no eff. factor in BTU/hr -> tons):

  IBM JS20:    51.03 Kbtu/hr (4.3 tons)
  SGI ALTIX:  
  CRAY XT3:    49.47 Kbtu/hr (4.1 tons)  (sep. disk cab: 21.02 Kbtu/hr (1.8 tons))
  CRAY XD1:    92.12 Kbtu/hr (7.7 tons)

    NOTES:  JS20:  (70000 BTU/hr\) / (96 nodes) ~= 729 BTU/hr/node
                   5*14*729= 51030 (BTU/hour)/(6 chassis == 1 cabinet)

Airflow rate requirements, per cabinet:

  IBM JS20:    2100 to 3000 CFM
  SGI ALTIX:  
  CRAY XT3:    3000 cfm (1.41 m3/s) (bottom-to-top flow;  disks: front-to-rear)
  CRAY XD1:    3600 cfm (1.70 m3/s) (front-to-rear)

Acoustical issues:

  IBM JS20:   
  SGI ALTIX:  
  CRAY XT3:   75 dba at 3.3 ft (1.0 m)
  CRAY XD1:   80 dba at 3.3 ft (1.0 m)