This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter’s approach.
Direct cable connections between data centre resources such as switches and servers and servers and storage was once common, and unfortunately the practice is surfacing again with the use of top of rack(ToR) and end of row (EoR) equipment mounting options. ToR and EoR equipment placement relies heavily on P2P cables, which can be problematic and costly if viewed as a replacement for standards-based structured cabling systems.
Consider the point to point (P2P) connections shown in Figure 1. Rack 1 depicts a ToR patching scenario between switch ports and servers without a structured cabling system, while Rack 2 uses similar server connections to a Rack 3 switch. While proponents of this approach tout a decrease in cabling as a cost offset, further examination may prove otherwise.
As the electronics in the various cabinets are reconfigured and reorganized, it may become hard to maintain neat P2P links and necessitate a move back to a structured system, negating any savings. What’s more, trying to retrofit a cable system in a live environment raises many questions about the chance for disturbing other systems and potential downtime.
Efficient port utilization can also be challenging. In a P2P design, switch ports are dedicated to servers within a particular cabinet. This can lead to an oversubscription of ports. Suppose Rack 1 needs only 26 server connections for the entire rack. If a 48-port switch (ToR switching) or 48-port blade (P2P server to switch) is dedicated to the cabinet, 22 unused ports are purchased and powered. A greater problem occurs when the full 48 ports are used. Adding even one new server will require the purchase of another 48 port switch. In this case, assuming two network connections for the new server, an oversubscription of 46 ports will be added to the cabinet.
Further, many of these P2P technologies offer limited channel lengths, ranging from 2-15m, limiting equipment location. With a structured cabling system, 10GBase-T is supported over 100 meters of category 6A, 7 and 7A cabling, allowing plenty of options for equipment placement.
Figure 1
The “any-to-all” alternative
The alternative is an any-to-all structured cabling system. The concept behind any-to-all is simple. Copper patch and fiber panels are installed in each cabinet that correspond to panels in a central patching area. This allows equipment to be installed and connected to any other piece of equipment via copper patch cords or fiber jumpers. The fixed portion of the channel remains unchanged, regardless of how electronics are shuffled between racks. Pathways and spaces are planned upfront to properly accommodate the cabling.
These channels are passive and carry no reoccurring maintenance costs. If planned properly, structured cabling systems will last 10-plus years, supporting two to three generations of electronics. The additional equipment needed for a P2P system will require replacement multiple times within a single structured cabling lifecycle. The equipment replacement costs, not including ongoing maintenance fees, will negate any upfront savings from using less cabling in a P2P system.
In Figure 2, we see an example of an any-to-all structured cabling system. Fiber connections all arrive in the central patching area in one location. This allows any piece of equipment requiring a fiber connection to be connected to any other fiber equipment port. For instance, if a cabinet has a switch that requires a fiber connection for a SAN on day one, but needs to be changed to a fiber switch connection at a later date, all that is required to connect the two ports is a fiber jumper change in the central patching area.
Figure 2
The same is true for copper. As with fiber, any copper port can be connected to any other copper port in the central patching area or within the zone. Switch ports are therefore not dedicated to cabinets that may not require them and active ports can be fully utilized. The ability to utilize all switch ports lowers the number of switches and power supplies and, therefore, decreases the networking investment.
In an any-to-all structured cabling scenario, the 48 ports dedicated to a single cabinet in a ToR design can now be divided to support any of several cabinets via the central patching area. Where autonomous LAN segments are required, VLANs or address segmentation can be used to block visibility to other segments. For example, in a data centre with 20 server cabinets, each housing 14 servers (based on U.S. average of 6kW per cabinet) and each requiring two network connections (560 total ports required), the port comparison is shown below:
Virtualization is being implemented in many data centres to decrease the number of server power supplies and to increase the operating efficiency (kW/bytes processed or IT Productivity per Embedded Watt IT-PEW) ratios within equipment. Increasing the number of power supplies (ToR) can negate virtualization savings. Further, as servers are retired, the number of needed switch ports decreases. In a ToR configuration, this can increase the number of oversubscribed ports. In an any-to-all scenario dark fiber or non-energized copper cables may exist, but these are passive, require no power, have no reoccurring maintenance/warranty costs and can be reused for other equipment in the future.
Figure 3
Every port contributes to the overall power requirements of a server. According to the U.S. Government Data centre Energy study from Public Law 109-431, approximately 50 per cent of data centre power consumption is power and cooling, 29 per cent is server consumption and only 5 per cent for networking equipment. From a networking standpoint, port consumption or power draw varies greatly between various architectures (i.e. SFP+, 10GBASE-T and Fiber). Many of these reported power statistics do not show entire consumption, but rather make a particular architecture sound attractive by only reporting power based on consumption of an individual port — exclusive of the rest of the switch and the higher power server network interface card at the other end of the channel. Cooling requirements are critical considerations.
Poor data centre equipment layout choices can cut usability by 50 per cent. Cooling requirements are often expressed as a function of power, but improper placement of equipment can wreak havoc on the best cooling plans. Point-to-point systems can land-lock equipment placement. The ability to place equipment where it makes most sense for power and cooling can save having to purchase additional PDU whips, and in some cases, supplemental or in row cooling for hot spots.
In P2P configurations, placement choices may be restricted to cabinets where open switch ports exist in order to avoid additional switch purchases rather than as part of the ecosystem decisions within the data centre. Hot spots can be reduced with an any-to-all structured cabling system by allowing equipment to be placed where it makes the most sense for power and cooling.
In conclusion, while there are several instances where P2P top-of-rack or end-of-row connections make sense, an overall study including total equipment cost, port utilization, maintenance and power cost over time should be undertaken including both facilities and networking to make the best overall decision.
Siemon is an industry leader specializing in the manufacture and innovation of high-quality, high-performance network cabling solutions. Headquartered in Connecticut, Siemon offers the most comprehensive suite of copper category 5e, category 6 (Class E), category 6A (Class EA) and category 7/7A (Class F/FA), and multimode and singlemode optical fiber cabling systems available.