Bedminster, N.J. –At first glance you think you’ve entered the war room in Stanley Kubrick’s 1964 classic film Dr. Strangelove, except it’s in colour.
The AT&T Global Network Operations Center (NOC) has 141 wall boards on display. These screens give the many network managers on the floor a 24/7 365 day end-to-end views of communications traffic around the world. “The Global NOC is mission control for AT&T,” said Steve Moser, network visitor program manager for AT&T, in Bedminster, N.J.These screens, which would dwarf any major sports book on the Las Vegas strip, shows voice and data traffic, undersea cabling, weather reports, news feeds, and even Twitter. In fact, AT&T network managers monitor Twitter for “AT&T”, “3G” and “iPhone” for complaints as way to improve customer service.
“People do not call any more to complain so we have to use social media tools,” Moser said.
Twitter used to have one of the more prominent spots on the wall board but AT&T had to take it down because of the vulgar language in some tweets.
The main emphasis of AT&T Global NOC is on prevention and consistency. “If there is no network; there’s no business and business relies upon us to deliver reliability,” Moser said.
In 2005, for example, when hurricane Katrina hit the gulf coast of Louisiana the amount of traffic acted as denial of service attack, even though it wasn’t. Moser said the Global NOC used dynamic allocation methods to keep the network running during a crucial time. The AT&T network managers basically monitored the flow of traffic and were able to keep the New Orleans area networks up and running by either re-routing traffic or by applying load balancing techniques in the surrounding areas of the gulf coast. If necessary, AT&T could have dispatched a network recovery team to the site of the emergency.
The Global NOC averages 20.8 petabytes of data per day on the network. That is the equivalent of about 42 kilometres of CDs stacked high. The network handles transactional data as well and this helps the network managers with data mining. The Katrina disaster is a rarity for AT&T, said Moser. Most of the time the network managers get a consistent view week after week and this is essential to businesses that use AT&T services.
By being predictable Moser said AT&T’s services become valuable to business and solution providers.
The size and scale of the Global NOC affords AT&T to be pre-emptive in the way it manages the network. Moser said that even the slightest variation in traffic can be adjusted and re-routed. “Our NOC has to be ahead of the curve so that it’s transparent to the customer,” he added.
About 80 per cent of work at the NOC goes un-noticed by people. Moser admitted that there are outages but on average there is a high level of reliability because of the pre-emptive approach.
The NOC services more than nine million devices and two billion text messages are processed per day at the centre. That is twice as much as voice messages, Moser said.
“We know about traffic because we monitor device usage and if we see a spike we can align that traffic for better flow,” he said.
As an example, at around 19:25 to 19:30 Central Standard Time on Wednesday the network manager’s knows there will be a spike in the east coast because of the Fox TV show American Idol. American Idol voting takes place around that time and it, similar to Katrina, would act as a denial of service attack.
But, the AT&T network managers know its coming and prepare for it.
Security is crucial to the operations of AT&T’s network and the NOC receives 64 billion alerts a month, which is more than any other network can handle, according to Moser.
The amount of cable necessary to wire AT&T’s NOC can shoot to the moon and back almost twice.