For the Global IP Network at NTT Ltd., the global Network Operations Center (NOC) is at the very heart of its mission-critical operations, as a part of the business available at all times, 365 days a year, to support customers and the infrastructure of the tier-1 global IP backbone identified as AS2914. It can be seen as the glue that links the various network teams together.
The global NOC has a number of traits that set it apart from many other operations centers. One is that – although it comprises a team with different roles – network analysts, senior network analysts, and NOC engineers – all staff are expected at a fundamental level to have a hand in everything, both for customers and infrastructure.
This means they can deal with matters from basic to more complex issues across the Global IP Network, with the NOC responsible for monitoring and operating all layer 1, 2 and 3 nodes. These range from administrative requests and general questions to maintenance, outages, fault isolation, traffic engineering, failure tracking, triage, resolution, and documentation of problems.
This broad approach generates trust among customers and speed of service, as it means the operator the clients contact at the NOC in the first place by e-mail, the Global Customer Portal or phone should be able to aid them until their issue has been sorted out.
“If a customer contacts us, the expectation is that the operator will be able to assist them from start to finish,” says Casey Smith, Director of the NOC. “We’re the first line of defence against network faults and disruptions, and also the last line of defence for change management.”
While operations centers at some other global IP providers have moved more to a service desk model for routing customer issues, the Global IP Network has maintained its end-to-end approach, ensuring that it retains direct control over the network. This means the NOC acts as a one-stop shop rather than comprising siloed teams.
“Strictly speaking, our NOC is not a service desk, it is a fully-equipped network operations center that runs the network,” says K. Maruf, Vice President of IP Engineering Operations.
If one of the NOC staff cannot immediately resolve a customer’s problem alone or it requires specialist expertise, other experts are always at hand to assist. The efficiency of this set-up is shown by the fact that over a period of about a year, only around 1% of customer requests ended up having to be escalated outside the NOC.
The NOC also intersects with the operations of all the other teams at the Global IP Network, meaning staff have widespread awareness of what is happening throughout the network.
At the same time, performing this variety of tasks and getting to grips with the roles of different teams can be a big challenge for technicians new to the NOC. However, there are many team members with long tenures who can help by sharing their experience in this area, along with comprehensive training, regularly refreshed procedures, and documentation to get staff up to speed quickly.
Vice President, IP Engineering Operations • Global IP Network
Director, NOC • Global IP Network
Staff in the NOC need to be level-headed and flexible because the workload can vary significantly on a day-to-day basis. “It’s like a roller coaster – one day it’s heavy, one day it’s not,” says Smith. “I give credit to our technicians; you have to be cool, calm and collected.”
But though the workload fluctuates, a typical day might see an engineer spending time on three main tasks: reviewing customer cases – namely customer-lodged outages, failures, queries or concerns; taking care of other network incidents and capacity-related issues; and implementing network changes such as replacements, migrations, software upgrades, and circuit maintenance.
For Smith, one of the most rewarding parts of the job is hearing positive feedback from customers to the team’s ability to resolve issues in-house, giving them a high-quality experience. NOC staff also benefit from having a tight-knit relationship with other teams. “Those guys are amazing at helping us or adapting to what we need,” says Smith.
Shift to virtual
While the NOC continues to have a focal presence in the Dallas area of Texas, USA, the COVID-19 pandemic accelerated plans to enable remote work for the staff.
This remote working has had some clear benefits, including the ability to recruit from a wider talent pool. Since last year, several new engineers have been hired in other places and time zones around the globe while others have migrated outside of Texas.
The team uses an audio and video bridge and other communications systems that are open 24×7, keeping the traditional NOC “war room” presence and communication flow going.
It also means that the NOC has disaster-resilience mechanisms in place for the future.
Last year a new operations management system was deployed as well, to modernize the methods and modes of the NOC’s work and operations, resulting in a reduction of the mean time to resolution (MTTR) to 1/4 of the average under the previous system.
Combining new ways of working with its existing efficient methods should set the NOC up to continue as an even more flexible “one-stop shop” for customers going forward – something it is well known for in the industry.