Inside NYU IT’s Global Command Center

The tools NYU uses to monitor IT locations and resources worldwide

By Joanna Chmarzewska
A few months ago, as the Chinese New Year began, NYU IT’s Global Command Center located in lower Manhattan started 24×7 monitoring and escalation support for most of the Shanghai Data Center’s environmental equipment (air conditioning, water leaks, etc.). With the Shanghai team out celebrating, this was a significant test of the Command Center’s global monitoring/support capabilities and a testament to the great collaborative effort required to make that happen.  

This is just one of a number of recent advancements at the NYU IT Global Command Center. Over the last few years, it has become a crucial piece of NYU’s monitoring and tracking of IT infrastructure and service-related issues, not only for New York City, but at many NYU locations. As of last year, the Abu Dhabi Data Center and systems were added, allowing our peers in Abu Dhabi to sleep peacefully while the Center keeps an eye on their operations. And currently, the Center is in the process of onboarding the monitoring of the Syracuse High Availability site (an emergency backup location for many of NYU’s crucial software and IT applications) and switch closets. With the addition of Syracuse, the Global Command Center will be monitoring a total of six data centers.

The monitoring of data center environmental conditions is only one part of a broad spectrum of tasks performed by the Global Command Center. It is also responsible for monitoring UPS/power, physical security, mechanical equipment, all mission-critical administrative and academic systems, data storage, network and connectivity, the processing and scheduling of batch jobs, as well as tape vaulting operations. All of this is made possible by the use of a broad range of monitoring tools: Nagios, ManageEngine, and BMS, to name just a few.

The Global Command Center has staff dedicated to using these systems to perform the aforementioned tasks, as well as experts from several different technology groups dedicated to monitoring and supporting NYU’s technology environment. They include system administrators, network engineers, and data center management staff, all co-located in the Command Center. That means that staff members can communicate across teams and collaborate easily on a service issue and, to a certain degree, look at the same screen of information to track the resolution of the problem. Issues are identified and resolved faster and more efficiently, enabling the prevention of service degradation or faster service restoration.

At work in the global command center

Photo: Moises Valencia

For most technology and network users at NYU, the experience and performance of the operation they’re trying to complete is of most importance to them, whether it’s a simple search on the web or running a complex report. Because providing a great customer experience matters most to the Global Command Center, it has a comprehensive focus on how the application and its supporting infrastructure — servers, databases, network, etc. — work together and how even a minimal disruption of the services can impact the NYU community.

The Global Command Center is striving to become not only technology-driven but also more collaborative, communicative, and responsive to the University’s needs. That means looking at IT from the customer’s point of view. This has led to improved internal processes and procedures, and adoption of many tools and technologies already used by other areas at NYU – like ServiceLink, Control-M, Coradiant, and Citrix – to standardize across the organization. The Center work very closely with the NYU IT Service Desk and can act as its backup (and vice versa) if the need arises.

To keep up with the ever-growing demand on services, some improvements to the layout of the Command Center were recently made, including upgrading to more ergonomic furniture and the installation of “thin client” workstations (where needed applications get much of their power from a shared remote server rather than being fully installed on a personal computer) with dual monitors, using Citrix technology (the same technology that underlies NYU’s Virtual Computer Lab and enables the use of software across networks). This virtual desktop technology allows staff at the Global Command Center to work from anywhere using the same desktop with all the tools already installed. Most importantly, the Center recently installed a new video wall comprising eight 55-inch display screens that allows the display of various monitoring tools. All these improvements have enhanced efficiency and collaboration, and make working in the Command Center more enjoyable – especially when you add the beautiful 12th floor view of the Empire State Building and Manhattan.

DSC_0491

Photo: Moises Valencia

NYU IT’s Global Command Center is constantly researching new technologies and reviewing the process of aligning tools and procedures to fully achieve what is widely known in the industry as a “Global NOC (Network Operations Center).” Global Command Center manager Joanna Chmarzewska describes her vision for the Center as “one solution to monitor it all, for the data center, network, systems, and application monitoring…anytime, anywhere.” Currently, the Global Command Center employs many individual monitoring tools to keep tabs on the health of NYU IT’s infrastructure. This year, as part of its strategic roadmap, the Center hopes to launch a single comprehensive tool that would capture user experience during an incident and trace that experience to an underlying problem, all the way through the backend, including everything in between—whether servers, databases, or third-party API calls—and pinpoint the source of the issues in real time. A single tool would equip the Global Command Center team and all supporting teams with the means to quickly identify, escalate, and resolve issues and perhaps act quickly enough to resolve them before they impact the NYU community.

The challenges of this initiative are considerable, particularly in a complex and rapidly changing environment. But, Chmarzewska says they are up to the challenge and looking forward to upcoming undertakings.

About the author: Joanna Chmarzewska is the Manager, Systems Administration. She manages the Global Command Center and File Transfer groups and has worked at NYU since 1997.