By Sherif Samaan
In October 2008, I was given leadership of an ITS project that might at first have appeared straightforward. The assignment was to plan and implement a move of all the servers in the East Data Center—comprising the hardware on which many of the ITS-supported systems run—to the new NYU South Data Center. This modern, robust, much-needed facility, at that time still under construction in downtown Manhattan, was being built to accommodate the University’s growing requirements for IT resources.
A data center is a facility where an organization houses computer systems and servers, and the associated data storage, communications, and networking equipment. I thought back to the last data center move in which I was involved, about seven years earlier. Our task then was to move approximately 30 standalone servers and roughly 5TB of total disk storage. That was a relatively uncomplicated “forklift” move, for the most part, one in which systems were simply shut down over two weekends and moved as-is.
But things had changed—considerably—since that earlier time. Some 300 servers with over 500TB of total disk storage, many of them IBM Blade servers like those described in previous Connect magazine articles,1 would be moving to the new South Data Center, and would include ITS-supported research computing clusters, administrative systems, and NYU school and departmental systems.
and abroad.
A move of this scope would require extensive planning, preparation, testing, coordination, and communication with NYU business offices, schools, and departments. Beyond the sheer number of servers, there would be other new complexities and challenges. First, servers today are more modularized—that is, they consist of separate but interconnected sets of components, a feature that enables easier and more economical expansion, as well as the provision of “failover” capabilities to increase performance and reliability. Furthermore, disk storage today is more centralized, which provides greater flexibility in the deployment and manageability of the disks, as well as improved performance and stability. A single disk storage unit may contain data from tens of servers spread across a data center. In addition, the customized “firewall” (or security) zones 2 with which ITS protects NYU systems from external cyber attacks are far more complicated than before, and some services operate on several servers, and within several different security zones.
Moreover, both the growth in data center services to the University and the move to the new South Data Center would require the creation of additional “subnets” within NYU-NET’s network infrastructure.3 Over the previous three years, the number of servers housed in the East Data Center had grown 300 percent, nearly exhausting the subnets available to the data center. Clearly, the new data center would need brand new subnets to accommodate both the current IT needs and future growth. Unfortunately, this meant that the IP address of each and every server that was being moved would have to be changed, a time-consuming, complex task involving careful mapping and checking in order to keep intact the communications of server with server—and, in some cases, with outside services.
Indeed, all this had to be coordinated with every ITS group, and the more than 40 NYU departments whose ITS-managed servers were to be included in the move to the new South Data Center.
Planning the Move
The project team for the South Data Center move would include staff members from all ITS departments. The team knew that this move would not be nearly as straightforward as previous projects, and it would need to be managed very differently. Our task would involve employing additional technologies and resources to help make this move as seamless and risk-free as possible.
Interdependencies and IP addresses. The first important task for the team was to make sure we had a clear understanding of how all the databases and applications involved in the move interacted and depended on each other. Identifying all of the interdependencies would take several weeks of close collaboration among the various ITS groups and ITS’ departmental clients. Once accomplished, however, it enabled the team to come up with a plan for changing the IP addresses on all 300 servers. The planned changes took place over several months, with each set of changes entailing a great deal of coordination and care, so as not to disrupt any applications.
Moving data—carefully. While all this was happening, the team was also working on the daunting task of moving a massive amount of data from one data center to the next without the risk of data loss or disk damage. Fortunately, ITS had already been looking at an IBM virtual storage and replication product called the SAN Volume Controller (SVC)4 as part of its future storage strategy. This product would prove to be invaluable for the move to the South Data Center, especially for the enterprise administrative systems. Working with IBM, ITS also obtained storage devices to use as temporary “swing” storage during the move, in addition to using some devices already on hand that were near the end of their leases.
uninterrupted power supplies
Readying the facility. While my team was tackling the technical challenges of moving servers and data, ITS was also faced with the task of readying and optimizing the physical facility, so as to properly house and monitor all these systems. Air conditioning, uninterrupted power supplies, water pumps, racks, network cabling, electrical work, and physical security all had to be in place and working. In addition, ITS had to design and prepare a state-of-the-art command center where the “environmentals” (heating and cooling equipment), IT equipment, and applications could be monitored 24 hours a day, seven days a week.
Making the Move
The move was scheduled in five segments, beginning in October 2009 and running through the end of December. Dividing it up made the move more manageable, and made it possible to accommodate events and activities within the University schedule—student registration, for example, and annual end-of-year fundraising. On the evening of December 29, 2009, the last of the servers was safely at the South Data Center and on December 30, 2009, the ITS clients and service partners whose systems had been moved in that last segment confirmed that all systems were operating properly.
Many people whose plates were already full gave up their weekends and time with friends and family, and pulled together to make this project a success. I am very proud and honored to have participated in this project, and to have witnessed their efforts and dedication firsthand. The NYU South Data Center project was not only an exercise in proper project planning and technical expertise, it was also a five-star team effort—across ITS and NYU departments—in which everyone demonstrated genuine devotion to ITS and NYU.
Footnotes
- “The Benefits of Blade Servers & Virtualization,” by Sherif Samaan, Connect Fall/Winter 2006 and “Raised Floor Evolution: Transforming Departmental Server Support,” by Joanna Ratajczak, Connect Fall/Winter 2007
- http://en.wikipedia.org/wiki/
Firewall_(computing) - Subnets are a means by which computers are grouped within a larger network. Every computer connected to the Internet has a unique Internet Protocol (IP) address — a series of digits that is assigned to the computer and used for routing information to it. A subnet is a contiguous range of IP addresses that share direct connectivity among them. These days, most subnets are primarily created logically, i.e., through programming.
- www-03.ibm.com/systems/storage/software/virtualization/svc
About the Author
Sherif Samaan is the Director for Computing Facilities Services within ITS Communications & Computing Services.