Data Center

Horizon Blue Cross Blue Shield of New Jersey (Horizon BCBSNJ) is a not-for-profit organization headquartered in Newark, New Jersey and is the state's largest health insurer. The company's goal is the same as it has been since 1932: to provide individuals and employers in New Jersey with convenient access to quality healthcare, wherever they live or work.

Horizon BCBSNJ has two data centers: one in Newark, New Jersey and the other in Monmouth County, New Jersey. Both are connected via multiple data links and are meant to back up each other in case of disaster. The data centers are a clustered environment that has a primary node in Newark and passive node in Monmouth and vice versa helping ensure both data centers are up and live at any given point of time to run applications between the mainframe environment and the WinTel environment. Both data centers have Cisco Catalyst® 6513 and 6509-E Switches in an end-of-row (EOR) architecture, where server connectivity is used as well as aggregation and connections back to the core. Horizon BCBSNJ uses blades and stand-alone servers, as well as VMware servers. The company also uses Cisco® Catalyst 6500 Series Network Analysis Modules (NAM) for unparalleled visibility into how Horizon BCBSNJ's network is performing and how users experience the delivery of applications and services to understand and improve application performance.

According to Martin Hansen, data network manager, AT&T Global Managed Services consultant at Horizon BCBSNJ, the company has a textbook architecture consisting of core, distribution, and access layers, but because of the effect of spanning trees on the network, Hansen began evaluating a data center upgrade.

"The main reason that we began looking at upgrading our network was because of our blade enclosures and the variety of HP switches that we have little control over, which are managed by different teams within Horizon BCBSNJ," says Hansen. "There would be what you could call `incidents' on the network, where the switch configurations did not match or something was wrong with the blade enclosure hardware or software, which caused a great deal of spanning tree issues. We knew that there was no way to completely eliminate the spanning tree, but we wanted a solution that would at least minimize the effect of it."

Vijay Bhaskaran, AT&T network engineering consultant at Horizon Blue Cross Blue Shield of New Jersey says that, in addition to the spanning tree issues, he also wanted to utilize virtualization at the switch level and the ability to interconnect the two data centers.

"One major feature or benefit that we wanted in an upgrade was the ability to utilize virtualization more than we had been able to use it in the past," says Bhaskaran. "We were looking for a solution that allowed us to span VLANs across the two data center to support Microsoft Server Clustering implementation, utilize all of the uplinks between the data centers, and at the same time minimize the risk of network outage inherit with spanning layer 2 VLANs

across two geographically separated data center."

Bhaskaran and the consulting team decided to conduct a 60-day test of the Cisco Catalyst 6500 Virtual Switching System (VSS) 1440 in a lab.

"We had already been running Catalyst 6513 and 6509 Ethernet switches in the data center and felt confident in trying out the Cisco Catalyst 6500 Virtual Switching System (VSS) to see if it would help reduce the spanning tree instances in our network," says Bhaskaran. "Setting up the VSS test was very easy, and the documentation on Cisco.com was comprehensive and straightforward. We used a couple of documents for configuring the lab, ran the test for two months, and found that VSS definitely addressed the issue by reducing the number of spanning tree instances, hence minimizing outages caused by spanning tree convergences."

At the end of the testing period, Bhaskaran selected four Cisco VSS's available on the Cisco Catalyst 6500 Series Supervisor Engine 720 with 10GE uplinks for redundant, scalable, and secure Layer 2 extension for multisite data center intercon¬nect optimized for Cisco Catalys 6500 Series Switches.

"After the test we engaged in the planning phase, trying to determine how to introduce these new switches into the environment without creating too much downtime on the customer side," says Bhaskaran. "During the deployment, when we were actually phasing out the old distribution system and deploying the VSS solution, our system was down for only three to four minutes. Our deployment went according to plan and was as smooth as we could have hoped it to be. We now have two VSS clusters in our server farm distributions, which include two 6500s in Monmouth and two in Newark."

Bhaskaran says the Catalyst 6500 Series NAM module that Horizon BCBSNJ already had in place provided the IT team unsurpassed performance to thoroughly monitor the Horizon network.

"We already had the NAM modules deployed in distribution where we were able to view traffic traversing the data center," says Bhaskaran. "Now that we have VSS, we are able to use one NAM in each of the VSS chassis, VSS-1 and VSS-2. The NAM's give us reports that enable us to accurately troubleshoot any issues on a Horizon report."

Ease of migration was another benefit of the upgrade to the VSS switches, according to Bhaskaran.

"We had two sets of switches for our Newark and Monmouth data centers, and initially enabled the VSS switches before configuring them as if they were already in our distribution," says Bhaskaran. "The layer 3 interfaces were running a lower HSRP priority in comparison to the existing interfaces. Then we connected the VSS's to the existing switches over a layer 2 link, and once they were up and running and connected to the existing distribution, we slowly migrated each access switch one link at a time to the VSS switches. Once all the access switches had been

connected to the VSS switches via multi-chassis Ether channels, we decommissioned the old switches."

Bhaskaran says that minimizing the spanning tree footprint and increasing network uptime are the key benefits that he and the IT team have achieved through the upgrade.

"One goal for the upgrade was to eliminate or at least minimize the effect of the spanning tree convergence on the network, and to allow for as much network uptime as possible," says Bhaskaran. "The VSS deployment gave us redundant, scalable, and secure Layer 2 extensions for our multi-site data center intercon¬nect while sharply decreasing network events and helping to create a much more productive experience for our users that has helped us achieve our business justification for the upgrade."

The ability to use all of the uplinks within VSS switches was another major benefit of the upgrade, according to Bhaskaran.

"Before the upgrade, with spanning tree stipulations, our implementation had been that some uplink ports were blocked," says Bhaskaran. "Since deploying VSS switches, we've achieved approximately a four to fivefold increase in bandwidth and we're utilizing all the links within VSS. We now can have an Active/Active arrangement on our uplinks between access and distribution layers, so essentially we get what we pay for rather than using just one active and one backup. Different IT groups within Horizon BCBSNJ are also seeing marked improvement in performance."

For more information on Cisco Catalyst 6500 Virtual Switching System available on the Cisco Catalyst 6500 Series Supervisor Engine 720 with 10GE uplinks, please visit http://www.cisco.com/go/vss.