Cisco on Cisco
Application-Oriented Networking Case Study: How Cisco IT Uses AON to Simplify Application Development and Maintenance
Cisco AON accelerates application development and reduces costs.
To improve employee performance and service levels, Cisco Systems® automates its processes by developing custom work applications. This is a major endeavor: application development and enhancement consume the efforts of approximately 900 full-time software engineers. As of 2006, the company monitored more than 10,000 applications, in ten application development environments.Cisco actively seeks technologies that reduce application development costs, accelerate time to market, and allow the IT group to do more with the same number of people.
Cisco IT's approach to application development is to assign developers to one of two groups. One group interfaces with a Cisco business unit-Human Resources (HR), sales, engineering, and others-to understand the business requirements, while the other group works in a centralized development center to develop repeatable application functions, such as digital signature verification, Secure Sockets Layer (SSL) termination, certificate validation, and service versioning. The second group faces an arduous task because of the complexity of the Cisco application environment. "Historically, we have had to multiply the resources required to develop a new capability, such as SSL, by the number of environments, server technologies, and languages in use at Cisco," says Hicham Tout, IT architect for platform services. Cisco environments include the external Website (www.cisco.com), intranet, and manufacturing extranet. Server technologies include WebSphere, Borland Application Server, and Microsoft SQL Server; and languages include Java, C, and Perl. "When developers wanted to add SSL capabilities to an application, for example, they first had to learn how the technology is implemented on every platform used," says Khaldoun Rayes, Cisco IT engineer for platform services. Similarly, service versioning had to be deployed separately in each of the dozens of application versions that Cisco introduces each quarter.
To avoid the need to repeat development work for multiple environments and languages, Cisco IT wanted to implement common application functions in the network. Goals were to accelerate time to market, give developers more time to focus on the business problem, and avoid the risk of incorrect implementation for security functions. "We wanted to standardize selected capabilities and make them independent of the technology or environment," says Tout. Moving common application functions to the network created a compelling business case. An internal survey revealed that moving a core group of functions to the network would enable Cisco to retire more than 100,000 lines of code. This, in turn, would reduce maintenance and free up hardware and memory on application servers.
An important requirement for the solution would be reducing the number of layers required for a message to move through the network: Web layer, application layer, messaging layer, transformation and mapping layer, adapter layer, and back-end. Simplifying this architecture would improve application performance and significantly reduce the costs of development, maintenance, and licensing. "In the past, we had to add a new service at each layer," says Sandeep Puri, IT engineer. "If all applications used a set of centralized services, we could introduce the services just once, at one layer, accelerating service introduction."
A related challenge for Cisco was reducing the high capital and operational expense of managing multiple gateways, including business-to-business (B2B), Web Services, and Electronic Data Interchange (EDI).
Cisco IT is simplifying application development-while also reducing development, capital, and operational costs-by offloading certain application functions to the application-oriented network (AON) module for Cisco switches and routers. AON is part of Cisco's vision for the Intelligent Information Network (IIN), which enables new levels of integration, collaboration, and productivity.
Deployed on switches or routers, Cisco application and service support utilities ordinarily require custom code. The utilities enable client devices to communicate with applications, and applications to communicate with other applications, performing functions related to reliability, manageability, and targeted service (Table 1). The major advantages of AON are eliminating the need to write and test code for multiple environments, standardizing how functions such as SSL termination are performed, and reducing processing and memory costs for application servers by making the applications smaller. "When AON provides application utilities, the developer still needs to understand the business logic, but not necessarily the nuances of implementing services such as digital signature verification in each application development environment," says Rich Gore, IT manager. A related benefit is that Cisco's network-security experts do not have to repeatedly review new code written to perform the same security function that already exists in other applications. Instead, a function can be developed once, implemented once in AON, and reused over and over.
|Table 1. Current AON Functions|
|Transport-level encryption termination (SSL v3)
Payload encryption termination (Extensible Markup Language, XML) HTTP-to-JMS protocol translation
HTTP-to-JMS protocol translation
Digital signatures, for strong authentication
Demilitarized zone-to-application layer secure connector, such as Secure Shell Protocol or spanning-tree algorithm
|Reliable delivery||Message and transaction-level logging
|Service versioning (allows multiple versions of single service to run simultaneously)
Message/content-based routing (routes messages based on contents and/or business rules) Transformation and mapping (transforms incoming XML to the format that the destination expects)
Acting as an invisible message router in the network, AON enables Cisco developers to authenticate, authorize, log, validate, transform, and map messages between different endpoints in A2A (application-to-application) and B2B communications. "In the old way of doing business, these capabilities were embedded in endpoints and had to be redeployed for each platform and language in use on the Cisco global network," says Tout. "By embedding those capabilities into the network, we are simplifying the application environment and reducing costs."
Figure 1. How AON Consolidates Application and Service Support Utilities
How is AON invisible? An AON node resides in the Cisco network as an inline application-aware device (Figure 1). It acts as an intelligent intermediary gateway that can either be explicitly addressed by applications or simply act as a pass-through. To make an application take advantage of prewritten AON functions, the developer simply inserts a short line of standard Extensible Markup Language (XML) code to request that the inline AON module perform that function for the application flow before it arrives at the server.
AON acts as a service broker in services-oriented architectures (SOAs), as an integration broker in application integration, and as a security integrator. It is not a general-purpose application server or Web Services orchestration engine.
Cisco IT decided to use the AON module instead of an application server with similar functionality to avoid the expense of server administration-particularly in branch offices, which typically do not have system administrators. "Designing AON as a module for a network switch or router that already has the I/O capacity also makes it easier to scale," says Patrick Andersson, IT manager, who notes that Cisco has successfully implemented application processing on a module with products such as Cisco Firewall Security Module (FWSM) and Cisco SSL Switching Module.
The AON solution that Cisco uses comprises three components: AON hardware modules, AON Development Studio software, and the AON Management console:
Figure 2. AON Module for Cisco 6500 Series Routers
AON module - Cisco Catalyst® 6500 Series AON Modules reside in several Cisco 6509 Routers located in the San Jose data center (Figure 2). Other companies might choose to deploy AON in smaller Cisco routers, or in branch offices instead of main data centers. Depending on company size and application traffic volume, companies can use all services on a single AON module or create clusters of AON modules devoted to a subset of services. Cisco chose the latter option. One cluster performs digital signature verification, another performs logging, and so on. If one Cisco Catalyst Switch becomes unavailable, the others in the cluster continue to perform the services. "We simply add more modules to keep pace with demand," says Puri. "We can scale to several hundred messages per second."
AON Development Studio - Cisco developers use AON Development Studio, an Integrated Development Environment (IDE), to develop a flow, or execution plan. "In a simple flow, AON might first verify a signature and then decrypt a message," says Rayes (Figure 3). After defining the flow in AON Development Studio, the developer uses the AON Management Console to instruct the AON module to begin applying the flow.
AON Management Console - The Cisco IT infrastructure team uses the management console for inter-application configuration (Figure 4). The Cisco IT networking team uses it to deploy network-specific configuration. Developers use it to apply the flow of application functions defined in AON Development Studio.
Figure 3. AON Development Studio
AON is analogous to a library of functions in a programming language such as C++, using standard XML software calls to invoke functions from within the application. Cisco developers no longer need to code these functions in every application and for each application environment, a time-consuming process that increases the risk of incorrect implementation. Instead, AON applies a validated function to each relevant application flow as application messages pass through the Cisco switch or router containing the AON module. "AON does not do anything that you could not do in code; it just does it in a smarter way-in the path of the transaction," says Amit Srivastava, IT project manager.
To call AON application utilities and services from their applications, Cisco developers use AON Development Studio to intercept any type of application traffic, including XML and Simple Object Access Protocol (SOAP). Depending on the application requirements, Cisco IT can configure AON in one of two modes. In Explicit Mode, which Cisco is currently using for all B2B and A2A applications, the network routes all requests for services to AON. The developer uses AON Development Studio to recognize particular requests and invoke the flow associated with that request. In Implicit Mode, the network is configured to route only those requests to or from a specific IP address or port to the AON module. AON intercepts these requests and then invokes the corresponding flow. Cisco IT plans to enable Implicit Mode for several A2A and service-brokering functions during 2006.
Figure 4. AON Management Console
Between January 2006 and April 2006, Cisco deployed the following three AON-based applications, and as of May 2006, many more applications were in various stages of planning and development (see Next Steps).
HireRight Application: Reduced Development and Debugging Time from 240 to 120 Hours
Cisco IT manager
Cisco hires 1500 people per quarter worldwide. The Cisco Systems HR group uses advanced technology to streamline its processes, and was the first HR group to integrate its online job application with a background-check system. However, one cumbersome manual step remained. The vendor that performed background checks for potential new hires sent back investigation results as an Adobe PDF file, and then a Cisco HR employee had to manually re-enter the information into the Cisco HR database. "The old process required us to spend almost an hour doing manual data entry for each new hire, sometimes creating a two-day backlog that delayed offers, and also introduced the possibility of errors," says Peggy Donatelli, HR manager. "Accuracy is very important because the HR database provides information to downstream systems, so a single error has to be corrected in multiple systems." Labor costs for manual data entry totaled about $150,000 annually.
To automate the process, Cisco IT originally planned to develop a Web Service to send XML information from the background-checking vendor to the HR database. But Cisco Information Security asked the developers to take another approach that did not allow outside systems access to a Cisco core system through the firewall. "The concern was the risk of an outsider sending malicious data into the HR database, or bogus job candidate information," says Sujata Joshi, IT HR project manager for Cisco.
Figure 5. HireRight HR Application
Cisco IT met the business and security requirements with an application that calls Cisco AON. The HireRight application, deployed in January 2006, provides the vendor that checks new-hire backgrounds with a secure, real-time connection to the Cisco HR database. The vendor can see when Cisco HR enters a new name to be investigated, and submits investigation results directly into the Cisco HR database (Figure 5). By avoiding the need for manual data re-entry, the application saves 45 to 60 minutes for every potential new employee being screened, and minimizes errors in the Cisco HR database. "Now, as soon as our background checks are complete and the offer signed, the database is automatically updated with no effort required from us," says Donatelli. Cisco can also submit job offers more quickly - often within two hours of receiving the investigation results. "Immediate data entry accelerates the hiring process, and no longer do we have errors introduced by manual data entry," says Donatelli.
Using AON Development Studio, Cisco IT was able to develop the application in just three weeks, or 120 hours of development time. Developers inserted XML tags in the application to call "bladelets," or prewritten AON application functions, which replaced dozens or hundreds of lines of code that otherwise would have had to be written and maintained. "We were also able to reduce the server memory needs for the HireRight application from 1800 MB to less than 100 MB, significantly reducing the server processing load," says Tout.
- AON function used in the HireRight application include:
- SSL v3 encryption with bidirectional authentication
- Certificate validation
- User authentication and logging
- Schema validation
- XML payload transformation between two dissimilar databases
- Content inspection
- Message verification and logging
"Using AON for the integration saved hours of writing and debugging time and increased security," says Joshi. "We were planning on about 240 hours of development effort without AON, but by making use of AON functions rather than coding them ourselves into the application, we spent only about 120 hours. I think that if we were to do a similar application again using AON, we could reduce development time to about 80 hours."
Pre-Order Check Service: Reduced Time to Add Digital Certificates for New Partners from One Day to Two Hours
The goal of the Enterprise B2B IT Group at Cisco is to make it convenient for partners to conduct e-commerce with Cisco Systems-for example, by specifying order configurations and viewing order status. The Enterprise B2B group does not actually build the configuration order-processing applications; rather, it builds the partner interfaces that expose the application functionality to partners not connected to the Cisco network.
The Enterprise B2B IT Group had previously developed seven Web Services, among them the Pre-Order Check. This is the "page before the last page," where trading partners and customers validate their orders before officially submitting them. Cisco IT wanted to enhance the Pre-Order Check service to increase security. It also wanted to make the service accessible to the 85 to 90 percent of partners that do not use Web Services and currently use a separate interface.
In summer of 2006, Cisco IT plans to replace the Pre-Order Check Web Service with an AON-based service (Figure 6).
AON functions used to develop the service include:
- SSL V3 with bidirectional authentication and/or digital signature verification
- Tibco-JMS integration
- SOAP/XML transformation
- Standard activity logging
- Message tracking
Benefits of using AON instead of Web Services for the Pre-Order Check include:
Figure 6. Pre-Order Check Service
Increased security. "One of the primary motivations for using AON was to use an industry-standard security stack rather than proprietary security," says Abhijeet Ranadive, IT architect. The AON application has simplified and improved security using SSL V3 with bidirectional authentication, digital signature verification, content-based inspection, and business activity logging.
Enhanced convenience for partners by providing the service in a multichannel, multiprotocol format. "With AON, we can expose the service using any channel: Web Services, FTP, or a future protocol," says Ranadive. Not only does AON improve convenience for partners, it avoids the need for Cisco IT to maintain two different interfaces to the business service.
Faster implementation. AON enabled faster, more flexible implementation because of its visual development environment and the fact that it provides built-in tools to maintain security and multichannel support. Support and troubleshooting will require fewer resources. "Adding digital certificates for a new partner previously took a day or longer, and with AON it takes just two to three hours," says Ranadive.
Facilitating support of the RosettaNet Web Services standard, planned for AON. RosettaNet Multimedia Message Service (MMS) is expected to replace proprietary Web Services approaches. Adapting the Pre-Order Check service to support RosettaNet MMS will require very little change to the interface.
Virtual Logistics Network: Acquired Flexibility to Support Partners' Multiple Payload Formats.
Logistics is an intricate business process at Cisco Systems, involving contract manufacturers, third-party logistics (3PL) providers in each of three global regions, local shipping carriers, and international freight forwarders. In Cisco's previous 3PL business model, Cisco served as its own logistics provider, receiving and forwarding traffic intended for all parties. Logistics is not Cisco's core competency, however, so the company wanted to outsource this responsibility to a fourth-party logistics provider (4PL).
Initially, Cisco connected its logistics database directly to the 4PL partner over a private leased line. The partner polled the Cisco logistics database at regular intervals to capture new records. The disadvantage of this arrangement was that it required customized integration, including tight coupling down to the data layer. "Cisco wanted a simpler way to connect its logistics database to the 4PL provider," says Sharmin Choksey, IT architect. "We needed the flexibility to change vendors and to support the different message payload formats used in different parts of the world."
Cisco IT is simplifying the architecture for the virtual logistics network by using AON for six critical logistics services. The first AON-based service in production is the Notify of Shipping Documentation message, which Cisco sends to the 4PL partner to initiate the process of building the documentation needed for customs and compliance.
"AON enables an event-driven architecture rather than a polling architecture," says Choksey. When a new record is inserted into Cisco's logistics database, it is pushed to the 4PL partner (Figure 7).
Figure 7. Virtual Logistics Network Application
A major advantage of AON compared to other application-integration approaches is that it supports AS2, which is a payload-neutral B2B protocol. Cisco expects the message payload format to change over time, and to vary by geographical region because of different customs and compliance requirements. In the past, changing the message payload format also required changing the business protocol. "But once you implement a business protocol, it is very difficult to change - especially when logistics partners have already made an investment," says Choksey. "AON's support for the payload-neutral AS2 protocol meets the need by decoupling the payload from the business protocol."
AON provides the following functions for the Notify of Shipping Documentation service:
- Reliable messaging, with at guaranteed delivery
- Message redelivery with contextual look-up
- Real-time messaging for a medium to large payload; message sizes range from 2 KB to 8 MB
- Message tracking
- Application logic
- Schema validation
- Nonrepudiation (planned, when this service is added to AON)
Major business benefits to Cisco of using AON for the Virtual Logistics Network application include:
Reducing dependence on vendors and technology by supporting a payload-neutral business protocol
Simplifying architecture, reducing costs of maintenance, support, and enhancements
Standards-based integration, which lowers costs of enhancements and modifications, and reduces required number of technologies, teams, and processes
Providing visibility into the message payload. Because Cisco AON is embedded directly into the network, it has the ability to intercept important business transactions, avoiding the need to augment applications with expensive and complex agents, probes, or proxies.
Following is a summary of the business and IT benefits of AON-based development for Cisco Systems:
Return on investment - The most important measure of success of AON for Cisco is ROI.
ROI factors include the following:
Avoidance of increased IT headcount - "We are definitely reducing the number of developer resources," says Puri, who adds that measurements are in progress. By standardizing a set of common application functions-digital signature verification, SSL v3 with bidirectional authorization, termination, logging, digital certificate validation, and protocol translation-Cisco IT expects to be able to reassign five full-time IT employees who currently provide development and support. By standardizing another set of functions-monitoring, content-based routing and inspection, and XML encryption and decryption-Cisco IT expects to be able to reassign an additional three full-time developers.
Avoidance of licensing costs - Cisco has avoided $300,000 in annual costs for licensing and maintenance for Web Services management software, a result of the AON applications described in this case study. Reduced hardware and memory requirements-Cisco estimates that AON will enable the company to retire more than 100,000 lines of custom code. This has already begun reducing processing and memory requirements for application servers. The memory requirements for the server used for HireRight, for example, dropped twenty-fold, from 2 GB to 100 MB.
Accelerating application delivery by reducing development lifecycle - AON reduces development resources by standardizing application capabilities, avoiding the need to implement them multiple times for each technology or environment in use in the Cisco global network. "When developers are assigned a project, they typically spend 50 percent of their time understanding the business requirements and 50 percent figuring out how to implement the functions in each environment," says Puri. "With AON, we can ask the network to perform capabilities common to all platforms-digital certificates, authentication, SSL, and others-so that developers can devote far more of their time to the business problem." AON also speeds up integration projects, previously complicated by the fact that seven Cisco IT groups are involved in development, focusing respectively on the network, middleware, security, various application teams, integration, B2B communication, and gateway. Partner organizations often have the same organizational divisions, resulting in the need for up to 14 organizations to collaborate in integration projects. AON simplifies collaboration because application teams can develop their workflows in a single environment instead of building the same solution in multiple environments. So far the time savings are most noticeable for service brokering, protocol translation, and data transformation.
Enhancing security - Most exploitable application vulnerabilities result from the implementation, not the algorithm, according to Brook Schoenfield, senior security architect for Cisco. "A good maxim is to implement common or tricky security services as part of the infrastructure," he says. AON enhances security by providing a common, simpler provisioning and configuration process, and by reducing the number of unprotected segments between applications and AON. What is ordinarily Layer 7 security is implemented instead in Layers 3 and 4, in the network device. In addition, AON provides a common and consistent implementation of security functions, including digital signatures, encryption and decryption, authentication, access control lists (ACLs), and validation. "By implementing, testing, and validating the PKI [Public Key Infrastructure] in AON, Cisco gained assurance that PKI is correct in every application," says Shoenfield.
Ability to globally change or update application functions or policies - AON maintains its functions in a central library so that an update to the function in the library immediately applies to potentially hundreds of applications that perform the function. Without AON, in contrast, developers would have to rewrite the code one application at a time, increasing time and expense.
The Cisco IT architects and developers that have developed services with AON offer the following observations:
The first application that a developer writes on AON will take the most time.
Moving application intelligence to the network shifts the development paradigm by requiring closer collaboration between developers and application infrastructure engineers, called Webmasters at Cisco Systems. This gives Cisco IT infrastructure teams the opportunity to learn more about the applications delivered over the infrastructure, inspiring them to come up with innovative design ideas. The effort is driven by the IT infrastructure team, while the development teams enjoy the benefits.
AON complements other application-development technologies rather than supplanting them. "It is important to take the long-term view of application services rather than simply deploying AON and expecting it to take care of all application-development challenges," says Puri. "For example, the AON digital signatures function is even more useful when complemented by a PKI."
The Cisco AON implementation team recommends the following best practices based on its experiences to date:
- Cluster AON modules based on functionality, perhaps application-specific capabilities in one cluster and common capabilities in another.
- For external-facing Web Services, the infrastructure AON cluster should reside in the DMZ, while the application AON cluster should reside in the protected net.
- Standardize on AONP(S) as the inter-cluster communication protocol because it exposes other features that also use this protocol, such as guaranteed delivery, quality of service, and message ordering.
- Standardize on a naming convention for resources, flows, and properties.
- For AON provisioning and management, Cisco recommends that IT groups:
- Develop a standard System Development Lifecycle (SDLC) strategy, including development, test, and production.
- Prevent collisions by utilizing standard name spacing.
- Create development environment sandboxes for each project team.
- Employ an automated promotion process.
Cisco IT engineer
Cisco is in the process of deploying an application that uses AON for integration with salesforce.com-specifically, enabling single sign-on for Cisco sales users and Cisco partners. The Cisco sales organization utilizes the application service provider (ASP) salesforce.com for sales force automation (SFA) services, including account, opportunity, contact, and lead information. Previously, salesforce.com provided authentication and authorization for Cisco users who wanted to access the service.
To facilitate collaboration between the Cisco sales organization and Cisco's partner community, the Cisco IT organization wanted to perform authentication and authorization within Cisco's own IP network. This would create two major benefits, according to Ramesh Vadlapatla, Cisco IT engineer. One would be the convenience of single sign-on for Cisco employees and partners. Previously Cisco sales users and partners could use a single password for salesforce.com and cisco.com but had to log in to each site separately. The other benefit would be greater security and access control. If Cisco added or removed access privileges for an employee, vendor, or partner, the change would be applied immediately rather than after a lengthy batch process integration between salesforce.com and Cisco.
To satisfy this business need, salesforce.com had suggested that Cisco purchase a third-party hardware product that provides Security Assertion Markup Language (SAML) integration between Cisco and salesforce.com by using the delegated authentication function in salesforce.com. However, Cisco did not want to incur the capital expense and ongoing operational costs of specialized hardware and integration into Cisco's environment.
The sales organization also experienced another business need pertaining to salesforce.com integration. The existing production environment enabled updated account, opportunity, contact, and lead information to flow from the salesforce.com environment to Cisco's internal application. Cisco Information Security does not allow integration between testing environments and external applications using Web Services, for security reasons. "We wanted to add authentication and certificate management to our Web Services, but could not do this without first validating these features with salesforce.com in an end-to-end testing environment," says Vadlapatla.
Cisco IT is using AON to quickly integrate salesforce.com with Cisco's Lightweight Directory Access Protocol (LDAP) infrastructure, increasing security, enabling single sign-on, and preventing terminated employees and partners from logging in to salesforce.com (Figure 8). AON will also enable Cisco IT to securely add development and staging environments. Adding authentication and certificate management-capabilities not possible in the previous environment-is expected to take one person just six weeks. "Standards-based integration will enable secure, federated access to applications using SAML, enabling Cisco employees and partners to log in just once to access applications shared between their respective organizations," Vadlapatla adds.
Figure 8. Integrating with Salesforce.com: Delegated Authentication/Single Sign-on
AON Development Studio also helped the developer present the solution to the Cisco business user who approved the effort. "The visual interface in AON Development Studio makes it easier for business users to understand the business process," says Arul Govindarajan, Cisco IT engineer. "By providing pre-built components called bladelets for common tasks, AON cuts down the time required to implement a solution. This also improves the quality because the bladelets are tested solutions."
The Sales IT organization plans to reuse the same application infrastructure to integrate its LDAP directory with other ASPs that require authentication.
Additional applications under consideration for AON include:
- Radio Frequency ID (RFID) for loading docks. Cisco anticipates that AON will lower the capital and operating expenses of RFID when compared with installing, servicing, and supporting RFID software on Linux servers in loading docks.
- RFID for asset tracking in global centers, helping to locate misplaced network and server equipment.
- Siebold migration. Migration of existing Web Methods B2B traffic to AON
- Lease Record Service
- Approved Vendor List
- Contract Coverage
- Serial Number Check
- Compliance with the Sarbanes-Oxley Act
The Cisco IT team responsible for compliance with the Sarbanes-Oxley Act is working with Cisco's external auditors to use AON for monitoring certain of its databases. "We want to use AON technology to capture specific transactions and user IDs to produce a report of activity, or a no-activity report," says Wendy Grande, IT Governance program manager "We believe that AON will be more effective than the traditional approach, which is to purchase software that will produce an activity log that a manager will then have to review and approve."
Cisco IT also plans to add AON modules to branch office routers. "We are still evaluating the business potential of using AON in different ways," says Gore. "It is like PCs were in 1978, when people were still discovering what they could do." One possibility, since AON provides message transformation services, is to translate voice messages to text in real time.
"Like every other enterprise IT group, Cisco IT has portfolios and limited resources," says Andersson. "By reducing development cycles and hardware and memory costs, AON is helping Cisco update its applications faster, for greater business effectiveness."
For more information on AON, go to: http://www.cisco.com/en/US/products/ps6692/Products_Sub_Category_Home.html