Cisco UCS Infrastructure with Docker Datacenter for Container Management

Design and Deployment Guide for Cisco UCS Infrastructure with Docker Datacenter for Container Management

NOTE: Works with document’s Advanced Properties “First Published” property. Click File | Properties | Advanced Properties | Custom.

Last Updated: October 2, 2017

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

                                                                                                                                                                           

About the Cisco Validated Design (CVD) Program

The CVD program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information visit

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2017 Cisco Systems, Inc. All rights reserved.

 

Table of Contents

Executive Summary. 6

Business Challenges. 6

Our Solution. 7

Implementation Overview.. 8

Highlights. 8

Solution Benefits. 9

Audience. 9

Purpose of the Document 9

Solution Overview.. 10

Solution Components. 11

Technology Overview.. 12

Cisco Unified Computing System.. 12

Cisco UCS Manager 13

Cisco UCS Fabric Interconnects. 13

Cisco UCS 5108 Blade Server Chassis. 14

Cisco UCS B200M4 Blade Server 14

Cisco UCS C220M4 Rack-Mount Server 14

Cisco UCS Fabric Extenders. 14

Cisco VIC Interface Cards. 15

Cisco UCS Differentiators. 15

Cisco Nexus 9000 Switches. 17

Docker Datacenter 17

Docker CS Engine. 18

Docker UCP. 19

Docker Trusted Registry. 22

Solution Design. 25

Architectural Overview.. 25

Sizing Considerations. 25

Solution Workflow.. 28

Cisco Nexus 9372PX Configuration. 28

Cisco UCS Manager Configuration. 28

Docker Datacenter Installation. 29

Software and Hardware Versions. 29

Solution Deployment 33

Cisco Nexus 9372PX. 33

Initial Configuration and Setup. 33

Feature Enablement 34

vLAN Creation. 34

Configure vPC.. 34

Configure Network Interfaces. 37

Cisco UCS Manager 39

Initial Setup of Cisco Fabric Interconnects. 39

Configure Ports for Server, Network and Storage Access. 40

Cisco UCS Manager – Synchronize to NTP. 40

Upgrading Cisco UCS Manager 40

Assigning Block of IP addresses for KVM Access. 41

Editing Chassis Discovery Policy. 41

Acknowledging Cisco UCS Chassis. 42

Enabling Server Ports. 42

Enabling Uplink Ports to Cisco Nexus 9000 Series Switches. 43

Configuring Port Channels on Uplink Ports to Cisco Nexus 9000 Series Switches. 44

Cisco UCS Configuration – LAN.. 46

Creating VLANs. 46

Creating LAN Pools. 47

Creating LAN Policies. 48

Creating vNIC Templates. 48

Cisco UCS Configuration – Server 50

Creating Server Policies. 50

Creating BIOS Policy. 50

Creating Boot Policy. 51

Creating Host Firmware Package Policy. 54

Creating Server Pools. 54

Creating UUID Suffix Pool 54

Creating Server Pools. 55

Cisco UCS Configuration – Storage. 58

Creating Storage Profile. 58

Creating Service Profile Templates. 62

Creating Service Profile Template. 62

Creating Service Profile Template for UCP Controller Nodes. 72

Creating Service Profile Template for UCP Nodes. 72

Service Profile Instantiation and Association. 73

Service Profile Instantiation. 73

Associating Service Profile Templates to Server Pools. 74

Installation of Red Hat Enterprise Linux Operating System.. 78

Docker Datacenter Installation. 87

Registering Nodes and Updating Host OS. 87

Installing and Configuring Ansible. 89

Installing NTP and Configuring Host OS System Clocks. 93

Installing Cisco Virtual Interface Card (VIC) eNIC (Ethernet Network Interface Card) Driver 96

Configuring Host OS Firewall for required ports. 98

Installation of Docker Repo and Engine. 103

Configuring Docker CS Engine for Device-Mapper Driver in Direct LVM-Mode. 109

Install and Configure Docker UCP Controller Nodes. 119

Backup Controller CAs and Add UCP Replicas. 121

Add UCP Nodes. 124

Install and Configure DTR and its Replicas. 125

Validate Docker UCP and DTR Cluster 132

Management and Monitoring Interface. 134

Solution Validation. 140

Application Container Deployment 140

Container Networks. 141

Deploying container/application container using overlay network. 143

DTR Operations. 149

Sample WordPress Application Container Deployment 153

High-Availability Tests. 156

Tests Performed on Docker Datacenter 156

Tests Performed on Cisco UCS Servers. 158

Upscale Tests. 159

Removal of Docker UCP Controller Replicas and UCP Nodes. 164

Bill of Materials. 167

Summary. 170

About Authors. 171

Acknowledgements. 171

 

 

Cisco Validated Design program consists of systems and solutions that are designed, tested, and documented to enable end-to-end customer deployments. These designs incorporate a wide range of technologies and products into solution portfolios that have been developed to address the business needs of our customers.

Cisco Unified Computing System^™ (Cisco UCS^®) servers adapt to meet rapidly changing business requirements, including just-in-time deployment of new computing resources to meet requirements and improve business outcomes. With Cisco UCS, you can tune your environment to support the unique needs of each application while powering all your server workloads on a centrally managed, highly scalable system. Cisco UCS brings the flexibility of non-virtualized and virtualized systems in a way that no other server architecture can, lowering costs and improving your return on investment (ROI).

Docker is an efficient platform for developers and IT operations to use to build, ship, and run distributed applications anywhere. With Microservices architecture shaping the next generation of IT, enterprises with large investments in monolithic applications are finding ways to adopt Docker as a strategy for modernizing their application architectures and keeping the organization competitive and cost effective. Containerization provides the agility, control, and portability that developers and IT operations require to build and deploy applications across any infrastructure. The Docker platform allows distributed applications to be easily composed into a lightweight application container that can change dynamically yet non-disruptively. This capability makes the applications portable across development, test, and production environments running on physical or virtual machines locally, in data centers, and across the networks of different cloud service providers. Docker Datacenter provides native container management tools, including the Docker Trusted Registry (DTR) and Universal Control Plane (UCP) components. It can be deployed on an organization’s premises or in a virtual private cloud (VPC) and connected to existing infrastructure and systems such as storage and Lightweight Directory Access Protocol (LDAP) or Microsoft Active Directory (AD) services (LDAP/AD services).

Cisco and Docker have joined hands to offer Container Management Solution on Cisco UCS Infrastructure with Docker Datacenter. This enables enterprises to modernize traditional applications and build Microservices architecture using the Docker platform and tools on Cisco’s proven Cisco UCS Integrated Infrastructure. The combination of Docker container technology and Cisco UCS server hardware enables a highly scalable, resilient, and elastic application deployment environment with the simplicity of the on-premise cloud like experience.

Business Challenges

Technological revolution has created new opportunities and challenges for businesses in this digital world. Many startups or smaller competitors are using disruptive innovations such as Microservices architecture to quickly develop and deploy applications and services to rapidly adopt changing markets and meet customer needs. These innovations also provide a path to modernize traditional business critical applications providing agility, flexibility and portability to reduce operational and management costs while improving performance and security. In order to keep up with new technologies or stay one step ahead, enterprises will have to overcome key challenges to accelerate product development, add value and compete better at lower cost.

Key Challenges:

·         Portability: Applications have dependencies around OS versions, libraries, Java versions, etc. Any changes to these dependencies can break portability which means applications developed on a specific environment may behave differently on a production environment hosted on-premise or cloud. Changing code and rebuilding/testing code leads to delay in product or service offering and loss of market share.

·         Agility:  Enterprises have many legacy applications, tools and complex development process slowing innovation significantly as product release cycle takes days to weeks due to complex flow from development to production.

·         Resource Utilization: Engineering and hardware resources in large enterprises are not used efficiently due to various organizations operating in silos. These silos that worked in the past are causing a huge overhead in development/release cycle and resource under-utilization as technology changes rapidly.

·         Policy Management: Large enterprises have redundant processes and policies in place, and are reluctant to adopt new technologies due to fear of breaking compliance causing delays in new development/release cycle.

Our Solution

This Cisco Validated Design focuses on Cisco UCS Integrated Infrastructure and Docker Datacenter that enables enterprises to rapidly deploy and manage production ready Container as a Service environment on a highly available and secure platform. Additionally, it helps automate DevOps process with an integrated security from development to production, accelerating product release cycle and reducing high OS or VM licensing costs and better resource utilization. The solution is well tested for high-availability, performance and scalability, optimizing time and resources to bring the system and solution online quickly. Cisco has partnered with Docker to deploy Docker Datacenter (which includes Docker Universal Control Plane, Docker Trusted Registry and Commercially Supported- Docker CS Engine) on Cisco UCS platform.

This solution covers production ready installation, provisioning, configuring and deploying application containers using Docker Datacenter on Cisco UCS B-Series and C-Series server platforms in two separate topologies for production and dev/test use cases. Docker Datacenter’s Universal Control Plane provide clustering of Docker Engine Nodes and clustering services, automates cluster and application container life cycle management and integrates with Docker Trusted Registry services for application container image services. Cisco UCS infrastructure provides the converged platform for the compute, network, storage and entire hardware lifecycle management through a single management control plane. The solution demonstrates:

·         Quick and easy installation of Cisco UCS Integrated Infrastructure, Docker Datacenter and Application Containers

·         Application Container Management through Docker Datacenter on compute nodes irrespective of form factors by utilizing Cisco UCS Manager capabilities

·         Create and configure network and storage across complete infrastructure for Application Containers

·         High-Availability test inducing node and container engine failures

·         Scalability apropos of networks, subnets, storage access, containers, and compute/infra nodes

·         Performance with regard to reducing the bring-up time of container seen with DTR integration in the stack

The combination of Cisco UCS and Docker Datacenter allows organizations to build and deploy containerized applications on an open, highly available and scalable platform leveraging existing hardware investments to provide an end-end secure platform to meet SLAs.

Implementation Overview

Docker Datacenter solution is integrated and validated on Cisco UCS Integrated Infrastructure. This solution is implemented on Cisco UCS B- and C-Series servers and Cisco Nexus platforms. The architecture covers high level install/configuration, provisioning process and the solution testing required for CVD. Cisco UCS blade and rack servers are provisioned through Service Profiles and OS installation is done manually. OS configuration and Docker Datacenter installation is automated through built-in Docker tools and Ansible. The end-to-end stack is tested for correctness (recommended software stack), performance, and scalability, high-availability and security policies. The containers are deployed and managed by Docker UCP. The deployment guide provides step by step instructions on setting up the complete stack and solution validation test results.

Highlights

·         Industry-Leading Converged Computing Infrastructure: Cisco UCS blade and rack servers enable customers to rapidly and efficiently deploy and manage compute, network and storage functions for containers. They enable customers to reduce IT costs through consolidation, manage more containers per computing node and make better use of infrastructure to provide an application container environment in real time.

·         Industry-Leading Container Application Infrastructure: Docker Datacenter brings container orchestration, management and security to the enterprise. It enables developers and system administrators to build, ship and run distributed applications anywhere.

·         Combined Use cases for the enterprise who want to leverage the solution:

-        Micro-service and cloud native application architecture: Enables stateless application container deployment for specific business use needs

-        Continuous integration and continuous deployment(CI/CD): Enable developers to develop and test applications more quickly and within relevant environment

-        DevOps: Break down barriers between development and operations teams to improve delivery of applications for the business

This platform allows you to quickly adopt CI/CD and DevOps use cases by adding components like Jenkins and integrating it with Docker Datacenter. It also provides an infrastructure platform for converting applications into Microservices and cloud native architectures and deploying them through stateless application containers.

·         Big Data: Empower organizations to use big data analytics using small foot print applications at a very large scale numbers

-        Infrastructure optimization: Decrease infrastructure costs while increasing efficiency. The lightweight nature of containers and the absence of hypervisors and guest operating systems enables enterprises to optimize resource and eliminate licensing costs

Solution Benefits

The benefits of Cisco UCS with Docker Datacenter include the following:

·         Cisco UCS

-        Simplicity: Reduced datacenter complexities through Cisco UCS converged infrastructure with a single management control plane for hardware lifecycle management

-        Rapid Deployment: Easily deploy and scale the solution

-        High Availability: Superior scalability and high-availability

-        Flexibility: Compute form factor agnostic

-        Faster ROI: Better response with optimal ROI

-        Resource Utilization: Optimized hardware footprint for production and dev/test deployments

·         Docker Datacenter

-        Agility: Gain the freedom to define environments and create and deploy applications quickly and easily, providing flexibility of IT operations that respond quickly to change

-        Control: Enable developers to own the code from the infrastructure to the application and quickly move from the build to the production environment. IT operations manageability features enable organizations to standardize, secure, and scale the operating environment

-        Portability: Docker Containers are self-contained and independent units that are portable between private infrastructure and public cloud environments without complexity or disruption

Audience

The audience for this document includes, but is not limited to, sales engineers, field consultants, professional services, IT managers, partner engineers, IT architects, and customers who want to take advantage of an infrastructure that is built to deliver IT efficiency and enable IT innovation. The reader of this document is expected to have the necessary training and background to install and configure Red Hat Enterprise Linux, Cisco Unified Computing System (UCS) and Cisco Nexus Switches as well as high level understanding of Docker Container components. External references are provided where applicable and it is recommended that the reader be familiar with these documents.

Readers are also expected to be familiar with the infrastructure, network and security policies of the customer installation.

Purpose of the Document  

This document highlights the benefits of using Cisco UCS infrastructure with Docker Datacenter to efficiently deploy, scale, and manage a production-ready application container environment for enterprise customers. The goal of this document is to demonstrate the value that Cisco UCS brings to the datacenter, such as single-point hardware lifecycle management and highly available converged compute and network infrastructure for application container deployments using Docker Datacenter.

Cisco UCS Integrated Infrastructure solutions speed up IT operations today and create the modern technology foundation you need for initiatives like private cloud, big data, and desktop virtualization. With Cisco UCS Manager and Cisco Single Connect Technology, hardware is automatically configured by application-centric policies—ushering in a new era of speed, consistency, and simplicity for datacenter operations. UCS brings the flexibility of virtualized systems to the physical world in a way no other server architecture can, lowering costs and improving your ROI.

Leveraging the centralized management of Cisco UCS Manager, this solution provides unified, embedded, policy-driven management to programmatically control server, network, and storage resources you can efficiently manage the scale-up/ -out infrastructure. Furthermore, Cisco Nexus - unified Fabric is a holistic network architecture comprising switching, security, and services that are designed for physical, virtual, and cloud environments. It uniquely integrates with servers, storage, and orchestration platforms for more efficient operations and greater scalability.

Cisco has partnered with Docker to provide Container Management solution to accelerate the IT transformation by enabling easy and faster deployments, greater flexibility of choice, business agility, efficiency, lower risk.

Docker has become the industry standard for developers and IT operations to build, ship and run distributed applications in baremetal, virtualized and cloud environments. As organizations adopt public, private or Hybrid cloud, Docker makes it easy to move applications between on premise and cloud environments. Docker can significantly improve hardware resource utilization, accelerate application lifecycle and reduce overall cost by automating IT processes, and deploying dockerized applications on-premise or in cloud environment.

Figure 1       High-Level Docker Infrastructure Diagram

Docker Datacenter delivers an integrated platform for developers and IT operations to collaborate in the enterprise software supply chain. Bringing security, policy and controls to the application lifecycle without sacrificing any agility or application portability. Docker Datacenter integrates to enterprise business – from on-premises and VPC deployment models, open APIs and interfaces, to flexibility for supporting a wide variety of workflows.

Solution Components

The solution offers redundant architecture from a compute, network, and storage perspective. The solution consists of the following key components:

·         Cisco Unified Computing System (UCS)

·         Cisco UCS Manager

·         Cisco UCS 6248UP Fabric Interconnects

·         Cisco 2208XP IO Module or Cisco UCS Fabric Extenders

·         Cisco B200 M4 Servers

·         Cisco C220 M4S Servers

·         Cisco VIC 1340

·         Cisco VIC 1227

·         Cisco Nexus 9372PX Switches

·         Docker Datacenter (DDC)

·         Docker CS Engine

·         Docker Universal Control Plane (UCP)

·         Docker Trusted Repository (DTR)

·         Red Hat Enterprise Linux 7.2

This section provides a brief introduction of the various hardware/ software components used in this solution.

Cisco Unified Computing System

The Cisco Unified Computing System is a next-generation solution for blade and rack server computing. The system integrates a low-latency; lossless 10 Gigabit Ethernet unified network fabric with enterprise-class, x86-architecture servers. The system is an integrated, scalable, multi-chassis platform in which all resources participate in a unified management domain. The Cisco Unified Computing System accelerates the delivery of new services simply, reliably, and securely through end-to-end provisioning and migration support for both virtualized and non-virtualized systems. Cisco Unified Computing System provides:

·         Comprehensive Management

·         Radical Simplification

·         High Performance

The Cisco Unified Computing System consists of the following components:

·         Compute - The system is based on an entirely new class of computing system that incorporates rack mount and blade servers based on Intel Xeon 26xx v4 Series Processors.

·         Network - The system is integrated onto a low-latency, lossless, 10-Gbps unified network fabric. This network foundation consolidates Local Area Networks (LAN’s), Storage Area Networks (SANs), and high-performance computing networks which are separate networks today. The unified fabric lowers costs by reducing the number of network adapters, switches, and cables, and by decreasing the power and cooling requirements.

·         Virtualization - The system unleashes the full potential of virtualization by enhancing the scalability, performance, and operational control of virtual environments. Cisco security, policy enforcement, and diagnostic features are now extended into virtualized environments to better support changing business and IT requirements.

·         Storage access - The system provides consolidated access to both SAN storage and Network Attached Storage (NAS) over the unified fabric. It is also an ideal system for Software defined Storage (SDS). Combining the benefits of single framework to manage both the compute and Storage servers in a single pane, Quality of Service (QOS) can be implemented if needed to inject IO throttling in the system. In addition, the server administrators can pre-assign storage-access policies to storage resources, for simplified storage connectivity and management leading to increased productivity. In addition to external storage, both rack and blade servers have internal storage which can be accessed through built-in hardware RAID controllers. With storage profile and disk configuration policy configured in Cisco UCS Manager, storage needs for the host OS and application data gets fulfilled by user defined RAID groups for high availability and better performance.

·         Management - the system uniquely integrates all system components to enable the entire solution to be managed as a single entity by the Cisco UCS Manager. The Cisco UCS Manager has an intuitive graphical user interface (GUI), a command-line interface (CLI), and a powerful scripting library module for Microsoft PowerShell built on a robust application programming interface (API) to manage all system configuration and operations.

Cisco Unified Computing System (Cisco UCS) fuses access layer networking and servers. This high-performance, next-generation server system provides a data center with a high degree of workload agility and scalability.

Cisco UCS Manager

Cisco Unified Computing System (UCS) Manager provides unified, embedded management for all software and hardware components in the Cisco UCS. Using Single Connect technology, it manages, controls, and administers multiple chassis for thousands of virtual machines. Administrators use the software to manage the entire Cisco Unified Computing System as a single logical entity through an intuitive GUI, a command-line interface (CLI), or an XML API. The Cisco UCS Manager resides on a pair of Cisco UCS 6200 Series Fabric Interconnects using a clustered, active-standby configuration for high-availability.

UCS Manager offers unified embedded management interface that integrates server, network, and storage. UCS Manager performs auto-discovery to detect inventory, manage, and provision system components that are added or changed. It offers comprehensive set of XML API for third part integration, exposes 9000 points of integration and facilitates custom development for automation, orchestration, and to achieve new levels of system visibility and control.

Service profiles benefit both virtualized and non-virtualized environments and increase the mobility of non-virtualized servers, such as when moving workloads from server to server or taking a server offline for service or upgrade. Profiles can also be used in conjunction with virtualization clusters to bring new resources online easily, complementing existing virtual machine mobility.

For more Cisco UCS Manager Information, refer to: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html

Cisco UCS Fabric Interconnects

The Fabric interconnects provide a single point for connectivity and management for the entire system. Typically deployed as an active-active pair, the system’s fabric interconnects integrate all components into a single, highly-available management domain controlled by Cisco UCS Manager. The fabric interconnects manage all I/O efficiently and securely at a single point, resulting in deterministic I/O latency regardless of a server or virtual machine’s topological location in the system.

Cisco UCS 6200 Series Fabric Interconnects support the system’s 80-Gbps unified fabric with low-latency, lossless, cut-through switching that supports IP, storage, and management traffic using a single set of cables. The fabric interconnects feature virtual interfaces that terminate both physical and virtual connections equivalently, establishing a virtualization-aware environment in which blade, rack servers, and virtual machines are interconnected using the same mechanisms. The Cisco UCS 6248UP is a 1-RU Fabric Interconnect that features up to 48 universal ports that can support 80 Gigabit Ethernet, Fiber Channel over Ethernet, or native Fiber Channel connectivity.

For more information, visit the following link: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-6200-series-fabric-interconnects/index.html

Cisco UCS 5108 Blade Server Chassis

The Cisco UCS 5100 Series Blade Server Chassis is a crucial building block of the Cisco Unified Computing System, delivering a scalable and flexible blade server chassis. The Cisco UCS 5108 Blade Server Chassis is six rack units (6RU) high and can mount in an industry-standard 19-inch rack. A single chassis can house up to eight half-width Cisco UCS B-Series Blade Servers and can accommodate both half-width and full-width blade form factors. Four single-phase, hot-swappable power supplies are accessible from the front of the chassis. These power supplies are 92 percent efficient and can be configured to support non-redundant, N+ 1 redundant and grid-redundant configurations. The rear of the chassis contains eight hot-swappable fans, four power connectors (one per power supply), and two I/O bays for Cisco UCS 2204XP or 2208XP Fabric Extenders. A passive mid-plane provides up to 40 Gbps of I/O bandwidth per server slot and up to 80 Gbps of I/O bandwidth for two slots. The chassis is capable of supporting future 80 Gigabit Ethernet standards.

For more information, please refer to the following link: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-5100-series-blade-server-chassis/index.html

Cisco UCS B200M4 Blade Server

The enterprise-class Cisco UCS B200 M4 blade server extends the capabilities of Cisco’s Unified Computing System portfolio in a half-width blade form factor. The Cisco UCS B200 M4 uses the power of the latest Intel® Xeon® E5-2600 v3 Series processor family CPUs with up to 768 GB of RAM (using 32 GB DIMMs), two solid-state drives (SSDs) or hard disk drives (HDDs), and up to 80 Gbps throughput connectivity. The UCS B200 M4 Blade Server mounts in a Cisco UCS 5100 Series blade server chassis or UCS Mini blade server chassis. It has 24 total slots for registered ECC DIMMs (RDIMMs) or load-reduced DIMMs (LR DIMMs) for up to 768 GB total memory capacity (B200 M4 configured with two CPUs using 32 GB DIMMs). It supports one connector for Cisco’s VIC 1340 or 1240 adapter, which provides Ethernet and FCoE.

For more information, see: http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-b200-m4-blade-server/index.html

Cisco UCS C220M4 Rack-Mount Server

The enterprise-class Cisco UCS C220 M4 server extends the capabilities of the Cisco Unified Computing System (UCS) portfolio in a one rack-unit (1RU) form-factor. The Cisco UCS C220 M4 uses the power of the latest Intel® Xeon® E5-2600 v3 and v4 Series processor family CPUs with up to 1536 GB of RAM (using 64 GB DIMMs), 8 Small Form-Factor (SFF) drives or 4 Large Form-Factor (LFF) drives, and up to 80 Gbps throughput connectivity. The Cisco UCS C220 M4 Rack Server can be used standalone, or as integrated part of the Unified Computing System. It has 24 DIMM for up to 1536 GB total memory capacity. It supports one connector for the Cisco VIC 1225, 1227 or 1380 adapters, which provide Ethernet and FCoE.

For more information, see: http://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c220-m4-rack-server/datasheet-c78-732386.html

Cisco UCS Fabric Extenders

The Cisco UCS 2208XP Fabric Extender has eight 10 Gigabit Ethernet, FCoE-capable, and Enhanced Small Form-Factor Pluggable (SFP+) ports that connect the blade chassis to the fabric interconnect. Each Cisco UCS 2208XP has thirty-two 10 Gigabit Ethernet ports connected through the mid-plane to each half-width slot in the chassis. Typically configured in pairs for redundancy, two fabric extenders provide up to 160 Gbps of I/O to the chassis.

Cisco VIC Interface Cards

The Cisco UCS Virtual Interface Card (VIC) 1340 is a 2-port 40-Gbps Ethernet or dual 4 x 10-Gbps Ethernet, Fiber Channel over Ethernet (FCoE) capable modular LAN on motherboard (mLOM) designed exclusively for the M4 generation of Cisco UCS B-Series Blade Servers.

All the blade servers for both Controllers and Computes will have MLOM VIC 1340 card. Each blade will have a capacity of 40 Gb of network traffic. The underlying network interfaces like will share this MLOM card.

The Cisco UCS VIC 1340 enables a policy-based, stateless, agile server infrastructure that can present over 256 PCIe standards-compliant interfaces to the host that can be dynamically configured as either network interface cards (NICs) or host bus adapters (HBAs).

For more information, see:

http://www.cisco.com/c/en/us/products/interfaces-modules/ucs-virtual-interface-card-1340/index.html

The Cisco UCS Virtual Interface Card 1227 offers a modular LAN on Motherboard (mLOM) adapter. The mLOM slot, new to Cisco rack servers, can be used to install a Cisco VIC without consuming a PCIe slot. This provides greater I/O expandability.

A Cisco innovation, the Cisco UCS Virtual Interface Card (VIC) 1227 is a dual-port, Enhanced Small Form-Factor Pluggable (SFP+), 10 Gigabit Ethernet and Fibre Channel over Ethernet (FCoE)-capable, PCI Express (PCIe) modular LAN on motherboard (mLOM) adapter. It is designed exclusively for the M4 generation of Cisco UCS C-Series Rack Servers and the Cisco UCS S-Series dense storage servers.

For more information, see:

http://www.cisco.com/c/en/us/products/interfaces-modules/ucs-virtual-interface-card-1227/index.html

Cisco UCS Differentiators

Cisco’s Unified Compute System is revolutionizing the way servers are managed in data-center. Following are the unique differentiators of UCS and UCS Manager:

1.    Embedded Management —In UCS, the servers are managed by the embedded firmware in the Fabric Interconnects, eliminating need for any external physical or virtual devices to manage the servers.

2.    Unified Fabric —In UCS, from blade server chassis or rack servers to FI, there is a single Ethernet cable used for LAN, SAN and management traffic. This converged I/O results in reduced cables, SFPs and adapters – reducing capital and operational expenses of overall solution.

3.    Auto Discovery —By simply inserting the blade server in the chassis or connecting rack server to the fabric interconnect, discovery and inventory of compute resource occurs automatically without any management intervention. The combination of unified fabric and auto-discovery enables the wire-once architecture of UCS, where compute capability of UCS can be extended easily while keeping the existing external connectivity to LAN, SAN and management networks.

4.    Policy Based Resource Classification —Once a compute resource is discovered by UCS Manager, it can be automatically classified to a given resource pool based on policies defined. This capability is useful in multi-tenant cloud computing. This CVD showcases the policy based resource classification of UCS Manager.

5.    Combined Rack and Blade Server Management —UCS Manager can manage B-Series blade servers and C-Series rack server under the same UCS domain. This feature, along with stateless computing makes compute resources truly hardware form factor agnostic.

6.    Model based Management Architecture —UCS Manager Architecture and management database is model based and data driven. An open XML API is provided to operate on the management model. This enables easy and scalable integration of UCS Manager with other management systems.

7.    Policies, Pools, Templates —The management approach in UCS Manager is based on defining policies, pools and templates, instead of cluttered configuration, which enables a simple, loosely coupled, data driven approach in managing compute, network and storage resources.

8.    Loose Referential Integrity —In UCS Manager, a service profile, port profile or policies can refer to other policies or logical resources with loose referential integrity. A referred policy cannot exist at the time of authoring the referring policy or a referred policy can be deleted even though other policies are referring to it. This provides different subject matter experts to work independently from each-other. This provides great flexibility where different experts from different domains, such as network, storage, security, server and virtualization work together to accomplish a complex task.

9.    Policy Resolution —In UCS Manager, a tree structure of organizational unit hierarchy can be created that mimics the real life tenants and/or organization relationships. Various policies, pools and templates can be defined at different levels of organization hierarchy. A policy referring to another policy by name is resolved in the organization hierarchy with closest policy match. If no policy with specific name is found in the hierarchy of the root organization, then special policy named “default” is searched. This policy resolution practice enables automation friendly management APIs and provides great flexibility to owners of different organizations.

10.  Service Profiles and Stateless Computing —a service profile is a logical representation of a server, carrying its various identities and policies. This logical server can be assigned to any physical compute resource as far as it meets the resource requirements. Stateless computing enables procurement of a server within minutes, which used to take days in legacy server management systems.

11.  Built-in Multi-Tenancy Support —The combination of policies, pools and templates, loose referential integrity, policy resolution in organization hierarchy and a service profiles based approach to compute resources makes UCS Manager inherently friendly to multi-tenant environment typically observed in private and public clouds.

12.  Extended Memory — the enterprise-class Cisco UCS B200 M4 blade server extends the capabilities of Cisco’s Unified Computing System portfolio in a half-width blade form factor. The Cisco UCS B200 M4 harnesses the power of the latest Intel® Xeon® E5-2600 v4 Series processor family CPUs with up to 1536 GB of RAM (using 64 GB DIMMs) – allowing huge VM to physical server ratio required in many deployments, or allowing large memory operations required by certain architectures like Big-Data.

13.  Virtualization Aware Network —VM-FEX technology makes the access network layer aware about host virtualization. This prevents domain pollution of compute and network domains with virtualization when virtual network is managed by port-profiles defined by the network administrators’ team. VM-FEX also off-loads hypervisor CPU by performing switching in the hardware, thus allowing hypervisor CPU to do more virtualization related tasks. VM-FEX technology is well integrated with VMware vCenter, Linux KVM and Hyper-V SR-IOV to simplify cloud management.

14.  Simplified QoS —Even though Fiber Channel and Ethernet are converged in UCS fabric, built-in support for QoS and lossless Ethernet makes it seamless. Network Quality of Service (QoS) is simplified in UCS Manager by representing all system classes in one GUI panel.

Cisco Nexus 9000 Switches

The Cisco Nexus 9000 Series delivers proven high performance and density, low latency, and exceptional power efficiency in a broad range of compact form factors. Operating in Cisco NX-OS Software mode or in Application Centric Infrastructure (ACI) mode, these switches are ideal for traditional or fully automated data center deployments.

The Cisco Nexus 9000 Series Switches offer both modular and fixed 10/40/100 Gigabit Ethernet switch configurations with scalability up to 30 Tbps of non-blocking performance with less than five-microsecond latency, 1152 x 10 Gbps or 288 x 40 Gbps non-blocking Layer 2 and Layer 3 Ethernet ports and wire speed VXLAN gateway, bridging, and routing.

Docker Datacenter

Docker - a containerization platform developed to simplify and standardize deployment in various environments. It is largely instrumental in spurring the adoption of this style of service design and management. Docker containers encapsulate all application components, such as dependencies and services. When all dependencies are encapsulated, applications become portable and can be dependably moved between development, test, and production environments. Docker makes container creation and management simple and integrates with many open source projects. Docker Datacenter comprises an enterprise container orchestration, application management and enterprise-grade security.

Docker Datacenter includes leading Docker open source projects, commercial software, integrations with validated and supported configurations:

·         Commercially supported Docker Engine for a robust container runtime

·         Universal Control Plane (UCP) with embedded Swarm scheduler for integrated management and orchestration of the Docker environment

·         Trusted Registry (DTR) for Docker image management, security, and collaboration

·         Security must be a multi-layered approach; content Trust provides the ability to sign images with digital keys and then verify the signature of those images

Figure 2       Docker Datacenter Platform

Docker CS Engine

Docker engine is the building block for the modern application platform. The Docker CS Engine is commercially supported version of Docker Engine for the enterprise. It's a lightweight container runtime and robust tooling that runs and build containers. Docker allows us to package the application code and dependencies together in an isolated container that shares the OS kernel on the host system. The in-host daemon communicates with Docker Client to execute commands to build, ship and run containers.

Figure 3       Docker CS Engine

Docker UCP

Docker Universal Control Plane (Docker UCP) is an enterprise on premise solution that includes user management, resource management, clustering and orchestration that integrates with the existing enterprise LDAP/AD for High-Availability, security and compliance. Docker UCP enables IT operation teams to deploy and manage the containerized applications in production.

The figure below shows the Docker UCP architecture and illustrates the built-in HA for UCP.

Figure 4       Docker UCP Architecture

UCP is a containerized application, which provides enterprise-grade cluster management solution. It gets installed on Docker CS Engine on all the nodes which are designated as controller nodes. After installing first node with Docker Engine, UCP application is installed and the other nodes are joined to the first node to form a Docker Datacenter cluster.

Figure 5       Docker Datacenter Cluster

A Docker UCP cluster has two types of nodes:

·         UCP Controller Node

·         UCP Node

UCP Controller Node

Docker UCP controller node manages the cluster and provides persistent cluster configurations. Within controller nodes there are two categories of nodes – Master and Replicas. A first node which gets installed with UCP is treated as a Master Controller node. And controller nodes joining the master controller node are termed as Replica nodes. Controller nodes can take up application container workload as well. This is a configurable option available at the admin and user level.

Figure 6       Docker UCP Controller

Table 1    Docker UCP Controller node containers and their description

 

Table 2    Following are the named volumes used by Docker UCP for persistent data

 

While installing Docker UCP these volumes get created with default volume driver. In our solution we have used Docker device-mapper driver in direct-lvm mode.

Docker Universal Control Plane works in high-availability mode. Adding replicas to first UCP Controller node makes the cluster HA ready. A minimum three-node Controller cluster is needed to tolerate one node failure. Adding replica nodes to the cluster allows user request to get load-balanced across controller master and replica nodes.

Docker UCP does not include load-balancing services. It needs external load-balancer to balance user requests across all the controller nodes. 

Docker UCP Node

Docker UCP nodes take the container workload. These are the nodes where containers get deployed. Nodes with Docker CS Engine join the existing UCP cluster. While joining the existing UCP Cluster the following containers get instantiated on the UCP node.

Figure 7       Containers on UCP node while joining the UCP cluster

Table 3    Containers formed during UCP Node Join

Docker Trusted Registry

Docker Trusted Registry (DTR) is an enterprise-grade image storage solution from Docker. DTR gives enterprises the security and compliance to store and manage their Docker images on-premises or in their virtual private cloud (VPC). It has a built-in authentication mechanism; supports role based access control, and allows integration with LDAP and Active Directory.

DTR is part of the Docker Datacenter Subscription which also includes, Universal Control Plane, and commercially supported Engine. DTR is easy to deploy, configure and integrate with your existing infrastructure and application delivery workflows.

The figure below illustrates the built-in HA for DTR within the control plane. HA ensures continuous availability for Docker Trusted Registry; in the event of the main instance failure, a replica instance takes over non-disruptively.

Figure 8       Docker Trusted Registry Architecture

 

Docker Trusted Registry (DTR) is a dockerized application that runs a Docker Universal Control Plane cluster.

Figure 9       DTR as Dockerized application in the UCP cluster

Table 4    With DTR installation the following containers are started

 

The communication between all DTR components is secured using TLS. Also, when installing DTR, two certificate authority (CA) are created. These CAs are used to create the certificates used by etcd and rethinkDB when communicating across nodes.

Table 5    Following networking gets created when DTR containers communicate with each other

 

Table 6    DTR uses following named volumes for persistent data

 

For load balancing and high-availability, multiple replicas of DTR are installed and get joined to form a cluster, within Docker Datacenter UCP cluster. When joining new replicas to the cluster, we create new DTR instances that are running the same set of services. Any change to the state of an instance is replicated across all the other instances.

Figure 10     DTR Cluster behind load balancer

Having a DTR cluster with multiple replicas, allows us to:

·         Load-balance user requests across the DTR replicas

·         Keep the DTR cluster up and running when a replica fails

DTR does not provide a load balancing service. An on-premises or cloud-based load balancer is required to balance requests across multiple DTR replicas. Load balancer is required to be configured:

·         On TCP ports 80 and 443

·         On a TCP load balancer which does not terminates the HTTPS connections

This section provides an overview of the hardware and software components used in this solution, as well as the design factors to be considered in order to make the system work as a single, highly available solution

Architectural Overview

This solution consists of two alternate topologies, targeted for production and dev/test deployments. For production environment, we propose an architecture having Cisco UCS B-series servers, with nodes dedicated for Docker Universal Control Plane (UCP) Controller and Docker Trusted Registry (DTR) services within Docker Datacenter. This document also provides a second architecture with a minimal number of servers to implement Docker Datacenter on Cisco UCS using C-Series rack servers. This alternative is a best fit for typical dev/test scenarios for example. Cisco UCS Fabric Interconnects and Cisco Nexus TOR switches are used in both the topologies separately. The Docker Datacenter runs on Red Hat Enterprise Linux Operating System. Cisco UCS servers provide converged and highly available hardware platform centrally managed by Cisco UCS Manager Software residing on Cisco Fabric Interconnects. As an important component of the Docker Datacenter product, Docker Universal Control Plane provides the redundancy and high-availability of the Docker Engine and management control plane. This solution holistically offers container management for diverse application environment to be deployed in DevOps and Production deployments.

Compute nodes for Docker are configured to run on Cisco B- and C-Series servers based on the selected reference architecture. In some cases, compute node resources could be shared between the control plane software stack and the container engine. For production deployments, Docker Universal Control Plane services are spread over three B-Series servers for providing management control plane redundancy and high-availability. DTR services and its replicas are configured to run on three dedicated B-Series servers as part of Docker Datacenter.

In the second architecture Docker UCP and DTR services are co-located on three C-Series rack servers. Docker UCP administrator has a choice to allow distribution of application container work-load to be spread across the Docker UCP controller nodes.

Both Docker UCP and DTR dashboards require an external load balancer to access management interfaces to make them operate in highly available mode. External load balancer provides virtual IP address to front end the UCP and DTR management dashboard separately. External load balancer is not listed as a solution component in this CVD as the load balancer is purely a customer’s choice and configuration follows a standard procedure.

To achieve DTR shared storage high availability for image repository among the DTR cluster nodes, Docker Datacenter requires external NFS setup. This solution uses NFS shared volume configuration for the DTR shared storage.

Sizing Considerations

This solution is designed and proposes two separate topologies based on Docker Datacenter (DDC) production as well as dev/test deployment requirements for the Enterprise.

For the production grade DDC deployment Docker recommends infrastructure services to be run on dedicated hosts. There are two infrastructure components, namely – Docker UCP Controller and DTR. Container workloads are to run on dedicated node termed as Docker UCP Nodes. For production requirements, one each of Docker UCP Controller and DTR service node should be running in cluster of baremetal servers. In order to have a minimum of one node failure tolerance, it is recommended to have three-nodes for running both controller and DTR services separately. Based on this recommendation, Docker UCP Controller and DTR services will run on three-node dedicated clusters. This solution proposes a ten node setup, where six nodes are consumed by the DDC’s infrastructure (controller and DTR) services and the remaining four nodes take up the Container workload. Administrators have an option to configure the UCP Controller nodes to take the container workload based on the deployment requirements. With this configurable item, the entire infrastructure is well optimized for running container workloads.

Cisco C-Series servers have ample memory and CPU resources that allow us to run the Docker UCP and DTR service containers on a same three-node cluster. With this design the overall Docker Datacenter deployment gets optimized to a four-node cluster configuration. The fourth node runs as a UCP node to take the container workload. The scheduler configuration settings allows administrators and users to deploy containers on the UCP controller nodes and using the scheduling strategy set to ‘spread’, we get all the four nodes available for container deployment.

Number of containers run on a node and thus the optimal achievable number from this topology depends on the type of containers and the applications that are run within the containers. This solution is designed and validated to run 300 containers per UCP Node. Adding is a node to the cluster follows a simple procedure with a minimal manual intervention. With a policy based logical server model, achieved through UCS manager, scaling-up the baremetal nodes are just a few clicks away. Service profile templates associated with configured server pool help in automatic deployment of service profiles, as long as there is additional hardware available in the server pool. Node addition workflow is covered in the section Upscale Tests.

The following figure illustrates the physical backend connectivity of the hardware components.

Figure 11     Physical Topology – First Architecture

Figure 12     Cabling Diagram – First Architecture

Figure 13     Physical Topology – Second Architecture

Solution Workflow

This section provides component wise configuration workflow for the solution.

Cisco Nexus 9372PX Configuration

The following workflow shows the high-level configuration steps for Cisco Nexus ToR switches used for north-bound connectivity to the UCSM domain. Application container requiring external data plane access goes through these ToR switches. The SVI for the tenant VLAN for external access is configured on these switches with L3 feature enablement.

For configuration details, refer to the Cisco Nexus 9000 Series switches configuration guides: http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/products-installation-andconfiguration-guides-list.html

Cisco UCS Manager Configuration

Cisco UCS Manager provides policy based infrastructure management for compute nodes provisioned for various services in this solution. Cisco UCSM is responsible for entire hardware life-cycle management through user defined policy implementation on the compute nodes. Some of the key features include Firmware Management, Faults/Alerts/Stats reporting, Maintenance scheduling, Identity management, Local Disk configuration, Network connectivity and QoS. The compute node hardware characteristics are applied to physical Blade servers through logical configuration known as Service Profile. This logical entity is a collection of various UCSM managed pools and policies to enable compute nodes to utilize hardware components optimally. The concept of stateless computing facilitates much greater scalability and greatly reduces application downtime in the event of hardware failure. The following workflow shows high-level tasks to bring up an UCS compute node. This also enables administrators to quickly provision the hardware resources in a short period of time and drastically reduces the application container bring up time.

 

Docker Datacenter Installation

Host setup day zero automation tasks are run on all the participating server nodes in the cluster using Ansible automation tool. This tool does not use a client/server model and does not need an additional build server. Any node can be initiated to run minimal host setup tasks even before the Docker datacenter is deployed on the nodes.

Software and Hardware Versions

The following tables provide software and hardware versions used in this solution for both the architectures.

Table 7 lists the Cisco UCS infrastructure components used in the solution for production deployment.

Table 7    Solution Component Details for Production Deployment

 

            Table 8 lists the hardware and software versions used in the solution for Production deployment.

Table 8    Hardware and Software Versions for Production Deployment

 

Table 9 lists the Cisco UCS infrastructure components used in the solution for second architecture.

Table 9    Solution Component Details for Second Architecture

 

Table 10 lists the hardware and software versions used in the solution for second architecture.

Table 10      Hardware and Software Versions for Second Architecture

Cisco Nexus 9372PX

Initial Configuration and Setup

This section outlines the initial configuration necessary for bringing up a new Cisco Nexus 9000.

Cisco Nexus A

To set up the initial configuration for the first Cisco Nexus switch complete the following steps:

1.    Connect to the serial or console port of the switch

Enter the configuration method: console

Abort Auto Provisioning and continue with normal setup? (yes/no[n]: y

---- System Admin Account Setup ----

Do you want to enforce secure password standard (yes/no[y] :

Enter the password for "admin":

Confirm the password for "admin":

---- Basic System Configuration Dialog VDC: 1 ----

This setup utility will guide you through the basic configuration of the system. Setup configures only enough connectivity for management of the system.

Please register Cisco Nexus9000 Family devices promptly with your supplier. Failure to register may affect response times for initial service calls. Nexus9000 devices must be registered to receive entitled support services.

Press Enter at anytime to skip a dialog. Use ctrl-c at anytime to skip the remaining dialogs.

Would you like to enter the basic configuration dialog (yes/no): y

Create another login account (yes/no) [n]: n

Configure read-only SNMP community string (yes/no) [n]:

Configure read-write SNMP community string (yes/no) [n]:

Enter the switch name: Docker-N9K-A

Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]:

Mgmt0 IPv4 address: 10.65.121.54

Mgmt0 IPv4 netmask: 255.255.255.0  
 Configure the default gateway? (yes/no) [y]:

IPv4 address of the default gateway: 192.168.155.1  

Configure advanced IP options? (yes/no) [n]:

Enable the telnet service? (yes/no) [n]:

Enable the ssh service? (yes/no) [y]:

Type of ssh key you would like to generate (dsa/rsa) [rsa]:

Number of rsa key bits <1024-2048> [1024]: 2048

Configure the ntp server? (yes/no) [n]: y

NTP server IPv4 address: 10.65.121.54

Configure default interface layer (L3/L2) [L2]:

Configure default switchport interface state (shut/noshut) [noshut]:

Configure CoPP system profile (strict/moderate/lenient/dense/skip) [strict]:

15.  Review the settings printed to the console. If they are correct, answer yes to apply and save the configuration

16.  Wait for the login prompt to make sure that the configuration has been saved prior to proceeding.

Cisco Nexus B

To set up the initial configuration for the second Cisco Nexus switch complete the following steps:

1.    Connect to the serial or console port of the switch

2.    The Cisco Nexus B switch should present a configuration dialog identical to that of Cisco Nexus A shown above. Provide the configuration parameters specific to Cisco Nexus B for the following configuration variables. All other parameters should be identical to that of Cisco Nexus A.

·         Admin password

·         Nexus B Hostname: Docker-N9K-B

·         Nexus B mgmt0 IP address: 10.65.121.55

·         Nexus B mgmt0 Netmask: 255.255.255.0

·         Nexus B mgmt0 Default Gateway: 192.168.155.1

Feature Enablement

The following commands enable the IP switching feature and set default spanning tree behaviors:

3.    On each Nexus 9000, enter the configuration mode:

 config terminal

4.    Use the following commands to enable the necessary features:

feature udld

feature lacp

feature vpc

feature interface-vlan

5.    Configure the spanning tree and save the running configuration to start-up:

spanning-tree port type network default

spanning-tree port type edge bpduguard default

spanning-tree port type edge bpdufilter default

copy run start

vLAN Creation

To create the necessary virtual local area networks (VLANs), complete the following step on both switches:

From the configuration mode, run the following commands:

vlan 603

name vlan603

Configure vPC

Configuring vPC Domain

Cisco Nexus A

To configure virtual port channels (vPCs) for switch A, complete the following steps:

1.    From the global configuration mode, create a new vPC domain:

vpc domain 10

2.    Make Cisco Nexus A the primary vPC peer by defining a low priority value:

role priority 10

3.    Use the management interfaces on the supervisors of the Cisco Nexus switches to establish a keepalive link:

peer-keepalive destination 10.65.121.55 source 10.65.121.54

4.    Enable following features for this vPC domain:

peer-switch
delay restore 150
peer-gateway
ip arp synchronize
auto-recovery

5.    Save the configuration.

copy run start

Cisco Nexus B

To configure vPCs for switch B, complete the following steps:

1.    From the global configuration mode, create a new vPC domain:

vpc domain 10

2.    Make Cisco Nexus A the primary vPC peer by defining a higher priority value on this switch:

role priority 20

3.    Use the management interfaces on the supervisors of the Cisco Nexus switches to establish a keepalive link:

peer-keepalive destination 10.65.121.54 source 10.65.121.55

4.    Enable following features for this vPC domain:

peer-switch
delay restore 150
peer-gateway
ip arp synchronize
auto-recovery

5.    Save the configuration:

copy run start

Configuring Network Interfaces for VPC Peer Links

Cisco Nexus A

1.    Define a port description for the interfaces connecting to VPC Peer Docker-N9K-B. 

interface Eth1/9
description VPC Peer Docker-N9K-B:e1/10
interface Eth1/10
description VPC Peer Docker-N9K-B:e1/9

2.    Apply a port channel to both VPC Peer links and bring up the interfaces.

interface Eth1/9,Eth1/10
channel-group 11 mode active
no shutdown

3.    Enable UDLD on both interfaces to detect unidirectional links.

udld enable

4.    Define a description for the port-channel connecting to Docker-N9K-B.

interface port-channel 11
description vPC peer-link

5.    Make the port-channel a switchport, and configure a trunk to allow in-band management, VM traffic, and the native VLAN.

switchport
switchport mode trunk
switchport trunk native vlan 603
spanning-tree port type network

6.    Make this port-channel the VPC peer link and bring it up.

vpc peer-link
no shutdown
copy run start

Cisco Nexus B

1.    Define a port description for the interfaces connecting to VPC Peer Docker-N9K-A. 

interface Eth1/9
description VPC Peer Docker-N9K-A:e1/10
interface Eth1/10
description VPC Peer Docker-N9K-A:e1/9

2.    Apply a port channel to both VPC Peer links and bring up the interfaces.

interface Eth1/9,Eth1/10
channel-group 11 mode active
no shutdown

3.    Enable UDLD on both interfaces to detect unidirectional links.

udld enable

4.    Define a description for the port-channel connecting to Docker-N9K-A.

interface port-channel 11
description vPC peer-link

5.    Make the port-channel a switchport, and configure a trunk to allow in-band management, VM traffic, and the native VLAN.

switchport
switchport mode trunk
switchport trunk native vlan 603
spanning-tree port type network

6.    Make this port-channel the VPC peer link and bring it up.

vpc peer-link
no shutdown
copy run start

Configure Network Interfaces

Cisco Nexus A

1.    Define a description for the port-channel connecting to Docker-FI-A.

interface port-channel 12
description Docker-FI-A

2.    Make the port-channel a switchport, and configure a trunk to allow in-band management, VM traffic, and the native VLANs.

switchport mode trunk
switchport trunk native vlan 603

3.    Make the port channel and associated interfaces spanning tree edge ports.

spanning-tree port type edge trunk
spanning-tree guard root
no lacp graceful-convergence

4.    Make this a VPC port-channel and bring it up.

vpc 12
no shutdown

5.    Define a port description for the interface connecting to Docker-FI-A.

interface Eth1/11

6.    Apply it to a port channel and bring up the interface.

channel-group 12 mode active
no shutdown

7.    Enable UDLD to detect unidirectional links.

udld enable

8.    Define a description for the port-channel connecting to Docker-FI-B.

interface port-channel
description Docker-FI-B

9.    Make the port-channel a switchport, and configure a trunk to allow in-band management, VM traffic VLANs and the native VLAN.

switchport mode trunk
switchport trunk native vlan 603

10.  Make the port channel and associated interfaces spanning tree edge ports.

spanning-tree port type edge trunk
spanning-tree guard root
no lacp graceful-convergence

11.  Make this a VPC port-channel and bring it up.

vpc 13
no shutdown

12.  Define a port description for the interface connecting to Docker-FI-B.

interface Eth1/12

13.  Apply it to a port channel and bring up the interface.

channel-group 13 mode active
no shutdown

14.  Enable UDLD to detect unidirectional links.

udld enable

copy run start

Cisco Nexus B

1.    Define a description for the port-channel connecting to Docker-FI-B.

interface port-channel 12
description Docker-FI-B

2.    Make the port-channel a switchport, and configure a trunk to allow in-band management, VM traffic, and the native VLANs.

switchport mode trunk
switchport trunk native vlan 603

3.    Make the port channel and associated interfaces spanning tree edge ports.

spanning-tree port type edge trunk
spanning-tree guard root
no lacp graceful-convergence

4.    Make this a VPC port-channel and bring it up.

vpc 12
no shutdown

5.    Define a port description for the interface connecting to Docker-FI-B.

interface Eth1/11

6.    Apply it to a port channel and bring up the interface.

channel-group 12 mode active
no shutdown

7.    Enable UDLD to detect unidirectional links.

udld enable

8.    Define a description for the port-channel connecting to Docker-FI-A.

interface port-channel 13
description Docker-FI-A

9.    Make the port-channel a switchport, and configure a trunk to allow in-band management, and VM traffic VLANs and the native VLAN.

switchport mode trunk
switchport trunk native vlan 603

10.  Make the port channel and associated interfaces spanning tree edge ports.

spanning-tree port type edge trunk
spanning-tree guard root
no lacp graceful-convergence

11.  Make this a VPC port-channel and bring it up.

vpc 13
no shutdown

12.  Define a port description for the interface connecting to Docker-N9K-A.

interface Eth1/12

13.  Apply it to a port channel and bring up the interface.

channel-group 13 mode active
no shutdown

14.  Enable UDLD to detect unidirectional links.

udld enable

copy run start

Cisco UCS Manager

Initial Setup of Cisco Fabric Interconnects

A pair of Cisco UCS 6248UP Fabric Interconnects is used in this design. The minimum configuration required for bringing up the FIs and the embedded Cisco UCS Manager (UCSM) is outlined below. All configurations after this will be done using Cisco UCS Manager.

Cisco UCS 6248UP FI – Primary (FI-A)

1.    Connect to the console port of the primary Cisco UCS FI.

Enter the configuration method: console
Enter the setup mode; setup newly or restore from backup.(setup/restore)? Setup You have chosen to setup a new fabric interconnect? Continue? (y/n): y
Enforce strong passwords? (y/n) [y]: y
Enter the password for "admin": <Enter Password>
Enter the same password for "admin": <Enter Password>
Is this fabric interconnect part of a cluster (select 'no' for standalone)? (yes/no) [n]: y
Which switch fabric (A|B): A
Enter the system name: Docker-FI
Physical switch Mgmt0 IPv4 address: 10.65.122.130
Physical switch Mgmt0 IPv4 netmask: 255.255.255.0
IPv4 address of the default gateway: 10.65.122.1
Cluster IPv4 address: 10.65.122.132
Configure DNS Server IPv4 address? (yes/no) [no]: y
DNS IPv4 address: 171.70.168.183
Configure the default domain name? y
Default domain name: <domain name>
Join centralized management environment (UCS Central)? (yes/no) [n]: <Enter>

2.    Review the settings printed to the console. If they are correct, answer yes to apply and save the configuration.

3.    Wait for the login prompt to make sure that the configuration has been saved prior to proceeding.

Cisco UCS 6248UP FI – Secondary (FI-B)

1.    Connect to the console port on the second FI on Cisco UCS 6248UP FI.

Enter the configuration method: console
Installer has detected the presence of a peer Fabric interconnect. This Fabric interconnect will be added to the cluster.  Do you want to continue {y|n}?   y
Enter the admin password for the peer fabric interconnect: <Enter Password>
Peer Fabric interconnect Mgmt0 IPv4 address: 10.65.122.130

Peer Fabric interconnect Mgmt0 IPv4 netmask: 255.255.255.0

Cluster IPv4 address: 10.65.122.131
Apply and save the configuration (select ‘no’ if you want to re-enter)?(yes/no): y

2.  Verify the above configuration by using Secure Shell (SSH) to login to each FI and verify the cluster status. Status should be as follows if the cluster is up and running properly.

Docker-FI-A# show cluster state

Now you are ready to log into Cisco UCS Manager using either the individual or cluster IPs of the Cisco UCS Fabric Interconnects.

Configure Ports for Server, Network and Storage Access

Logging into Cisco UCS Manager

To log into the Cisco Unified Computing System (UCS) environment, complete the following steps:

1.    Open a web browser and navigate to the Cisco UCS 6248 Fabric Interconnect cluster IP address configured in earlier step.

2.    Click Launch Cisco UCS Manager link to download the Cisco UCS Manager software.

3.    If prompted, accept security certificates as necessary.

4.    When prompted, enter admin as the user name and enter the administrative password.

5.    Click Login to log in to Cisco UCS Manager.

6.    Select Yes or No to authorize Anonymous Reporting if desired and click OK.

Cisco UCS Manager – Synchronize to NTP

To synchronize the Cisco UCS environment to the NTP server, complete the following steps:

1.    From Cisco UCS Manager, click Admin tab in the navigation pane.

2.    Select All > Timezone Management > Timezone.

3.    Right-click and select Add NTP Server.

4.    Specify NTP Server IP (for example, 171.68.38.66) and click OK twice to save edits. The Time Zone can also be specified in the Properties section of the Time Zone window.

Upgrading Cisco UCS Manager

This document assumes that the Cisco UCS Manager is running the version outlined in the Software Matrix. If an upgrade is required, follow the procedures outlined in the Cisco UCS Install and Upgrade Guides.

Assigning Block of IP addresses for KVM Access

To create a block of IP addresses for in-band access to servers in the Cisco UCS environment, complete the following steps. The addresses are used for Keyboard, Video, and Mouse (KVM) access to individual servers managed by Cisco UCS Manager.

1.    From Cisco UCS Manager, click LAN tab in the navigation pane.

2.    Select LAN > Pools > root > IP Pools.

3.    Right-click and select Create IP Pool.

4.    Specify a Name (for example, ext-mgmt) for the pool. Click Next.

5.    Click [+] Add to add a new IP Block. Click Next.

6.    Enter the starting IP address (From), the number of IP addresses in the block (Size), the Subnet Mask, Default Gateway and DNS information. Click OK.

7.    Click Finish to create the IP block.

Editing Chassis Discovery Policy

Setting the discovery policy simplifies the addition of Cisco UCS Blade Server chassis and Cisco Fabric Extenders. To modify the chassis discovery policy, complete the following steps:

1.    From Cisco UCS Manager, click Equipment tab in the navigation pane and select Equipment in the list on the left.

2.    In the right pane, click Policies tab.

3.    Under Global Policies, set the Chassis/FEX Discovery Policy to match the number of uplink ports that are cabled between the chassis or fabric extenders (FEXes) and the fabric interconnects.

4.    Set the Link Grouping Preference to Port Channel.

5.    Click Save Changes and then OK to complete.

Acknowledging Cisco UCS Chassis

To acknowledge all Cisco UCS chassis, complete the following steps:

1.    From Cisco UCS Manager, click Equipment tab in the navigation pane.

2.       Expand Chassis and for each chassis in the deployment, right-click and select Acknowledge Chassis.

3.       In the Acknowledge Chassis pop-up, click Yes and then click OK.

Enabling Server Ports

To configure ports connected to Cisco UCS servers as Server ports, complete the following steps:

1.    From Cisco UCS Manager, click Equipment tab in the navigation pane.

2.       Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3.    Expand Ethernet Ports.

4.    Select the ports that are connected to Cisco UCS Blade server chassis. Right-click and select Configure as Server Port.

5.    Click Yes and then OK to confirm the changes.

6.    Repeat above steps for Fabric Interconnect B (secondary) ports that connect to servers.

7.    Verify that the ports connected to the servers are now configured as server ports. The view below is filtered to only show Server ports.

Enabling Uplink Ports to Cisco Nexus 9000 Series Switches

To configure ports connected to Cisco Nexus switches as Network ports, complete the following steps:

1.    From Cisco UCS Manager, click Equipment tab in the navigation pane.

2.       Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3.    Expand Ethernet Ports.

4.    Select the first port (for example, Port 11) that connects to Cisco Nexus A switch, right-click and select Configure as Uplink Port > Click Yes to confirm the uplink ports and click OK. Repeat for second port (for example, Port 16) that connects to Cisco Nexus B switch. 

5.    Repeat above steps for Fabric Interconnect B (secondary) uplink ports that connect to Cisco Nexus A and B switches.

Verify that the ports connected to the servers are now configured as server ports. The view below is filtered to only show Network ports.

Configuring Port Channels on Uplink Ports to Cisco Nexus 9000 Series Switches

In this procedure, two port channels are created, one from Fabric A to both the Cisco Nexus switches and one from Fabric B to both the Cisco Nexus switches.

To configure port channels on Uplink/Network ports connected to Cisco Nexus switches, complete the following steps:

1.    From Cisco UCS Manager, click LAN tab in the navigation pane.

2.    Select LAN > LAN Cloud > Fabric A > Port Channels.

3.    Right-click and select Create Port Channel.

4.    In the Create Port Channel window, specify a Name and unique ID.

5.    In the Create Port Channel window, select the ports to put in the channel (for example, Eth1/11 and Eth1/12). Click Finish to create the port channel.

6.    Verify the resulting configuration.

7.    Repeat above steps for Fabric B and verify the configuration.

Cisco UCS Configuration – LAN

Creating VLANs

Complete these steps to create necessary VLANs. 

1.    From Cisco UCS Manager, click LAN tab in the navigation pane.

2.    Select LAN > LAN Cloud > VLANs.

3.    Right-click and select Create VLANs. Specify a name (for example, 603) and VLAN ID (for example, 603).

4.    If the newly created VLAN is a native VLAN, select VLAN, right-click and select Set as Native VLAN from the list. Either option is acceptable, but it needs to match what the upstream switch is set to.

Creating LAN Pools

Creating MAC Address Pools

The MAC addresses in this pool will be used for traffic through Fabric Interconnect A and Fabric Interconnect B.

1.    From Cisco UCS Manager, click LAN tab in the navigation pane.

2.    Select LAN > Pools > root > MAC Pools.

3.    Right-click and select Create Mac Pool.

4.    Specify a name (for example, Docker) that identifies this pool.

5.    Leave the Assignment Order as Default and click Next.

6.    Click [+] Add to add a new MAC pool.

7.    For ease-of-troubleshooting, change the 4^th and 5^th octet to AA:AA traffic using Fabric Interconnect A. Generally speaking, the first three octets of a mac-address should not be changed.

8.    Select a size (for example, 24) and select OK and then click Finish to add the MAC pool.

Creating LAN Policies

Creating vNIC Templates

To create virtual network interface card (vNIC) templates for Cisco UCS hosts, complete the following steps. Two vNICs are created for redundancy – one through Fabric A and another through Fabric B. All host traffic is carried across these two vNICs in this design.

Creating vNIC Template for Fabric A

1.    From Cisco UCS Manager, select LAN tab in the navigation pane.

2.    Select LAN > Policies > root > vNIC Templates. 

3.    Right-click and select Create vNIC Template.

4.    Specify a template Name (for example, Docker-eth0) for the policy.

5.    Keep Fabric A selected and keep Enable Failover checkbox checked.

6.    Under Target, make sure that the VM checkbox is NOT selected.

7.    Select Updating Template as the Template Type.

8.    Under VLANs, select the checkboxes for all VLAN traffic that a host needs to see (for example, 603) and select the Native VLAN radio button.  

9.    For CDN Source, select User Defined radio button. This option ensures that the defined vnic name gets reflected as the adapter’s network interface name during OS installation.

10.  For CDN Name, enter a suitable name.

11.  Keep the MTU as 1500. 

12.  For MAC Pool, select the previously configured LAN pool (for example, Docker).

13.  Choose the default values in the Connection Policies section.

14.  Click OK to create the vNIC template.

Creating vNIC Template for Fabric B

Repeat the above steps to create a vNIC template (for example, Docker-eth0) through Fabric B.

Cisco UCS Configuration – Server

Creating Server Policies

In this procedure, various server policies that are used in this solution are created.

Creating BIOS Policy

To create a server BIOS policy for Cisco UCS hosts, complete the following steps:

1.    In Cisco UCS Manager, click Servers tab in the navigation pane.

2.    Select Policies > root > BIOS Policies.

3.    Right-click and select Create BIOS Policy.

4.    In the Main screen, enter BIOS Policy Name (for example, Docker-BiosPol) and change the Consistent Device Naming to enabled. Click Next.

5.    Keep the other options in all the other tabs at Platform Default.

6.    Click Finish and OK to create the BIOS policy.

Creating Boot Policy

To create the boot policy, complete the following steps:

1.    In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.    Select Policies > root > Boot Policies.

3.    Right-click and select Create Boot Policy.

4.    In the Create Boot Policy window, enter the policy name (for example, Docker-LocalBoot).

5.    Expand the Local Devices section of the window and select Add CD/DVD. The Local CD/DVD and Remote CD/DVD will get greyed out.

6.    Now select Add Local Lun under Add Local Disk. In the pop-up window select Primary radio button and for the Lun Name enter boot lun name (for example, Boot-Lun). Click OK to add the local lun image.

7.    After the creation Boot Policy, you can view the created boot options as shown.

Creating Host Firmware Package Policy

Firmware management policies allow the administrator to select the corresponding packages for a given server configuration. These policies often include packages for adapter, BIOS, board controller, FC adapters, host bus adapter (HBA) option ROM, and storage controller properties. To create a firmware management policy for a given server configuration in the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click Servers tab in the navigation pane.

2.    Select Policies > root > Host Firmware Packages.

3.    Right-click on Host Firmware Packages and select Create Host Firmware Package.

4.    Enter the name of the host firmware package (for example, 3.1.2c).

5.    Leave Simple selected.

6.    Select the package versions for the different type of servers (Blade, Rack) in the deployment (for example, 3.1(2c) for Blade and Rack servers.

7.    Click OK twice to create the host firmware package.

Creating Server Pools

Creating UUID Suffix Pool

To configure the necessary universally unique identifier (UUID) suffix pool for the Cisco UCS environment, complete the following steps:

1.    From Cisco UCS Manager, select Servers tab in the navigation pane.

2.    Select Servers > Pools > root > UUID Suffix Pools.

3.    Right-click and select Create UUID Suffix Pool.

4.    Specify a Name (for example, Docker) for the UUID suffix pool and click Next.

5.    Click [+] Add to add a block of UUIDs. Alternatively, you can also modify the default pool and allocate/modify a UUID block.

6.    Keep the From field at the default setting. Specify a block size (for example, 12) that is sufficient to support the available blade or server resources.

7.    Click OK, click Finish and click OK again to create UUID Pool.

Creating Server Pools

We have created three server pools, one each for DTR nodes, UCP controller nodes, and UCP nodes. Since we have three DTR nodes, three UCP-Ctrl nodes and 4 UCP nodes in our solution, we have created separate pools for each of these categories coming from two different chassis.

To configure the necessary server pool for the Cisco UCS environment, complete the following steps:

1.    In Cisco UCS Manager, click Servers tab in the navigation pane.

2.       Select Pools > root.

3.       Right-click Server Pools and select Create Server Pool.

4.       Enter name of the server pool (for example, Docker-DTR).

5.       Optional: Enter a description for the server pool.

6.       Click Next.

7.       Select two (or more) servers to be used for the VMware management cluster and click >> to add them to the server pool.

8.    Click Finish to complete.

9.    Similarly create two more Server Pools (for example, Docker-Ctrl and Docker-UCP-Node). The created Server Pools can be viewed under Server Pools.

 

Cisco UCS Configuration – Storage

Creating Storage Profile

Storage Profiles provide a systematic way to automate the steps for provisioning Disk Groups, RAID Levels, LUNs, boot drives, hot spares, and other related resources. They are used in combination with Service Profile Templates to map the associations between logically defined storage resources and servers.

Having a storage profile created will reduce the task of configuring two virtual disks in the RAID Controller Option ROM or create a custom file system layout at the time of OS installation.

We have created a Storage profile with two local luns one each for boot and data. Complete the following to create a storage profile:

1.    In Cisco UCS Manager, click Storage tab in the navigation pane.

2.    Select Storage > Storage Profiles.

3.    Right-click Storage Profiles and select Create Storage Profile.

4.    Enter the name for the Storage Profile (for example, Docker-StgProf).

5.    In the Local Luns tab, click + on the right plane of the Create Storage Profile Window.

6.    Create Local Lun window appears. Keep the Create Local Lun radio button selected.

7.    Enter the lun name (for example, Boot-Lun) and specify the desired size (for example, 60GB).

8.    For select Disk Group configuration, Click Create Disk Group Policy.

9.    Enter the Disk Group name (for example, Docker-DG). Keep the RAID Level as RAID 1 Mirrored, since blades come with only 2 disks.

10.  Select Disk Group Configuration (Manual) radio button. Click + to add two slots; one for Boot-Lun and the other for Data-Lun. Keep the other fields as is and click OK.

11.  Repeat step 10 to create another Local Disk Configuration Reference. Click OK.

12.  Select the created disk group from the Select Disk Group Configuration drop-down (for example, Docker-DG). Click OK to create Boot-Lun.

13.  Repeat step 7 to create Data-Lun. Enter the appropriate name (for example, Data-Lun) and size (for example, 200GB) and select Disk Group Configuration (for example, Docker-DG). Click OK to create Data-Lun.

14.  The created Local Luns can be view under Storage Profiles and Disk Group Policy under Storage Policies.

Creating Service Profile Templates

In this procedure, three service profile templates are created: one each for DTR nodes, UCP controller nodes and UCP nodes. The first profile template is created, then cloned and renamed for the second and third profiles. Since there are three DTR, three UCP controller and four UCP nodes, we instantiate service profiles for these categories from the three different service profile templates.

Creating Service Profile Template

To create service profile templates (for example, DTR nodes), complete the following steps:

1.    From Cisco UCS Manager, click Servers tab in the navigation pane.

2.    Select Servers > Service Profile Template > root.

3.    Right-click root and select Create Service Profile Template to open the Create Service Profile Template wizard.

4.    In the Identify the Service Profile Template screen, configure the following:

a.     Enter name (for example, Docker-DTR) for the service profile template.

b.    Select Updating Template radio button.

c.     Under UUID, select the previously configured UUID pool (for example, Docker).

d.    Click Next.

5.    In the Storage Provisioning screen, configure the following:

a.     Go to Storage Profile Policy tab.

b.    In the Storage profile drop-down, select a policy. Choose the previously configured policy (for example, Docker-StgProf). Local Luns tab lists the previously configured local luns.

c.     Click Next.

6.    In the Networking screen, configure the following:

a.     Restore the default setting for Dynamic vNIC Connection Policy.

b.    Click Expert radio button to configure the LAN connectivity.

c.     Click on [+] Add to add a vNIC to the template.

d.    In the Create vNIC dialog box:

-        Enter the name (for example, eth0) of the vNIC.

-        Check the Use vNIC Template check box.

-        In the vNIC Template list, choose the previously created vNIC Template for Fabric A boot (for example, Docker-eth0).

-        In the Adapter Policy list, choose Linux.

-        Click OK to add this vNIC to the template.

e.    Click on [+] Add to add a 2^nd vNIC to the template.

f.      In the Create vNIC dialog box:

-        Enter the name (for example, eth1) of the vNIC.

-        Check the Use vNIC Template check box.

-        In the vNIC Template list, choose the previously created vNIC Template for Fabric B boot (for example, Docker-eth1).

-        In the Adapter Policy list, choose Linux.

-        Click OK to add this vNIC to the template.

g.    Review the configuration on the Networking screen of the wizard. Make sure that both the vNICs were created. Click Next.

7.    Click Next in the SAN Connectivity, Zoning, vNIC/vHBA Placement, and vMedia Policy screens.

8.    In the Set Boot Order screen, select the previously created boot policy from the Boot Policy drop-down (for example, Docker-LocalBoot).

9.    Click Next.

10.  Click Next in Maintenance Policy screen.

11.  In the Server Assignment screen, configure the following:

a.     For Pool Assignment, choose the previously created policy from the list (for example, Docker-DTR).

b.    Leave the Power State as UP for when the Profile is applied to a server

c.     For Server Pool Qualification, select the previously created policy from the list (for example, all-chassis).

d.    Expand the Firmware Management section. For the Host Firmware Package, select the previously selected policy from the list (for example, 3.1.2c).

e.    Click Next.

12.  In the Operation Policies screen, configure the following:

a.     For the BIOS Policy list, select the previously configured policy (for example, Docker-BiosPol).

13.  Expand Management IP Address, select the IP address pool (for example, ext-mgmt(0/18)) from the management IP Address policy drown-down.

14.  Click Finish to complete the creation of the Service Profile Template. Created service profile template will get listed under Service Profile Templates as shown in the below figure.

 

Creating Service Profile Template for UCP Controller Nodes

Repeat the steps 1 to 14 detailed in the previous sub-section for creating a service profile template for UCP controller nodes.

After creating the service profile template, the template for UCP controller nodes will look similar to the below screenshot.

Creating Service Profile Template for UCP Nodes

Repeat the steps 1 to 14 detailed in the previous sub-section for creating a service profile template for UCP nodes.

After creating the service profile template, the template for UCP nodes will look similar to the below screenshot.

Service Profile Instantiation and Association

Service Profile Instantiation

In this process we have instantiated service profiles for DTR, UCP controller and UCP nodes from their respective templates that we created in the previous section.

To create service profiles from template, complete the following steps:

1.    From Cisco UCS Manager, click Servers tab in the navigation pane.

2.    Expand Servers > Service Profile Templates.

3.    Right-click on the specific template (for example, Docker-DTR) and select Create Service Profiles from Template to open the Create Service Profile window.

4.    In the Create Service Profile window, enter the profile name (for example, DDC-DTR-), enter the suffix to start the instances and enter the number of instances to be instantiated.

5.    Similarly instantiate other two service profiles (for example, Docker-UCP-Ctrl and Docker-UCP-Nodes.

Associating Service Profile Templates to Server Pools

In this process we associate the three service profile templates to the previously created server pools for DTR, UCP controller and UCP nodes.

Follow the steps to complete server pool association:

1.    From Cisco UCS Manager, click Servers tab in the navigation pane.

2.    Expand Servers > Service Profile Templates.

3.    Right-click on the specific template (for example, Docker-DTR) and select Associate with Server Pools to open the Associate with Server Pool window.

4.    Select the specific sever pool that you want to associate the template to (for example, Docker-DTR).

5.    Right-click on the specific template (for example, Docker-DTR) and select Associate with Server Pools to open the Associate with Server Pool window. Select the Server Pool Qualification as all-chassis as we have the DTR, UCP controller and UCP nodes distributed in two different chassis.

6.    Click OK. A pop-up window will open prompting you to confirm the server pool association.

7.    Similarly associate the other two templates to the specific server pools (for example, Docker-UCP-Ctrl and Docker-UCP-Node) as shown below.

 

8.    After the server pool association, you can view all the services profiles shown in associated state as shown in below.

Installation of Red Hat Enterprise Linux Operating System

After the service profile association, Red Hat Enterprise Linux 7.2 Operating System is installed on all the blade servers. The following section provides detailed procedure for installing Red Hat Linux 7.2 OS on the Boot-Lun created using UCSM Storage Profile.

Complete the following to install Red Hat Enterprise Linux 7.2 OS.

1.    From Cisco UCS Manager, click Servers tab in the navigation pane.

2.    Expand Servers > Service Profile Templates.

3.    Select a node (for example, UCP-Ctrl-1) and click KVM Console.

4.    In the KVM Console select Virtual Media tab. From the drop-down menu select Activate virtual Devices.

5.    Once the virtual devices are activated, select Map CD/DVD option.

6.    Click Browse to locate the Red Hat Enterprise Linux Server 7.2 installer ISO image file.

7.    The image gets mapped to CD/DVD.

8.    In the KVM window, select the KVM tab to monitor during boot.

9.    In the KVM window, select the Macros > Static Macros > Ctrl-Alt-Del button in the upper left corner.

10.  Click OK.

11.  Click OK to reboot the system.

12.  On reboot, the machine detects the presence of the Red Hat Enterprise Linux Server 7.2 install media. At the prompt press F6 to select the Boot Device. Select Cisco vKVM-Mapped vDVD.

13.  Select the option - Install Red Hat Enterprise Linux 7.2.

14.  Click Date and Time and set the local time. And click Done.

15.  Click Software Selection. Under Base Environment, select the option Server with GUI. Select the necessary add-ons for the selected environment. Click Done.

16.  Click Installation Destination. In the Installation Destination window select the 60GB Lun for installing OS. Click Done.

 

17.  Click Network and Host. Select the Ethernet interface eno1 and enter the host name (for example, UCP-Ctrl-1.cisco.com). Click Configure.. and enter the IPv4 address, subnet mask and Gateway, and DNS name.

18.  Click Done. And click Begin Installation.

19.  While the installation in is progress, click Root Password to assert a password and then click User Creation to set user credentials.

20.  Once the installation is complete, upon prompting click Reboot.

Docker Datacenter Installation

Docker Datacenter is a software subscription that includes 3 products:

·         Docker CS Engine

·         Docker Trusted Registry

·         Docker Universal Control Plane

Both Docker UCP and DTR can be installed on-premises or on cloud as part of the installation of Docker Datacenter (DDC) on Cisco UCS blades running a baremetal operating system. Operating system does not need any additional libraries or software stack. Stock OS is sufficient to start and all the required software bits get installed as part of Docker Engine installation.

In the second architecture we have co-located the UCP Controller and DTR services together. We need to run UCP services on a TCP port other than 443. As the DTR services also use the same port for accessing its dashboard. In this alternative UCP is installed with 4443 and DTR will remain on default port 443.

There are mainly 3 steps to bring up the entire DDC on a cluster of baremetal hosts. We are required to install OS followed by Docker CS Engine and DDC. Support matrix for DDC includes RHEL 7.2 release. We have used the same in this solution.

Registering Nodes and Updating Host OS

1.    After the first reboot of the nodes we need to subscribe the nodes to Red Hat Subscription Manager (RHSM) for getting the latest operating system software updates. This step requires a valid subscription license with Red Hat. These steps are to be performed on all the cluster nodes.

2.    Attach nodes to Red Hat Enterprise Linux Server product pool:

3.    Verify that the release is set to a specific version of the operating system and disable all the software repos that gets attached automatically as part of system subscription. We need to enable only a specific set of repos for the stack:

4.    Enable specific software repos – rhel-7-server-rpms, rhel-7-server-optional-rpms & rhel-7-server-extras-rpms. This step is followed by an ‘yum update’ on all the nodes:

5.    Once yum update is complete, reboot all the nodes.

Installing and Configuring Ansible

In order to speed up post installation tasks on a cluster consisting of 10 nodes, we have used an open source automation tool called Ansible. This powerful tool needs very few steps to configure nodes and get up and running. This tool is not based on a ‘server-client’ model for executing automated tasks. Ansible tool can be installed through EPEL (Extra Packages for Enterprise Linux) repos.

Ansible does not require a build host, any host with in a group of hosts subjected for post install configuration can take a role of Ansible controller node. Ansible controller node does not require any additional software or packages. It's the node from where we run Ansible commands/playbook for automated configuration of all the nodes including the controller nodes itself. Note that the controller node is also part of Docker Datacenter. Steps to install and configure Ansible are listed below:

1.    Generate and populate ssh key of the Ansible controller node to itself and rest of the nodes in the cluster. This is required for password-less ssh login to all the nodes in order to execute configuration tasks.

2.    Using ‘ssh-copy-id’ copy key is generated to all the nodes including control node:

3.    Install epel rpm, for which we have to download latest the ‘epel’ rpm for attaching epel repos to all the cluster nodes:

4.    Copy the rpms on the remaining nodes and install:

5.    Check for the available Ansible version, we have used version 2.2:

6.    Install Ansible using yum on all the nodes:

7.    If the nodes does not have FQDN as hostname, we will have to update the ‘/etc/host’ file with all cluster node entries:

8.    Ansible relies on the concept of host groups, all the configuration tasks are targeted to specific group of hosts. Since all our nodes are similar in nature for software stack deployment, we created a single host group ‘Docker’ as shown below:

To run the other tasks specific to DTR nodes we created a host group for DTR nodes, as shown above.

9.    Verify Ansible is configured correctly on all the nodes by using its ICMP module, you can check if the nodes are reachable:

10.  Populate the /etc/hosts file on the remaining nodes from controller node:

Installing NTP and Configuring Host OS System Clocks

1.    Docker Datacenter requires nodes participating in the cluster to have their system time be in sync with the external source. We do this by installing NTP services and configuring it to remain in sync with the system time:

2.    Add your NTP source in /etc/ntp.conf file and copy the file to all other nodes:

3.    Copy the conf file to all the other nodes:

4.    Enable NTPD and start the service, followed by status check to make sure system clocks are in sync on all the nodes:

5.    Start the NTPD service on all the nodes:

6.    Check the status of the NTPD service on all the nodes:

7.    Querying the NTP source to check on the clock-skew if any:

Installing Cisco Virtual Interface Card (VIC) eNIC (Ethernet Network Interface Card) Driver

RHEL 7.2 comes with inbox eNIC driver for the Cisco VIC, we need to update it to the latest version for the UCS Manager release used in the solution. RPM needs to be extracted from downloaded Cisco UCS B-Series Server Software bundle from cisco.com.

1.    Installing of eNIC driver can be done through rpm install and this requires host reboot.

2.    Copy the rpm file to all the nodes using Ansible:

3.    Use the yum module to install the eNIC driver rpm file on all node through Ansible:

4.    Reboot all the nodes after installing the latest eNIC driver:

 

 

 

 

Configuring Host OS Firewall for required ports

Docker Datacenter requires some TCP and UDP ports to be opened to facilitate communication between its container infrastructure services running on cluster nodes. This needs to be done before installing Docker CS Engine and the Docker UCP. For opening ports on hosts, ‘firewall-cmd’ is used for configuring ‘firewalld-service’ as shown below. For every port type such as TCP and UDP, refer the table below to see the specific ports to be opened and then making them permanent on the host.

Table 1.         TCP and UDP ports to be opened on hosts for Docker UCP

 

The screenshot below shows the command to open the TCP 443 port using firewall-cmd:

Making the opened ports permanent:

For a range of consecutive ports, we can use the command shown below to open the successive ports 2375 to 2376:

Following is an example for opening 4789 UDP port and making the port permanent:

Repeat the commands for the rest of the ports on all the nodes and complete opening the firewall ports. Once this task is complete, restart ‘firewalld-service’ as below:

To confirm the list of ports opened run the following command as shown below:

Installation of Docker Repo and Engine

Before we proceed to install the Docker repo and Docker Engine we need to make sure we have removed the epel repos from all the nodes. This is needed as these repos are not required anymore and can possibly create issue during other software installations.

1.    To remove epel repo from all the nodes:

2.    Do the yum database cleanup:

3.    Install ‘yum-utils’ package:

4.    Add the Docker public key for Docker CS Engine packages on all the hosts:

5.    Add Docker repository on all the hosts:

6.    Execute ‘yum repolist’ command to refresh the host software repos:

7.    Install Docker Engine on all the nodes using Ansible:

8.    Verify that all the nodes have Docker Engine installed successfully:

Configuring Docker CS Engine for Device-Mapper Driver in Direct LVM-Mode

Device Mapper is a kernel-based framework that underpins many advanced volume management technologies on Linux. Docker’s device-mapper storage driver leverages the thin provisioning and snapshotting capabilities of this framework for image and container management. The preferred configuration for production deployments is direct-lvm. This mode uses block devices to create the thin pool. The following procedure shows you how to configure a Docker host to use the device-mapper storage driver in a direct-lvm configuration.

We will be using /dev/sdb device on each node for creating thin pool for Docker use. Each node has 200 GB size lun getting discovered as /dev/sdb, through Cisco UCS Manager storage profile configuration in RAID-1 level.

1.     Create physical volume on /dev/sdb on all the hosts:

2.     Verify that the physical volume gets created cleanly:

3.     Create a volume group named Docker using physical volume which we previously created:

4.     Create a thin pool named thinpool. In this example, the logical data is 95% of the ‘Docker’ volume group size. Leaving 5% free space allows for auto expanding of either the data or metadata if space runs low.

5.     Convert the pool into thinpool:

        

6.     Configure auto-execution of thin pool via lvm profile –

       

We have created Docker-thinpool profile by setting auto-extend threshold to 80% and auto-extend space to 20%. These values are set to make sure that we get 80% of the total available space at one go and later when the utilization reaches 80% fully; we would get 20% of the remaining extended space.

7.     Apply newly created lvm-profile:

8.     Setup LV monitoring:

9.     Configure Docker daemon with specific device-mapper options. Now that storage is configured we need to tell Docker daemon to use it. We do this by creating daemon.jason file with the following configuration parameters and place it as /etc/docker/daemon.json:

9.    If /etc/docker does not exist, create it before-hand on all the nodes and copy the daemon.json:

10.  Enable Docker service and start the service on all the nodes:

11.  Verify that Docker service is up and running on all the nodes:

12.  Verify that the Docker service is running with storage driver set to device-mapper:

13.  Check that the LVs are monitored as configured:

14.  Check to see there aren’t any containers running on any of the cluster nodes:

With this all the cluster nodes are installed with Docker CS Engine and they are configured for installing Docker UCP and DTR services.

This brings us to a state where all cluster nodes are installed with Docker CS Engine and they are configured for installing Docker UCP and DTR services.

Install and Configure Docker UCP Controller Nodes

Docker UCP is a containerized application that requires Docker CS Engine 1.10.0 or above to run. Installation work flow is split into two steps. We identify the first UCP Controller node, install UCP on it and then start adding UCP Controller replicas and UCP Nodes. This way they form a large cluster running Docker Datacenter. Docker Trusted Registry is again a containerized application running on one of the UCP nodes.

1.    Docker UCP installation is one single command as shown below:

This command takes the following parameters:

·         Container name: --name ucp

·         UCP version tag: docker/ucp:1.1.4 *** this is important as we are specifically saying what version needed to be installed. Otherwise ‘docker run’ command will download and install the latest UCP version.

·         UCP URL/Host address: --host-address <ip-address of the 1st Controller Node>

2.    Install licenses to the Docker UCP Controller node:

3.    Follow the instruction to upload the license file by clicking ‘upload License’.

Accessing UCP Dashboard:

This shows that we have one Docker UCP Controller up and running.

Backup Controller CAs and Add UCP Replicas

We take a backup of root-CA certificates and copy them on the rest of the nodes. This enables them to join the existing single node cluster securely. After taking backup we can add Controller replicas and UCP nodes.

1.    Take backup of root-ca to be used when adding replicas and UCP nodes to form cluster:

In case the above command fails then we need to pass the cluster-id instead of using -–interactive switch in the command. To get the cluster-id run the below commands:

2.    Copy root-ca backup to all other nodes:

3.    Login to the UCP Controller node and execute the UCP installation as below:

This command takes the following parameters –

·         To join existing cluster – join

·         Replica or Primary - --replica

·         Root-CA backup - -v /root/backup.tar:/backup.tar

·         UCP Server URL – It's the 1st UCP Controller ip address through which we access UCP dashboard – https://<ip-address>:443

With these switches second controller node will join the first node to form a cluster. We need to follow the same steps for adding the third controller node.

4.    To join the third Controller node, we will issue the same command as above:

5.    After the third node has joined we need to restart the Docker services on controller nodes one by one, starting from the first UCP controller node.

6.    All the three nodes joins to form a cluster, we will have Docker UCP dashboard status shown as below:

 

Add UCP Nodes

In this step we will now continue to add rest of the 7 hosts to the newly formed cluster. Even the DTR nodes will be added as UCP nodes first. For all the remaining nodes, command will be same. And here we are not going to give ‘--replica’ to join the nodes to the cluster.

1.    Joining the first DTR node to the cluster:

2.    Joining the first UCP node to the cluster:

3.    Similarly we need to add the remaining DTR and UCP nodes by logging into them and running same command as shown above. In all cases UCP server URL remains same. Post installation the Docker services must be started on all the cluster nodes.

After all the nodes join the cluster, the UCP Dash board should look as shown in the screenshot below:

Install and Configure DTR and its Replicas

1.    Install DTR application on the designated DTR nodes such as – DTR1, DTR2 and DTR3. Before this we need to open port 80 on all cluster nodes.

2.    Restart the firewalld services:

3.    Download CA certificates on the first UCP Controller node using curl to UCP Controller URL:

4.    To install DTR application on the DTR nodes, execute the following command on the first UCP Controller Node:

Following command parameters are used in this command:

·         Specific DTR versioning tag – docker/dtr:2.0.3

·         UCP URL – 1st UCP Controller URL – --ucp-url https://<ip-address>

·         UCP Node - --ucp-node <1st DTR Node FQDN>

·         DTR External URL - --dtr-external-url <ip address of the 1st DTR Node>

·         UCP CA Certs - --ucp-ca <cert file with path where we downloaded certs using curl>

5.    Join other DTR Nodes as DTR replicas:

6.    We will now add the DTR application to the third DTR Node:

7.    UCP dashboard will show three DTR applications getting installed on three DTR nodes as below:

8.    The DTR containers running on DTR Nodes are:

9.    Login to DTR URL (which is the first DTR Node ip address):

10.  Next step is to configure NFS file system on all three DTR Nodes to have the shared access.

11.  Open firewall ports for NFS services on DTR nodes only:

12.  Mount external NFS file share on all the three DTR Nodes as shown below:

13.  Once the NFS mount configured, DTR dashboard reflects it automatically:

14.  Configure the domain in the DTR dashboard:

15.  Configure DTR domain in the UCP Cluster:

 

Validate Docker UCP and DTR Cluster

This section validates the Docker UCP/DTR cluster deployment. To check the basic sanity follow these steps:

1.    Run ‘docker ps’ to see the status of all infrastructure containers and DTR application containers. There should not be any stopped and/or exited containers:

2.    Verify the Docker device-mapper in direct-lvm mode is functioning correctly and containers are getting thin storage provisioned via Docker thinpool:

Management and Monitoring Interface

Docker Universal Control Plane (UCP) is a Docker Datacenter component, which provides a GUI dashboard and CL interface to manage and monitor the complete cluster nodes in a single plane of management. Docker UCP is installed as an infrastructure service container on all the designated UCP Controller nodes and it is responsible for managing the entire cluster. UCP can be made highly available through configuring an external load balancer. External load balancer keeps the dashboard available in the event of failure of any of the UCP Controller nodes by assigning a VIP that front ends the individual UCP URLs running on UCP Controller nodes. UCP also integrates DTR services and presents them as an application container in the Docker UCP GUI dashboard. The dashboard covers entire cluster lifecycle management as well as containers, images, network and volumes. Management interface enables you to fetch container images to the cluster from where the containers can be deployed using network/storage plugin of user choice. Also, this provides an option to start/stop, kill and remove containers and their associated images, network and volumes. UCP also does application container lifecycle management along with cluster management.

Universal Control Plane has a built-in security and access control. It supports integration options with LDAP and Active Directory. Also, supports Role Based Access Control (RBAC). Only authorized users can access and make changes to the cluster.

Health status and associated monitoring data gets refreshed periodically and gets updated on the dash board accordingly.

GUI Example

CLI Example

     # docker info

Containers: 66

 Running: 63

 Paused: 0

 Stopped: 3

Images: 133

Server Version: swarm/1.2.5

Role: primary

Strategy: spread

Filters: health, port, containerslots, dependency, affinity, constraint

Nodes: 10

 DDC-DTR-1.cisco.com: 10.65.122.64:12376

  └ ID: 7TYN:HVC5:XWFI:SH7Z:AYYW:2GZX:MFLK:TG72:QF4E:BLOH:ZOBX:76EJ

  └ Status: Healthy

  └ Containers: 7 (7 Running, 0 Paused, 0 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:32Z

  └ ServerVersion: 1.12.3-cs4

 DDC-DTR-2.cisco.com: 10.65.122.65:12376

  └ ID: UFDJ:UFNQ:47IE:MIOF:RTNF:UX66:CKJ4:6M2O:5EUC:CRCN:UA4U:CFYH

  └ Status: Healthy

  └ Containers: 7 (7 Running, 0 Paused, 0 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:59Z

  └ ServerVersion: 1.12.3-cs4

 DDC-DTR-3.cisco.com: 10.65.122.66:12376

  └ ID: IUPK:LN4O:PD2N:2TXN:QTDC:ZE6L:YQVB:XJV6:WW6W:GI4M:EYO6:KIBE

  └ Status: Healthy

  └ Containers: 7 (7 Running, 0 Paused, 0 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:46Z

  └ ServerVersion: 1.12.3-cs4

 UCP-Ctrl-1.cisco.com: 10.65.122.61:12376

  └ ID: 3OPE:S2YE:KLPG:GMDH:XTW6:IBHD:RYFF:2XSE:WKZP:BDTP:KDW4:T7NA

  └ Status: Healthy

  └ Containers: 10 (10 Running, 0 Paused, 0 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:45Z

  └ ServerVersion: 1.12.3-cs4

 UCP-Ctrl-2.cisco.com: 10.65.122.62:12376

  └ ID: 7KIA:NZBQ:NMYT:LMYL:CZBD:TWE2:JTHK:45HX:YBWW:KOHC:TEGO:3GQO

  └ Status: Healthy

  └ Containers: 11 (11 Running, 0 Paused, 0 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:37Z

  └ ServerVersion: 1.12.3-cs4

 UCP-Ctrl-3.cisco.com: 10.65.122.63:12376

  └ ID: KU2O:W4OU:TGHX:DGU7:7A2I:J44S:J7ZT:D7LD:HSBI:HCS3:BFEO:HFMV

  └ Status: Healthy

  └ Containers: 10 (10 Running, 0 Paused, 0 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:24Z

  └ ServerVersion: 1.12.3-cs4

 UCP-Node-1.cisco.com: 10.65.122.67:12376

  └ ID: D4F5:NHRE:SA3H:7IZ3:DTYK:HG6V:MJFQ:B446:GX5G:QRLS:6MVG:LYVM

  └ Status: Healthy

  └ Containers: 4 (3 Running, 0 Paused, 1 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:41Z

  └ ServerVersion: 1.12.3-cs4

 UCP-Node-2.cisco.com: 10.65.122.68:12376

  └ ID: 7SHV:S6YR:KTCX:Q722:YBHA:WSBV:QRGE:UR3C:34AC:JTUU:Z2PN:LFGK

  └ Status: Healthy

  └ Containers: 3 (2 Running, 0 Paused, 1 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:59Z

  └ ServerVersion: 1.12.3-cs4

 UCP-Node-3.cisco.com: 10.65.122.69:12376

  └ ID: OG2S:KTYE:6TYS:LUKK:5NAC:OWDI:6FM2:2ENB:ESAS:7BS5:NDYY:S2LA

  └ Status: Healthy

  └ Containers: 3 (3 Running, 0 Paused, 0 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:08Z

  └ ServerVersion: 1.12.3-cs4

 UCP-Node-4.cisco.com: 10.65.122.70:12376

  └ ID: 7VTY:5U5D:24QE:VCRY:OOA3:AD2D:RE3H:6YSA:RXR4:ELXD:E46H:YDSG

  └ Status: Healthy

  └ Containers: 4 (3 Running, 0 Paused, 1 Stopped)

  └ Reserved CPUs: 0 / 58

  └ Reserved Memory: 0 B / 131.8 GiB

  └ Labels: kernelversion=3.10.0-327.36.3.el7.x86_64, operatingsystem=Red Hat Enterprise Linux, storagedriver=devicemapper

  └ UpdatedAt: 2017-02-02T08:36:43Z

  └ ServerVersion: 1.12.3-cs4

Cluster Managers: 3

 10.65.122.61: Healthy

  └ Orca Controller: https://10.65.122.61:443

  └ Swarm Manager: tcp://10.65.122.61:2376

  └ KV: etcd://10.65.122.61:12379

 10.65.122.62: Healthy

  └ Orca Controller: https://10.65.122.62:443

  └ Swarm Manager: tcp://10.65.122.62:2376

  └ KV: etcd://10.65.122.62:12379

 10.65.122.63: Healthy

  └ Orca Controller: https://10.65.122.63:443

  └ Swarm Manager: tcp://10.65.122.63:2376

  └ KV: etcd://10.65.122.63:12379

Plugins:

 Volume:

 Network:

Swarm:

 NodeID:

 Is Manager: false

 Node Address:

Security Options:

Kernel Version: 3.10.0-327.36.3.el7.x86_64

Operating System: linux

Architecture: amd64

CPUs: 580

Total Memory: 1.287 TiB

Name: ucp-controller-UCP-Ctrl-1.cisco.com

ID: EATX:GRQR:KCHI:I7FG:SI5Q:YQ6Q:G6RE:6LBB:2XIX:JY6R:D3BU:4KLI

Docker Root Dir:

Debug Mode (client): false

Debug Mode (server): false

Labels:

 com.docker.ucp.license_key=wasvIv_S3w_xAfArYYf31vO6TIXfzD31M0gStiRLSOS5

 com.docker.ucp.license_max_engines=20

 com.docker.ucp.license_expires=2018-01-06 23:47:53 +0000 UTC

 

Application Container Deployment

To validate this solution we have performed some feature functional tests, high-availability tests, stack scale-up and scale-down tests. Feature functional tests included routine container life-cycle management operations through Docker UCP client bundle and GUI. Basic operation covers, creating containers, start/stop of containers, killing containers, creating network, and using the created networks for running containers. Docker UCP client requires Docker Toolbox to be installed on older versions of Mac or Windows client machine. Docker Toolbox provides tools such as Docker Compose, Docker command line environment along with the others. The following test validation tasks were accomplished using these tools whenever required. Docker Toolbox can be obtained from below URLs:

·         Docker Toolbox Overview - https://docs.docker.com/toolbox/overview/

·         Docker for Mac or Windows - https://www.docker.com/products/docker-toolbox

Docker client bundle can be installed on a mac or windows box where Docker Toolbox has been installed. It can also be run on any of the UCP Controller, DTR or UCP nodes, without having to install the Docker Toolbox.

Procedure to install Docker client bundle is simple. Follow these steps to download the client bundle:

1.    Download the client bundle from Docker UCP GUI by navigating to admin > Profile > Create a Client Bundle.

2.    Unzip the client bundle file into a new folder on the client machine that has the Docker Toolbox installed:

3.    Set the environment variable by running env.sh.

4.    Now you are good to use the client, as shown in above example. Setting up environment variable enables you to use your admin credential to connect to your deployed Docker Datacenter. Once the environment is set, you can do all container management operations through Docker shell commands sitting outside the Docker UCP cluster itself.

Container Networks

There are various ways in which container-to-container and container-to-host connectivity is provided. This sub-section provides details on the container networking types supported for Docker containers:

·         None – None is a simple mode of container networking. In this the deployed container receives a network stack, which lacks an external network interface; but does receive a loopback interface. This mode of container networking is mainly used for testing and staging a container and where container does not need external communications.

·         Bridge – A Linux bridge provides a host internal network in which containers on the same host may communicate, but the IP address assigned to them are not accessible from outside of the host. Bridge networking leverages iptables for NAT and port-mapping thus providing single host networking. Bridge network is a default Docker network type, for example, docker0 is a bridge name where one end of a virtual network interface pair is connected between the bridge and the container. NAT is used to provide communication beyond the host. Bridge networking solves the problems of port-conflict and also solves performance over head associated using NAT. It also provides isolation to containers running on one host.

Figure 1.              Bridge Network

The first image in the figure shows how a bridge network communicates and the adjacent images shows how a service on Container-2 gets exposed to outside world via host port 8080 mapped to port 80 on the container.

·         Host – In this type of network, a newly created container shares its network namespace with the host, providing higher performance and eliminating the need for NAT. The bottle-neck in this type of network is port-conflicts and since container has full-access to hosts interfaces it run into security risks.

Figure 2.     Host Network

           

·         Overlay – It uses networking tunnels to deliver communication across hosts. This allows containers to behave as if they are on the same host by tunneling network subnets from one host to the other. Essentially, it is like spanning one network across multiple hosts. VxLAN encapsulation is the tunneling choice for Docker libnetworks. Multi-host networking requires additional parameters when launching Docker daemon, as well as a key-value store. Overlay networks focus on the cross-host communication challenge. Containers on the same host that are connected to two different overlay networks cannot communicate with each other via local bridge – they are segmented from one another.

Figure 3.     Overlay Network

Deploying container/application container using overlay network

Follow these steps to deploy container/application container using overlay network:

1.    Log in to any Linux machine with Docker UCP client bundle installed. To list out existing available networks, run the command ‘docker network ls’ as shown:

2.    For creating an overlay network, we need to specify the overlay driver and the network name as shown below:

The ‘docker network create’ command requires driver, name of the network and the subnet. After creating the network we can verify its successful creation by listing available networks. The last column as shown in the screenshot above lists the scope of the network created, ‘local’ refers to the network created on the host locally, while ‘global’ scope tells network is created across all the nodes of the cluster.

3.    To check the details of the created overlay network, we use ‘docker inspect’ command:

 

4.    Deploy two test ‘busybox’ containers on different hosts using the created overlay network. We are creating two containers for establishing container to container communication:

5.    Let us check the network stack created inside the container to verify that the interface is plumbed correctly and it gets the ip address from the subnet we have specified earlier and the routes it is populated with. We use the command ‘docker exec’ to verify this:

6.    From the above screenshot we see that the first container’s ip address as 10.100.100.2. We will verify its reachability and make sure that it can ping the external gateway:

7.    From the above screenshot we see that the first container’s ip address as 10.100.100.3. We will now verify if the container-1 and container-2 are pinging.

8.    Deploy an application container for example ‘nginx’ with port-mapping done for httpd services, in order to access its web-service from the upstream network:

We have mapped the port 5555 on the host to port 80 using ‘-p’ port-mapper switch to deploy an ‘nginx’ container. We can see its deployed on our first UCP-Node-1 host.

9.    Open https://<UCP Primary Node ip-address> in a web browser to login to the Docker UCP. The dashboard shows the ‘nginx’ container being deployed:

10.  Open <node ip where container get deployed>:5555 in a web browser. With the port-mapping in place, we can reach our nginx web-server from outside world, by accessing the port#5555 as shown below:

DTR Operations

This section details about validating the Docker Trusted Registry workflow. As mentioned earlier, DTR provides a way to create a repository of Docker images used for container deployment locally for the enterprise. This eliminates the need for accessing external repos in public domain for downloading images for the containers. This option provides a better control on deploying images for the enterprise and secures application container environment.

1.    Login to the DTR GUI and create a repository. In this example we have chosen ‘busybox’ as the repository:

2.    Download the sample Docker image (busybox) to the local host. Use ‘docker pull’ command to download the image from the Docker public repo:

The downloaded image gets pulled to all our cluster nodes as shown in the screenshot above. This enables us to run the container using this image on any of the host.

3.    To list all the images that we have on the setup use ‘docker image’ command:

4.    Tag the image with version that we want to put in our DTR repo. Tags help in version control. So any dev changes to this image version will have an incremental version number and will be pushed to the DTR repo. This way we have an option to quickly upgrade and/or downgrade to the required version.

5.    To push our tagged image to local DTR repository we need to log-in to the local DTR and push the image as shown below:

6.    Verify that the image was successfully pushed from the DTR dashboard:

7.    Select 1.0 tag in the TAGS tab and press INFO tab to get the command to pull this image from the local repo for deployment:

8.    Now validate the container deployment using local DTR image repo:

9.    The complete URL of the busybox image provides the path for the ‘docker run’ command from where the image for the container deployment can be taken. Note that we have already logged in to our local DTR before using ‘docker login’ command; this facilitated us to get connected on the fly, use the image from there and deploy a container from it. The command ‘docker ps’ lists all the containers.

Sample WordPress Application Container Deployment

Docker provides an automated way to deploy entire application stack at one go and scale with ease and simplicity. Docker Compose is another tool for such an application deployment use case. The tool needs a yml file, detailing all the configuration parameters for the stack, including network and volume storage. When we run ‘docker compose’ using the configuration file we get the whole application stack up in just a few minutes. And these applications are running inside the containers that were deployed.

In this section, we are going to deploy a two-tier container application called WordPress, which is a popular blogging website world-wide. The application stack has a front-end WordPress and back-end database mariadb.

1.    Create a yml file for Docker Compose as shown in the screenshot:

2.    Create a directory name ‘wordpress’ and put the yml file into it. This yml file is taken by default when running docker compose:

So this way application containers gets pulled from the Docker hub and gets installed.

3.    The status of wordpress application containers is shown below:

4.    You can see the wordpress status on Docker UCP dashboard:

5.    The screenshots below shows the initial WordPress application configuration pages:

a.     Select the language. Click Continue.

b.    Enter the user credentials and other details as needed.

c.     Login to the WordPress application.

d.    After the successful login, you will see the WordPress dashboard.

High-Availability Tests

This solution is designed to provide service availability for running the container applications in the event of a failure at the hardware and/or software stack. Performance impact is expected in the service degraded state. To test these scenarios and their impact on system performance, faults were injected on an up and running setup. Docker UCP Controller, UCP and DTR nodes were subjected to the various failure scenarios and their performance was observed. Node failure was simulated by rebooting and shutting of one node at a time and removal of the node from the chassis.

Tests Performed on Docker Datacenter

1.    UCP Node reboot/shutdown:

-        Tests passed with all the running containers coming up online without any delay.

2.    DTR Node reboot:

-        Tests passed with DTR application containers coming up online without any delay.

3.    UCP Controller Master/Replica node reboot:

-        Tests passed. UCP management control plane through CLI/GUI was available for cluster/application container administration. Controller node reboot did not impact container data-path and subsequent re-convergence of the cluster.

4.    UCP Controller Master/Replica node shutdown:

-        Tests passed. Controller node shutdown does not impact cluster and application container management operations and associated data path. All such activities remain unaffected and were serviced by surviving controller nodes. Cluster re-convergence succeeded on shutdown node bring-up.

-        Containers which were already deployed and the applications running were seen not affected on the UCP nodes during the unavailability of that UCP controller node.

5.    Docker Engine Service Restart on UCP nodes:

-        Tests passed with the infrastructure containers and deployed application containers coming up online without any delay.

6.    Docker Engine Service Restart on DTR nodes:

-        Tests passed with DTR application containers coming up without any delay.

7.    Docker Engine Service Restart on Controller Master/Replica Nodes:

-        Tests passed; however, there was a delay seen in restoring the services.

-        Restoration time was similar to the UCP Controller Master/Replica node reboot (#3) above.

8.    Infrastructure component level service/process restart:

-        Docker Datacenter services runs on cluster nodes as infrastructure containers except for the Docker Engine, which is a ‘systemd’ process controlled at the operating system level through ‘systemctl’ control calls. Docker Engine restart tests have been covered under #7 above. As part of this test case, we identified a few key services and components of the software stack for each node types.

a.     Controller and DTR nodes have the following infrastructure containers running:

b.    UCP nodes have the following containers running:

-        On UCP and DTR nodes, the infrastructure containers were restarted and the containers came up without any issues. On the controller master and replica nodes, tests were not performed to the above mentioned Docker defect. 

Tests Performed on Cisco UCS Servers

Cisco UCS provides a robust system which has no single point of failure, right from Cisco Fabric Interconnects to adapters on Cisco UCS blade servers. However, there are failures which cannot be handled at the system level, such failures are:

·         CPU failures

·         Memory or DIMM failures

·         Cisco VIC failures

·         Motherboard failures

CPU/Motherboard/VIC (if only one present) failures results in blade/node getting out of service thereby causing the cluster to operate in a reduced capacity. Application containers running on them will have to be started on the other surviving nodes manually in case such a failure is encountered. A fully loaded cluster can also be on a performance tax till the required capacity is restored.

Hardware failures that are addressed at the system level include:

1.    Cisco Nexus Switch – Cisco UCS Fabric interconnects are connected to upstream Cisco Nexus 9000 Series switches in a vPC mode with redundant uplinks in a portchannel configuration. It is observed that Cisco Nexus 9000 Series switch failure did not impact the application data path. Tests passed without any issues.

2.    Cisco UCS Fabric Interconnect – Cisco UCS has Fabric Interconnects in redundant mode to provide no-single point of failure for the application container’s data path. Since vNICs are configured for Fabric failover, in the event of any uplink, upstream Cisco Nexus 9000 Series switch and Cisco Fabric Interconnect failure data path of the application containers fails over automatically to the redundant fabric interconnect. Following tests were performed and have passed with no impact on the container application’s data path:

-        While a container was pinging to the external gateway address, uplink ports on the primary Fabric Interconnect were disabled.

-        Primary Cisco Fabric Interconnect was shutdown.

-        Cisco Nexus 9000 Series switch was shutdown.

3.    Cisco UCS Fabric Extenders (IOM) – Cisco UCS chassis has two IOMs providing converged connectivity to the blade and Fabric Interconnects. It carries both management control and data plane traffic to the upstream switch and acts as an extension to fabric interconnect for the blade IO. Tests were performed to test if there were any interruptions in traffic. When the containers were pinging internally to each other between the blades and to the gateway address, the IOM-A was removed and inserted back. No packet loss observed.

4.    Hard Disk – All the blade nodes have RAID-1 configured for both OS boot LUN and Docker data LUN. While nodes were up and were running containers, one of the node’s hard disk was removed and inserted back. Test passed with no outage at any level - OS/ DDC/ Containers.

Upscale Tests

Cisco UCS infrastructure has a built-in mechanism to scale the deployed application environment through service profile templates and server pool concepts. It is very easy to scale-up the infrastructure. By inserting new blade hardware and adding it into the server pool, gets discovered automatically and gets a new service profile associated to it. Once the service profile is associated the blade gets a new node entity to be included in the Docker Datacenter pod. This procedure remains the same for any type of nodes, be it a UCP Controller, UCP or DTR node till the point where OS and Docker Engine is installed. Depending on the node type, node joining command to the existing UCP cluster varies. Blade discovery at the UCS Manager level is common; however, in order to differentiate and distribute in the two different chassis we have created three service profile templates one each for UCP Controller, UCP and DTR nodes. Service profiles are instantiated from one of these service profile templates based on the need for scale of a particular type of node. Here are the steps to scale up (for example, a UCP node):

1.    Insert a new blade in any available chassis slot and get it discovered:

2.    After successful discovery, add the server in to UCP Node server pool:

3.    After successfully adding the node to the server pool, create new service profile (for example, UCP-Node-5) from the specific template based on the node type (For example, Docker-UCP-Node):

4.    Service profile association will automatically kick-in as the template was already associated to the server pool.

5.    Follow the OS install procedure explained in the earlier section to install the RHEL7.2 stock operating system on the blade.

6.    Once the Operating System gets installed, perform post-install configuration tasks, such as subscribing to RHN, updating the system, configuring NTP, updating /etc/hosts file to include all the cluster nodes, open up firewall ports etc as detailed in the earlier section.

7.    Install Docker engine repo, as explained in the earlier section.

8.    Install Docker Engine and configure Docker data LUN of sixe 200 GB size, using device mapper storage driver in ‘direct-lvm’ mode as described in the earlier section.

9.    Enable and start Docker-Engine and follow the remaining steps to include the new node into the cluster:

10.  For adding another controller node, all the above steps remain same except that while adding a new node to the cluster we need to run the following command as shown in the screenshot below:

 

Removal of Docker UCP Controller Replicas and UCP Nodes

UCP Node Removal – This does not require any special steps. UCP ‘uninstall’ command is needed to be run on the UCP node which is getting removed from the cluster. Cluster management control plane automatically removes the said node from monitoring and scheduling purposes after fixed timeout period. Inventory also gets refreshed accordingly.

docker run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:1.1.4 uninstall –i

UCP Controller Replica Removal – There are 2 stages to do this. In case of cluster scale down scenarios, where user wants to remove a replica controller node, first ucp uninstallation needed to be run on the node, after that removed replica node records are required to be cleaned up from the existing cluster etcd(key-pair store database). In case of removal of failed replica node scenarios, only database cleanup is needed

Uninstalling UCP on replica node:

docker run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:1.1.4 uninstall -i

      Cleaning up etcd/kv state store data base:

1.          On one of the controller nodes check the overall cluster status:

docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem ls /orca/v1/controllers

To see the cluster health:

docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem cluster-health

2.    Remove the node using the command:

docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem rm /orca/v1/controllers/<node-ip>

docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem rm /orca/v1/config/clientca_<node-ip>:12381

docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem rm /orca/v1/config/clientca_<node-ip>:12382

3.    Remove the failed node from the cluster PKI member list:

To list the members:

docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem member list

Removal of the failed member:

docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem member list

ETCDCTL="docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem"

echo $ETCDCTL

CONTROLLER_IP="<node-ip>"

${ETCDCTL} member remove <node-id as shown in the cluster health status command for the failed node>

${ETCDCTL} rm /orca/v1/config/clientca_$CONTROLLER_IP:12381

${ETCDCTL} rm /orca/v1/config/clientca_$CONTROLLER_IP:12382

${ETCDCTL} rm /orca/v1/controllers/$CONTROLLER_IP

${ETCDCTL} rm /orca/v1/swarm/0/managers/$CONTROLLER_IP:2376

docker exec -it ucp-kv etcdctl --endpoint https://127.0.0.1:2379 --ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem member list

The following infrastructure components are needed for UCS B-Series first architecture:

 

 

The following infrastructure components are needed for UCS C-Series second architecture:

The emergence of the Docker platform and the underlying support in the Linux and Windows kernel has enabled a shift in the way that traditional applications are managed and new applications are designed and built, moving to more efficient Microservices architectures. Microservices architectures are an approach to modernizing and building complex applications through small, independent components that communicate with each other over language-independent APIs.

The integration of Docker Datacenter with Cisco UCS converged infrastructure enabled through automation tools like UCS Python SDK and Ansible provides container application platform to get deployed and managed quickly at scale. It also provides a very good starting point for enterprise IT to make in-roads into DevOps and CI/CD model for application environment for quick business need turn around.

Cisco and Docker’s container management solution, Docker Datacenter, accelerates your IT transformation by enabling easier and faster deployments, greater flexibility, heightened business agility, increased efficiency with lowered risk and higher ROI for enterprise customers.

Rajesh Kharya, Technical Leader, Cisco UCS Solutions Engineering, Cisco Systems Inc.

Seasoned data center Infrastructure professional specializing in Unix/Linux, Networking, Unix File systems, Storage Management and Cisco's Converged compute/storage/network/virtualization platform – Cisco UCS. Functional roles include Unix/Linux administration, Software Feature QA leadership to Solutions Development and Architecture. Currently involved in Solution development projects Containers on UCS baremetal and OpenStack on UCS platform. With over 15 years of experience in the domain, excited and glad to be part of technological transitions happening in the datacenter.

Uday Shetty, Senior Software Engineer, Strategic Development, Docker,  Inc.

Uday Shetty has more than 25+ years of experience in Enterprise Applications at Sun/Oracle and Docker, working with ISVs and Strategic Partners on new technology adoptions and Reference Architectures. Uday’s expertise includes Java, J2EE, Docker for Linux/Windows, Docker Datacenter for Enterprises and building Docker Datacenter templates for Cloud providers.

Sindhu Sudhir, Technical Writer/TME, Cisco Systems Inc.

Sindhu has 7+ years of experience in the IT industry including technical documentation/ editing, software development, infrastructure automation and has strong knowledge of networking. Her recent contributions to the Cisco UCS solution engineering team include co-authoring UCS C240 I/O characterization and Containers on UCS platform whitepapers.

Acknowledgements

·         Cisco Systems: Vadi Bhatt, Vishwanath Jakka

·         Docker: Bradley Wong

NOTE: Available paragraph styles are listed in the Quick Styles Gallery in the Styles group on the Home tab. Alternatively, they can be accessed via the Styles window (press Alt + Ctrl + Shift + S).