Cisco_Penn1_Hybrid_Work_Case_Study

 

Introduction

Cisco’s PENN 1 office located in Midtown Manhattan, New York City is one of Cisco’s newly reimagined office spaces designed to provide the ultimate hybrid work experience for employees and visitors while also leveraging smart building technology to help reach Cisco’s sustainability goals and provide real estate and facilities teams the operational insights needed to understand how the space is being used. The PENN 1 office was the first of its kind at Cisco and as such, was a learning experience. This document provides a case study of how networking and collaboration technologies were deployed alongside smart building technology and sensors providing a sustainable hybrid work experience. This document attempts to highlight some of the lessons learned but is not intended to be a generic design guide and not all decisions made for this deployment will apply to all customer use-cases. Technology has also evolved since the office was open and when applicable, we will note changes that have been incorporated into newer offices.

For a quick video tour of the office, please visit this link. This document views the deployment of hybrid work technologies at PENN 1 from a more technical lens, highlighting some of the devices and design decisions that were made. For more detail into the spaces and some of the architectural design decisions made, please take a look through the look book here.

As is the case with any production deployment, implementing such a solution requires a phased execution approach. Phases of implementation and the realization of targeted outcomes and requirements at each phase result in a journey map to deliver on the expected end state outcomes and business impact. In addition to streamlining operations, after the execution of each phase, the team leveraged learnings as well as knowledge of capability roadmaps to identify and implement appropriate optimizations.

The PENN 1 office was initially created as a proof of concept and as such, many of the technologies used had no pre-existing IT standards for their deployment. Additionally, there was a desire to reimagine the purpose of the office which required rethinking how space was allocated. As a result, a virtual team was created of subject matter experts from Cisco’s IT, sales, and workplace resources organizations, who were responsible for designing and deploying the office and then taking the learnings to create new IT standards that could be leveraged for future offices. This effort took place through 2021 and early 2022, so design decisions made here should be viewed through the lens of the state of the art at that time. Additionally, supply chain issues at the time influenced some of the product choice decisions. The document will point out alternate product selections that have been used in newer offices that have been implemented since the opening of PENN 1.

Outcomes and Requirements

The goal of the office was to deliver on the following outcomes:

●   Transform all office workspaces, from open area seating, quiet rooms, and huddle rooms all the way to large collaboration spaces, meetings rooms, and training rooms. The goal was to create an environment that was more conducive to collaborative experiences (“we” spaces).

●   Connect people in new ways when they are in the office while being inclusive of remote participants, recognizing that 98% of all meetings will have at least one remote participant.

●   Help Cisco reach its sustainability goals by leveraging low-voltage technologies and automation to power as much of the infrastructure as possible.

●   Enhance health and well-being as well as provide a safe workspace that employees can trust.

 

Requirements associated with solution deployment also mapped directly back to Cisco's external solution messaging. Therefore, it shouldn't be a surprise that solution requirements at a high level were inclusive of:

●   Frictionless Employee and Guest Experience

o    Secure, high-performance, enterprise-grade connectivity (Wireless and Wired)

o    Pervasive Video Collaboration for all workspaces (Desk, Conference, Huddle)

o    Signage (Occupancy, Safety, Environment)

o    Interactive Signage (Find Points of Interest, Workspaces, Conference Rooms)

o    Ad-Hoc Room Booking / Hot-Desking

●   Smart Workspaces

o    Reporting & Dashboards for Workspace Occupancy, Environment (Temperature / Humidity), and Safety (Air Quality)

o    API integrations with building management systems

o    Data Export to cloud storage for consumption of historical raw data

●   Facilities Management / Reporting

o    Workspace Occupancy and Utilization Reporting / Dashboards

o    Workspace Environment & Safety Reporting / Dashboards

o    Occupancy and Environmental Data Pipeline for Building Management Systems / custom integrations

Design Approach

The team leveraged an outcome focused design strategy while considering the following:

High Level Journey

With a specific focus on Hybrid Work from Office, the customer journey of building the PENN 1 office included the following phases:

●   Modernizing the infrastructure

●   Designing for digital building/smart building capabilities

●   Automating the deployment of the required wireless and wired capabilities to accommodate the rollout of location analytics.

●   Implementing integrated security leveraging Identity Services Engine (ISE) to ensure asset visibility and access control.

●   Deployment of Webex collaboration devices and Cisco Spaces

●   Deployment of IoT technologies (Meraki cameras, Molex, Igor, and Mecho PoE lighting, shades, sensors, and controls)

 

In addition to IT operations related considerations, the design and implementation teams had to ensure that all technology decisions were aligned to overall architectural goals. This is different than having various technologies randomly added over time without end state outcomes and requirements guiding decisions. When optimizing for collaborative “we” spaces rather than the traditional “me” space, the team understood that collaboration is not a single activity. Sharing information, making decisions, and brainstorming each has a specific set of technology requirements (single or dual display, display size, video-centric vs. interactive board). Once defined, the team was able to focus on the details that supported the activity. They were able to work with facilities management to refine things like table shape and orientation, surface size, the posture of seating, as well as the overall layout of the space including ambiance, density, and acoustics. Additional details associated with the process to design for a unified collaborative experience are not included as part of this documented case study.

“As Is” vs “To Be”

The As Is state was no different than any typical office with cubicles, private offices, and large conference rooms. The goal was to transform the large conference room spaces to spaces that would provide an immersive video conferencing experience built for the modern hybrid workforce. Instead of having a large footprint of private offices, there was an expectation to transform these areas into small huddle spaces to support ad-hoc collaborative engagements. Spaces occupied by cubicles would need to be transformed into areas available for hot-desking as well as provide enough of a footprint for a community lounge area.

In the end, the office was split into two distinct areas. One half of the office is considered open to customers and guests while the other half is employee-only space. Larger conference and collaboration spaces are all located in the customer side while smaller huddle spaces, open area seating, and quiet rooms are in the employee-only side.

A design decision was made to make only larger spaces bookable through the calendaring system while smaller spaces are available on a first-come, first-served basis to optimize space utilization. Cisco Spaces, covered later in this document, facilitates the ability for employees to find the spaces they need when they need them.

A variety of different spaces were created, each with a distinct purpose. For example, some spaces are designed to accommodate large meetings while others are designed for whiteboarding and real-time brainstorming activities.

For a more in-depth look into how each of the individual spaces was built including CAD drawings of the spaces, furniture used, acoustic treatments, power, data wiring, and more, design guides are available for each type of room. Please visit this site and fill out the form to gain access to the detailed design documents.

Occupancy Reference Architecture

The office had to be designed to capture and leverage occupancy data and integrations with Webex Control Hub and 3rd party systems to take advantage of that data in a way that delivered on the agreed upon outcomes and requirements previously specified.

The PENN 1 Environment

This section highlights details related to the topology, hardware, software, and design selections that were made as part of the PENN 1 Hybrid Work from Office deployment.

Network High Level Design

Cisco maintains a variety of network standards and the target network design for the deployment at PENN 1 is based on the Cisco “Gold” Topology standard.

You may notice that the environment at PENN 1 isolates the IT and OT environments, keeping OT devices such as PoE lighting, shades, and IoT devices in a network outside the standard IT environment. Future deployments will converge the IT and OT stacks into a single infrastructure and eliminate the dedicated DC and OT gateways for easier manageability. The decision to keep IT and OT networks separate was purely one of roles and responsibilities of managing the networks as opposed to technical limitations.

The office also includes a small Hyperflex-based datacenter environment to host virtual machines needed to control/manage the low voltage and IoT systems needed to create a smart building. These applications are detailed later in this document.

Wired Design, Platforms, and Connectivity

Physical Infrastructure Overview

Infrastructure Cabling

The PENN 1 office is 54,000 sq ft requiring 2 wiring closets [1 Building Distribution (BD) and 1 Floor Distribution (FD)] to meet the Ethernet distance requirements. The diagram below provides details on the fiber and copper infrastructure cabling between the Minimum Point of Entry (MPoE), 9.1 BD racks, 9.2 FD racks, and Customer Experience Center (CXC) rack.

The PENN 1 cabling design separated the endpoint cabling drops so the Production IT Access Network drops for the production wired and wireless desktop networks were terminated in the 9.1 BD and 9.2 FD IT racks, and the Operational Technology PoE Network drops for the PoE lighting, shading and HVAC control were terminated in the 9.1 BD and 9.2 FD OT PoE racks.

                                  

The brand new 23AWG yellow patch cables used for the OT PoE Network are made from recycled cables from the previous Cisco office deployment, helping to further Cisco’s sustainability goals through recycling.

UPS Strategy

There are many different strategies for providing backup power during power outages and one of the challenges with the addition of new POE powered IOT endpoints such POE lighting and shading is balancing the cost, size, and weight of the battery backup solution versus the desired runtime for various IT and OT POE powered endpoints.

PENN1 was originally deployed with the IT network leveraging the All on UPS strategy and the OT network leveraging the None on UPS strategy, but now with more insights into our energy usage we are moving towards a UPS strategy driven by our PoE endpoint Service Owners and their UPS runtime requirements. As shown in the table above, certain Service Owners might require a 60-minute runtime and other Service Owners might not care if their POE endpoints go down so with a converged network infrastructure the only differentiator is the UPS runtime requirement.

During a building power outage, Cisco workplace policy states that users should vacate the office and PENN 1 has a dual-feed POE and building power emergency lighting solution in case either service is disrupted.

Smart PDUs

In order to gain insights into energy usage and capture historical energy usage data Raritan PX3-5551V Smart PDUs were deployed in the OT PoE Network 9.1 BD and 9.2 FD racks.

                               

The browser interface of the Raritan iX7 PDU controller module allows easy access to the configurations and to the power and monitoring data.

We are leveraging Sunbird Power IQ to capture the historical energy usage and manage the Smart PDUs.

Power IQ also allows us to generate various energy usage reports for offices, networks, racks, and IT devices such as Catalyst 9300 stacks, Catalyst 9400 chassis or Cisco UCS servers.

You can't manage what you can't measure, and the Smart PDUs provide us with the insights into the energy coming into our racks to power our PoE network infrastructure equipment.

The OT network Raritan Smart PDU and the IT network Server Technology Smart PDU energy data is being integrated into the new Cisco Spaces Energy app.

OOB Console Network

Cisco’s standard is for all network devices to have out-of-band async console network connections for management and troubleshooting so all the PENN 1 Catalyst 9300 async management ports patched to the Catalyst 8300 console server.

Catalyst 9300

The Catalyst 9300 series C9300-24H and C9300-48H are purpose built for smart building network deployments providing 24 or 48 ports of gigabit ethernet with 90W UPoE+. With 1900W Power supplies installed, the C9300-24H provides 24 ports of 90W UPoE+ and the C9300-48H provides 32 ports of 90W UPoE+. These Catalyst 9300 series switches provide power redundancy features (redundant power supplies, StackPower, Port Priorities), PoE redundancy features (Perpetual PoE, Fast PoE, 2-Event Classification) and PoE Assurance features (Telemetry, Analytics Widgets, Troubleshooting). The switches are IEEE 802.3bt compliant and backwards compatible with all PoE Standards and additional switching details for these models can be found by accessing the following data sheet.

For the PENN 1 deployment, the C9300-24H was chosen for supply chain availability and capability to provide 90W UPoE+ on all 24 ports. Now that the new C9300X models are available, future IT deployments will utilize the C9300X-48HX with 1900W power supplies which can provide 36 ports of 90W UPoE+.

HyperFlex M5 Edge with Intersight

The Cisco HyperFlex M5 Edge with Intersight was chosen for its high availability to provide the local onsite compute resources required by the PoE lighting and shading deployments. Lighting vendor motion/occupancy detection and action requires low-latency response from the lighting management applications to provide a good end-user experience. (i.e., user walks into a conference room and the lights turn on immediately instead of having a several second delay)

The required HyperFlex Edge VMs were configured according to the IoT partner published specifications.

OT PoE Network Design Overview

The following is the PENN 1 as-built network overview diagram of the IT and OT networks. As mentioned earlier, the OT infrastructure at PENN 1 was built as a POC environment, so Cisco considered that part of the network to be a “lab” environment that must adhere to different requirements than a standard infrastructure. Although the “lab” nomenclature is used, this is indeed a production deployment.

As previously mentioned in the Infrastructure Cabling section, the PENN 1 cabling design separated the IT and OT networks with the IT wired and wireless cabling drops terminating in the IT racks and the OT 90W PoE cabling drops terminating in the OT racks. Thirty C9300-24H were installed in the OT racks to support the Molex lighting/sensors, Igor lighting/sensors, Mecho shading/sensors, Teknion Connected Desks/sensors, Delta Controls VAV, Raritan PDUs, and Anker USB-C data/power adapters (all described in further detail later in this document).

The OT PoE access layer switches were deployed leveraging the Campus LAN Layer 2 Access with Simplified Distribution Cisco Validated Design (located here).

The decision to use the Campus LAN Layer 2 Access with Simplified Distribution was driven by the desire to reduce complexity by running fewer protocols and increase resiliency. In the simplified distribution layer, the distribution-layer node consists of a single logical entity that can be implemented using a pair of physically separate switches via StackWise Virtual operating as one device or using a physical stack of switches using StackWise operating as one device. Resiliency is provided by physically redundant components like power supplies, supervisors, and modules, as well as stateful switchover to redundant logical control planes. Also, little or no tuning is required to provide near-second or sub-second convergence around failures or disruptions.

The OT PoE access layer switch ports were configured using the Deploying 90W Cisco UPoE+ with Cisco Catalyst 9000 Switches Deployment Guide and the guide can be found here.

For PENN 1, the decision was to allocate a dedicated a VLAN for each IoT partner. The IP subnet sizing was driven by the number of the IoT partner endpoints deployed in each VLAN. Some of the subnet sizing was also driven by limitations of IoT vendors’ devices (e.g., some vendors do not support subnets larger than a /23) and 4 of the VLANs were configured with Port-based DHCP.

To support the IoT partner lighting and shading IP network commissioning requirements, there are several DHCP IP address assignment options for the various 90W UPoE+ endpoints.

The first option is DHCP using DHCP server reservations which statically binds the endpoint MAC address to IP address so that the IP addressing remains consistent for lights and wall controllers placed together in rooms. The DHCP reservations are simple to implement but hard to scale and maintain because when a UPoE+ endpoint fails and must be replaced, the DHCP server reservation must also be manually updated for the new MAC address.

Port-based DHCP is the second option that assigns/binds a specific IP address to a specific switch port so that no matter what device/MAC address is connected to the switch port, it will always receive the same IP address. For example, if switch port gi1/0/1 is configured with port-based DHCP IP address 10.1.1.1, any device connected to this port will be assigned 10.1.1.1. Port-based DHCP can be configured on a C9300 switch stack using the IOS-XE DHCP Server and the main advantage is that no external DHCP Server is required, which is great for isolated deployments. The major disadvantage is if you have multiple switch stacks in your network, then each stack requires its’ own IOX-XE DHCP Server to be configured which limits the scalability of this DHCP option. Additional information on Port-based DHCP running on C9300 IOS-XE is located here.

The third option is DHCP Option 82 which is also known as the DHCP Relay Agent Information. DHCP Option 82 enables the DHCP server to allocate dynamic IP addresses based on the relay information inserted and sent by a relay agent. The C9300 switch stacks can be configured to insert relay information (Circuit ID and Remote ID) by enabling DHCP Snooping and the DHCP server can be configured to assign specific IP addresses based on the received relay information so that no matter what device/MAC address is connected to a specific switch port, it will always receive the same IP address.

At the time of the PENN 1 deployment, Cisco’s global DHCP infrastructure did not support DHCP Option 82, so Port-based DHCP was configured on the 5 C9300 switch stacks in PENN 1. Since that time, the DHCP infrastructure has been upgraded so that our more recent Smart Building deployments are leveraging DHCP Option 82.

The HyperFlex Edge VM IP addressing was assigned for each of the IoT partner VLANs.

You should now have insight into the PENN 1 OT PoE network design, platforms, connectivity and VLAN and IP assignments. The next section provides a brief overview of the IoT Partner PoE Technology Solutions showcased at PENN 1.

IoT Partner PoE Technology Solutions

Molex Lighting

The Molex PoE lighting solution installed in PENN 1 covers 60 percent of the 54,000 sq ft.

The Molex PoE lighting solution consists of both hardware and software. The following is the device count for the hardware installed in PENN 1:

●   107 - PoE Constant Voltage Gateways

●   336 - PoE Gateway 2.0

●   6 - PoE Wireless Gateways

●   746 - LED Drivers

●   87 - Wall Switches

●   121 - Advanced Lighting Sensors

●   13 - Advanced Environmental Sensors

 

The Molex PoE lighting solution software includes the following applications:

●   CoreSync Manager

●   API/BACnet Gateway

●   Design Tool (included with CoreSync Manager)

●   Facility Manager

●   MoDiag (included with CoreSync Manager)

 

Here is an overview diagram for how the Molex POE Gateway, LED Driver, Wireless Gateway, and Advanced Sensors are connected and deployed at PENN 1 with the goal to maximize the usage of each 90W UPoE+ port on the PoE Switch by daisy-chaining multiple fixtures:

The following is the Design Tool interface where all the physical lighting fixtures are added to the floorplan.

Once all the lighting fixtures have been added, user zones are created to group the fixtures into rooms or spaces and assign a lighting schedule or actions. A user zone can be programmed to turn on/off at specified times or triggered by activity detected by a motion sensor.

The Facility Manager is utilized by local support to monitor and manage the Molex PoE lighting solution.

 

Initial commissioning of the Molex POE lighting solution is performed using MoDiag and it also used for troubleshooting and upgrading Molex device firmware.

Igor Lighting

The Igor PoE lighting solution installed in PENN 1 covers the other 40 percent of the 54,000 sq ft space. Typically, an office space would only have one PoE lighting vendor, but with PENN 1 being a Smart Building showcase, a second PoE lighting vendor was chosen for the deployment to compare and contrast product capabilities and test interoperability with the Cisco network.

Note that as of this writing, Igor has gone out of business and their products are now being supported by Digital Building Solutions; however we are still leaving the information regarding our deployment of the Igor solution here just to document the as-is state of PENN 1 and lessons learned. You will see how the solution is very similar to the Molex deployment.

Here is an overview of the Igor architecture and components with the sensors, luminaires, and wall controls connected to the Igor Network or Device Nodes. The Igor Network Nodes connect to the C9300 90W UPoE+ switches, and the Igor Gateway Software runs on the HyperFlex Edge VM, which communicates with the Igor Cloud Portal.

The Igor PoE lighting solution consists of both hardware and software. The following is the device count for the hardware installed in Penn1:

●   27 - Rev 5 60W Standard Network Nodes

●   150 - Rev 6 90W Linear Network Nodes

●   46 - Rev 7 90W CV Network Nodes

●   61 - Device Nodes

●   39 - Light, motion, temperature sensors

●   16 - Wall Controls

 

The Igor PoE lighting solution software includes the following applications:

●   Gateway Software

●   Node Configuration Manager

●   Firmware Updater

 

Here is an overview diagram for how the Igor Network and Device Nodes are deployed at PENN 1:

The C9300 switch port provides 90W UPoE+ to the Igor Network Node which is required for power and data negotiation with the C9300 switch. Depending on the wattage of the lighting fixtures, there could be one or more Device Nodes connected to the Network Node with the goal to fully utilize the 90W provided by each switch port.

The Igor Gateway Software Dashboard provides a web browser interface for configuration and monitoring of the Igor PoE lighting solution.

The Igor Nexos Cloud Portal provides a web browser interface for access to historical data, utilization, and insights; however, our design goal was to aggregate this data into Cisco Spaces along with other data points to have Cisco Spaces provide a holistic view of the building occupancy.

Mecho Shading

The Mecho PoE shading solution implemented in PENN 1 consists of PoE gateways, motors to raise/lower the shades, and the SolarTrac management system.

There are 39 Mecho motors powered by 39 NuLED PoE gateways and 4 Somfy motors powered by 4 Molex gateways. Typically, all the motors and gateways would be the same, but with PENN 1 being a Smart Building showcase, a second motor and PoE gateway was deployed in the office.

The Mecho SolarTrac management system provides a web browser interface for configuration and monitoring of the Mecho PoE shading solution. The physical PoE gateways with motors are added to the floorplan and indicated by the green circles. The blue triangles indicate the physical light sensors, and the green squares indicate the zones configured for the shades. The SolarTrac management system provides API for the programming of integrated outcomes and voice control of the shades.

Delta Controls VAV

PENN 1 leverages building-provided HVAC and supplemental HVAC was added for several of the large conference rooms. An ABM System BMS was installed along with 7 PoE variable air volume controllers, 2 PoE room controllers, and 7 wireless wall controllers. The ABM Systems APIs allow for us to program integrated outcomes, such as adding additional cooling when conference room occupancy count provided by the Webex collaboration units exceeds predefined thresholds.

Teknion Connected Desks

PENN 1 has several Teknion Connected Desks leveraging one 90W UPoE+ connection to the C9300 to provide 60W USB-C charging, 5W USB-A charging, and the motorized desk can be raised/lowered. The Teknion desk can also include occupancy, temperature, humidity, and air quality sensors that are integrated with Cisco Spaces and the Smart Workspaces application. The Teknion integrated occupancy sensor provides down to the desk occupancy data for open workspace hot-desking.

Anker USB-C Power/Data Adapters

PENN 1 has several Anker USB-C adapters providing 60W of charging capacity and 10/100/1000 Mbps of wired Ethernet connectivity. This allows for desks to be completely powered by PoE, giving the workplace resources teams flexibility and agility to move desks around without having to re-run high voltage power lines.

Wireless Design, Platforms, and Connectivity

The wireless network provides ubiquitous data and voice connectivity for employees, guests, and connectivity for IoT devices and is the primary mode of network connectivity at PENN 1. With the emergence of high-density networks and the IoT, organizations are more dependent on wireless networks than ever before. Increasing numbers of devices connect to the network every year, ranging from high-performance client devices to low-bandwidth IoT devices.

Cisco wireless solutions are resilient, have the integrated security organizations need, and employ adaptive and insightful intelligence providing useful insight into the network. With intent-based networking built on Cisco Digital Network Architecture (Cisco DNA), Cisco’s wireless solutions go beyond the latest Wi-Fi 6 (802.11ax) standard and are ready for the growing user expectations, IoT devices, and next gen cloud-driven applications. With the ability to handle the increased mobile traffic (as well as support IoT at scale), Cisco’s first Wi-Fi 6 APs with superior RF innovations expand wireless access with intelligence and provide a secure, reliable high quality wireless experience for all networks.

The wireless network at PENN 1 was designed to scale to the bandwidth and density requirements necessary to support the increased data utilization of these next generation applications. The design and platform selection had to account for today's usage, as well as for additional future demands associated with the introduction of emerging technologies like virtual and augmented reality.

For the PENN 1 wireless network deployment, Cisco leveraged the Catalyst 9800 Series WLAN controllers and Catalyst 9130 Wi-Fi 6 APs. The Catalyst 9800 Series wireless controllers combine RF excellence with Cisco IOS-XE benefits. The Catalyst 9100 Series APs can handle the challenges of the next-generation network.

Two Cisco Catalyst 9800-L wireless LAN controllers are in use at PENN 1 to ensure high availability and sufficient throughput for client needs. Forty-four Cisco Catalyst 9130 Wi-Fi 6 APs are deployed in a high-density design to provide excellent coverage and seamless roaming for the 54,000 sq ft office space. The high-density AP placement also maximizes location capabilities in concert with Cisco Spaces.

At the time of the PENN 1 deployment, Wi-Fi 6E access points were not yet available, however newer offices are deploying the CW9166 access points with Wi-Fi 6E and environmental sensors.

C9800-L Wireless LAN Controller

C9130AX and CW9166 Access Points

Wireless Network Design Overview

AP Placement

A high-density wireless deployment was chosen to provide excellent coverage and seamless roaming. This is especially important in a hybrid work environment where users will often be using video streaming to connect with people outside the office. The demands of high throughput, low-latency video connections necessitate a high-density wireless deployment. AP placement was also designed to maximize location-detection capabilities to allow Cisco Spaces to monitor occupancy in different areas of the floor.

Cisco Spaces

Webex Control Hub Integration

Cisco Spaces was integrated with Webex Control Hub to gather occupancy data from Webex devices located in the various collaboration spaces throughout the floor. This integration is a simple OAuth2 authentication to tie the Cisco Spaces account to the Webex Control Hub organization. Details on the integration are provided later in the document.

IoT Wired Gateway

For Cisco Spaces to leverage the sensor data available on the OT PoE wired network from the Molex, Igor, Mecho and Teknion sensors at PENN 1, the IoT Wired Gateway is configured on the C9300 distribution-layer node which then communicates with the Cisco Spaces Connector to send the sensor data to Cisco Spaces for use by the Smart Workspaces application.

Here is example of the VLAN and IP address configuration required for configuring the IoT Wired Gateway, which runs on the C9300 in a Docker container.

Logging into the Cisco Spaces Connector with a web browser provides access to view the status of the Connector.

Logging into Cisco Spaces with a web browser provides access to view the status of the IoT Wired Gateway.

From Cisco Spaces you can also view the current sensor data from the POE powered sensors as shown here for the Molex Advanced Environment Sensor.

Here is the sensor data from a Teknion Connect Desk showing desk occupancy.

Here is an overview of the Cisco Spaces sensor and energy data flows from the various endpoints to the Connector up to Spaces.

Smart Workspace Application

The Cisco Spaces Smart Workspace application provides the PENN 1 end users with a 3D floorplan enabling insight into occupancy, air quality, CO2 levels, temperature and humidity, and the ability to view and hold open workspaces and rooms.

The Smart Workspace application runs on Webex collaboration devices throughout PENN 1.

When employees and guests approach the sign, they are greeted with a 3D floor plan of the space which provides a quick view into the space utilization, with available rooms shown in green, occupied rooms in red, and booked, but empty, rooms in yellow. A room finder feature allows users to quickly find the space they need based on criteria such as number of seats and availability of video conferencing equipment. Users can simply tap the “Hold Room” button on the interface and head over to the room. When the room detects a person is present, the map is updated in real-time.

Energy Utilization Application (Beta)

The Cisco Spaces Energy Utilization application provides data and insights into the energy usage as shown here for the C9300 switch stacks.

Cisco Spaces has integration with Raritan, Panduit, and Server Technology Smart PDUs and here is the output for the 8 Raritan PDUs in PENN 1.

Meraki Security

Meraki MV12 Security Camera

The MV12 cloud-managed smart camera is part of the MV smart camera family which brings physical security and advanced analytics together in a compact form factor and currently there are 27 MV12 cameras in PENN 1 to provide physical safety and security and business intelligence and insights.

Webex Collaboration

Webex Collaboration Devices

The PENN 1 office has 92 Cisco Collaboration devices throughout the office. The goal was to make every space video-enabled to facilitate an inclusive hybrid work experience. The look book goes into detail on the devices used for each different type of space at PENN 1. Note that newer offices will be using the Room Kit series of devices as opposed to the integrated Room devices which have built-in screens. The Room Bar, Board Pro, and Room Kit EQ are the preferred devices in newer office spaces and have capabilities beyond the devices deployed at PENN 1.

Recognizing that employees need to collaborate not only with other employees, but also customers and outside vendors, these devices allow employees to join meetings on any meeting platform – Webex, Zoom, Microsoft Teams, or Google Meet – while incorporating advanced features such as noise removal and frames.

Wall-mounted Room Navigators mounted outside the doors of each meeting space are critical to the user experience.

 

With a simple glance down the hallway, employees can easily visualize room availability (which is updated in real-time based on detection of people in the rooms) making it very easy to find an available room. Running Ethernet cables to the door jambs to enable connecting and powering the Navigators was important and something that should be planned for in new construction, although retrofitting after the fact is possible as we have done in other offices.

Webex Room Hub

Modern conference rooms typically have a touch-panel or other control system in the room to control things like lighting in the room, access to AV equipment, and more. Vendors such as Crestron and AMX are often used for these purposes. At PENN 1, we wanted to simplify the experience by providing a single interface to the room. This was accomplished using Webex Room Hub.

Webex Room Hub is an open-source project to control smart building features (such as lights and shades) in office buildings from the Webex Room devices. It consists of a lightweight middleware server, macros, and UI extensions that are installed on the Webex devices. Out of the box, it supports a couple of light and shade systems standard at Cisco, but it is easy to replace or add drivers for any HTTP based smart integration. The application enables both touch controls in the room using the existing Room Navigator panel as well as voice-enabled controls using the Webex Assistant Skills feature, which allows for extending the Webex Assistant. By leveraging the existing Room Navigator, PENN 1 does not require more traditional room control systems (e.g., Crestron) because the functionality can be fully implemented using the Room Hub application.

The project requires that the customer has:

●   Webex devices, such as Webex Room Kit, Webex Board, Webex Desk, etc.

●   A smart building setup

●   Access to install macros to the devices, either with Control Hub or local device access

●   A way to host a docker image that is reachable by video devices and the smart system integrations

 

These smart integrations are supported out of the box:

●   Igor lighting system

●   Molex lighting system

●   Philips Hue lighting system

●   SolarTrac 4 shades

●   Webex-based “Report Issue” feedback

 

PENN 1 has several conference rooms with voice control, allowing users to change lighting and shades without having to touch any controls in the room.

Global DNA Center Deployment

PNP

DNA Center Plug and Play was not leveraged for the OT PoE C9300 switch deployment due to lack of WAN network connectivity from PENN 1 to the Cisco internal network and global DNA Center servers at the time of the switch deployment. The required configuration and commissioning of the PoE lighting usually takes place very early in the construction process, so this required the installation and configuration of the 30 C9300s for the OT PoE network to take place prior to the office having WAN connectivity.

Device Management

The 30 C9300s for the OT PoE network were later added to our global DNA Center Deployment for device management and other services provided by DNA Center.

PoE Analytics

For PENN 1, we are leveraging PoE Analytics to monitor the C9300 switch stack power budgets and gain insights into our PoE endpoints.

Global ISE Deployment

The deployment of ISE was part of broader ongoing initiative to rollout integrated security. Use cases are inclusive of asset visibility, access control, and authenticated guest access.

Architecture

The production ISE deployment consists of 29 ISE Nodes and 20 Policy Server Nodes (PSNs) in 7 Server Node Groups all running as virtual instances and with one in each of the major data centers. The Server Node Groups represent a group of PSNs in the same location, behind Load-Balancers.

As can be seen from the diagram below, the primary Policy Administration Node (PAN) and Monitoring (MnT) servers are in Mountain View, California, and the Secondary PAN and MnT servers are in Allen, Texas. Initially the team had the Primary PAN & MnT ISE in Allen, but this caused issues with replications to Bangalore as a result of exceeding latency requirements. The team needed to keep latency below 200ms between PSN / PAN / MnT due to the amount of traffic to avoid issues.

Deployment Architecture

High Availability Design

As part of the high availability design, the team leveraged Virtual IPs for the different Server Node Groups and the Services within those Node Groups, in addition to Radius Server Node Groups.

They considered 6 Radius Server groups on each of our NADS – Primary, Secondary, Tertiary, Quaternary, Quinary, Senary.

The Load Balancer is used to send a User-Probe/Synthetic Authentication to each of the PSN’s every 10 seconds, and if it receives a failed Auth, then it removes the PSN from that Server Node Group. The load balancer continues sending requests and adds back the PSN once authentication is successful.

The ISE deployment supports internal “Automatic” Failover, which is an optimization that was added over time due to the release and validation of new capabilities.

Each Service has a VIP – Wireless/Wired/VPN/CVO/Extranet as this provides flexibility to upgrade specific services without impacting others. This means the version of ISE used for VPN could be upgraded without impacting Wireless.

The PENN 1 Deployment Journey

This section highlights the deployment of capabilities, targeted technical results, lessons learned, and recommendations captured as part of the team's journey to deploy the PENN 1 office. Deploying the cross-architectural solution inclusive of Cisco Spaces, Webex Suite, Wireless LAN Controllers, Cisco DNA Center, ISE, and ThousandEyes has many moving parts and engineering stakeholders will be able to benefit from Cisco’s experience as a frontrunner deploying the solution in production.

Preparing, Planning, Designing

In addition to the business outcomes and requirements focused on transforming the end user and facilities management experiences, workplace resources captured technical result requirements associated with various aspects of the design and deployment. This additional area of focus factored into the overall design strategy and represented an opportunity to measure IT operations success as part of an implementation governance plan. A sampling of some of the technical results that were highlighted as being important included:

●   Streamlined deployment of Hybrid Work from Office infrastructure and components

●   Simplified integration of key platforms and devices

●   Seamless device setup

●   Flexible, secure onboarding of new infrastructure and devices to deliver on the Hybrid Work from Office experience for end users and delivery of operational outcomes for Facilities Management

●   Maximizing the guest experience

●   Insight into employee and guest behavior

 

With the business and technical requirements in focus, the team went through the preparation, planning, and design phases.

Prepare

The team had to establish the organizational requirements, develop a network strategy consistent with a wider scale rollout of capabilities, and draft a high-level conceptual architecture incorporating the technologies that best support the architecture. This was critical in terms of understanding scope, schedule, talent, and financial requirements to execute the Hybrid Work from Office network design and solution rollout strategy.

Plan

The team identified initial network requirements based on goals, the facility, user needs, and Facilities Management needs. The plan phase involved characterizing the site, assessing the existing network deployment, and performing a gap analysis to determine whether the existing system infrastructure, site configuration, and the operational environment could support the proposed design. A project plan was leveraged to help manage the tasks, responsibilities, critical milestones, and resources required to transform the office to a Hybrid Work from Office destination. The project plan aligned with the scope, cost, and resource parameters established as part of the original business requirements. Planning required executing on specific activities such as:

●   Leveraging Cisco DNA Center (rolled out during an earlier part of the journey) to create a 2D floor map with appropriate scale and highlighting AP placement.

●   Running a predictive site survey using Ekahau to ensure site-specific data is available for wireless design guidance (AP density, channel width, radio configuration, and benchmarks)

●   Reviewing existing segmentation and security access controls associated with device onboarding, profiling, as well as policy impacting firewall/internet connectivity to identify and plan for changes. 

●   Reviewing existing QoS configurations and preparing for the deployment of appropriate QoS considerations to accommodate user/device/IoT onboarding nuances associated with transforming the space to accommodate the rollout of Hybrid Work technology.

●   Creating a CAD file of the site to be used within Cisco Spaces to enable the creation of 2D maps as well as 3D rich maps for the Smart Workspaces application.

Design

The initial requirements that were derived in the planning phase were used to drive network design activities. The team drafted the required network design specification to meet the Hybrid Work from Office business and technical requirements. The specification accounted for availability, reliability, security, scalability, and performance criteria. The design specification acted as the basis for the implementation activities covered below.

Lessons Learned and Recommendations

Implementation: Deployment of Fundamentals

The network was built (and additional components were incorporated by the team according to the design specifications) with the goal of integrating devices without disrupting the existing network or creating points of vulnerability. The deployment of the fundamentals required setting up all centralized management platforms and services that weren't already implemented. The team needed to ensure that if existing platforms weren't at the right versions of software that they were upgraded to accommodate the deployment of required capabilities. In some cases, upgrades were required to support integrations associated with the Hybrid Work from Office solution.

The following is a sampling of the technical results associated with the deployment of the fundamentals and highlighted as being important:

●   Provide visibility into physical spaces to understand user and device behaviors based on location data

●   Visualize “right now” near-real-time metrics from location

●   Leverage custom reports based on location, SSID, time range, and other areas of focus to identify trends

●   Provide deeper insight into location metrics such as number of visitors, number of visits, dwell time/dwell time breakdown, and device location

●   Address safety and sustainability use cases (occupancy, environmental metrics, and energy monitoring)

●   Setup the infrastructure to allow end users to host or join a video meeting from any device (i.e., mobile, tablet, laptop, or video device) with one consistent experience

●   Enable call control capabilities that are easy to procure, onboard, and manage through a central management portal and manage migration strategies that can leverage on-premises investments

●   Simplify software and license management

Identity Services Engine (ISE)

The team performed ISE set up, provisioning, and DNA Center integration to accommodate asset visibility, access control, and authenticated guest access considerations. The implementation required active directory integration, AAA, and 802.1x setup. As a result of having ISE deployed, the team was able to gather contextual information associated with Who, What, When, Where & How. They were also able to take advantage of tagging to classify wired and wireless devices. Although the current deployment is focused primarily on admission control, there are plans to deliver on some of the more advanced security use cases associated with a Secure Access solution implementation in the future.

Cisco DNA Center

The team leveraged the shared instance of Cisco DNA Center for IT map (heat map) creation and to prepare for secure onboarding of switching infrastructure.

Cisco DNA Infrastructure

The Cisco DNA infrastructure (inclusive of wired and wireless) needed to be upgraded to the appropriate versions as indicated in the table below.

 

Upgrading the infrastructure required referencing a compatibility matrix to ensure proper interoperability and the ability to accommodate the newer Access Points, sensors, as well as integration with Cisco Spaces to take advantage of the latest capabilities at the time. WLANs were confirmed to be deployed within relevant segments to accommodate both guest and corporate users and ensure they could continue to track and remove bad actors (e.g., rogue APs, rogue clients, and interferers).

Cisco Spaces

Although there are many Cisco Spaces configuration guides available here, the team partnered very closely with the Cisco Spaces product team to provision the solution leveraging the process flow shown in the image below:

The team needed to provision the Cisco Spaces Connector on the network to import/sync the Cisco DNA Center map. The connector was installed on a local Hyperflex Edge along with 3rd party Light/Shade management software. Installation of the Cisco Spaces Connector was done to ensure that Cisco Spaces would work with Detect & Locate, IoT Explorer, and IoT Services capabilities seamlessly. The team determined that functionality provided through the integration was required to meet the overall design specifications.

Webex Control Hub, Meeting Site Creation, Call Setup

The Webex Control Hub cloud-based management infrastructure and the creation of the org was already in place leveraging the corporate instance. As this was the case, the initial setup (inclusive of the following considerations) was performed and could be taken advantage of without the need for additional work:       

●   Domain claim

●   SSO

●   Hybrid Calendar (Initial integration and setup vs. mapping that will happen during device deployment evaluation)

●   Directory integration and user onboarding

 

Webex onboarding can be performed by using the resources linked off of the Get Started in Control Hub page. Authorization is required to access the WalkMe videos that map to the following deployment checklist:

All the foundational Webex Meeting Site and call setup requirements were in place and could be leveraged as part of the solution deployment. Although this case study will not address the specific details associated with this area of focus, the Hybrid Work from Office device onboarding and the integration with Cisco Spaces are covered as part of this documented case study.

Lessons Learned and Recommendations

Secure Onboarding

WLAN Controllers and Access Points

The WLAN controllers were used to deploy access points within the environment. All APs point back to DNA Center for proper AP group mapping, placement in DNA Center hierarchy, and placement on the map.

Accurate placement of APs on the map is critical to accurate location analytics in Cisco Spaces, so take the time to ensure the correct APs are placed in the correct locations of the 2D maps.

End User and Hybrid Work Device Onboarding

Onboarding devices leveraged the integration between Cisco DNA Center and ISE. The integration was used to automate the deployment of VLAN and VRF configurations and implement user/device policies to onboard users and devices into the appropriate network segments. As part of standard policy, devices are authenticated with 802.1x and AAA using Cisco Identity Services Engine (ISE) to provision access services via centrally configured policies. ISE also made it possible to profile non-802.1x devices using network telemetry, ISE data, and DNA Center’s cloud-based AI Endpoint Analytics. The onboarded Hybrid Work Devices included:

●   Desk Series

●   Room Series

●   Board Series

●   Signage

 

The team had to ensure the appropriate Internet connectivity, port access, and DNS resolution for the devices to function seamlessly.

The bulk of the setup for Hybrid Work Devices required accessing Webex Control Hub. As all the spaces in PENN 1 are shared spaces, all the devices were provisioned as shared workspaces in Webex. Once on the network, device onboarding was as simple as just entering an activation code on the device to associate the device with the workspace. Hot-desking capabilities and Hybrid Calendaring were also configured within Webex Control Hub where appropriate.

Physical Collab Device Setup

Physical device setup could not be overlooked as the setup of some Hybrid Work Devices is quite time consuming. The team had the overall goal of simplifying physical provisioning of various Hybrid Work devices. They leveraged previous experiences, learnings, and knowledge to connect ancillary offerings to Hybrid Work devices such as headsets, mics, speakers, touch controllers, etc.

Lessons Learned and Recommendations

 

Hybrid Work from Office Integrations

The team was focused on delivering the following results as part of the various product integrations (inclusive of Cisco Spaces, DNA Center, Catalyst Wireless, Webex Control Hub, and Signage):

●   Provide data to partner applications to facilitate indoor wayfinding, conference room utilization tracking, asset tracking, and BLE-based engagement apps

●   Extend and enrich enterprise software applications with contextual location data (marketing automation, CRM, POS, building automation, HRMS, etc.)

●   Correlate data from Cisco Spaces Apps to create richer metrics

●   Ingest Presence, People Count, and environmental data in Smart Workspaces

Cisco Spaces, Webex Control Hub Integration, and Signage (Rich Maps)

The integration between Cisco Spaces and Webex Control Hub was required to perform rich map setup so that signage devices were able to display maps properly.

The high-level steps required for rich map set up and device integration included:

1) Validating Cisco Spaces was set up properly

●   Account with correct license (ACT or Smart Workspaces license)

●   Locations added in location hierarchy (i.e., floor maps, floor numbers, AP placement) - this is part of DNAC to Cisco Spaces integration

●   Added correct metadata (manually) to each location (e.g., time zone, occupancy limits - building and floor level)

 

2) Setting up Webex Control Hub

●   Enabled Workspace metrics at Webex Organization level during initial set up (had to revisit during device set up to enable per device)

●   Ensured devices are online and licensed (navigators)

●   During device provisioning, turned on device sensors to capture workspace metrics (presence, people count, environment, etc.) - The team was able to leverage pre-deployed template configurations

●   Configured Workspaces (calendar, map room types, and room level occupancy)

 

3) Performing the Cisco Spaces to Control Hub integration

●   Activated Control Hub

●   Selected the workspaces where the information needed to be sent from

●   Generated token and pasted into Cisco Spaces. NOTE: This is now automated through an OAuth login flow directly from Webex Control Hub.

 

4) Creating Rich Map

●   Uploaded CAD files to Cisco Spaces portal and ensured each file was associated with location hierarchy

●   Cisco Spaces took the CAD file, converted to a Rich Map, and admin notification occurred once the file was ready for use

●   Admin reviewed, requested changes, and accepted the file

●   Maps (bearing, zoom and pan) were edited and finalized

●   Webex devices were mapped to rich map IDs (i.e., Rooms). This is done from the Cisco Spaces Web UI.

 

Note: At the time of configuration, the team's ability to configure independently was not possible and the capability was under development. All provisioning was handled by the Cisco Spaces team and was a manual process. This is no longer a limitation and can all be done by the customer inside of the space manager in Cisco Spaces.

5) Creating signage

●   Created signage URLs for buildings and floors within Cisco Spaces (ability to configure was limited at the time of implementation and required provisioning to be handled by the Cisco Spaces team)

●   Enabled digital signage for Webex device in Control Hub leveraging created URLs

Lessons Learned and Recommendations

Guest Access

The team had multiple options when considering how they wanted to accommodate the seamless onboarding of visitors. They have traditionally deployed ISE at other locations. They were, however, very interested in Open Roaming functionality available with Cisco Spaces, which is not supported with ISE.

The decision came down to choosing between using authenticated or unauthenticated approaches. The Cisco Spaces Captive Portal did not support sponsored access at the time of deployment, but one could use access codes with proper permissions. Unfortunately, with no device security posturing capabilities offered as part of the Cisco Spaces solution, the team had to adhere to InfoSec requirements and continue leveraging ISE for this location.

Lessons Learned and Recommendations

Implementation: Hybrid Work IoT and 3rd Party Integrations

The team was tasked with identifying and leveraging prebuilt integrations with validated partners to deliver on additional smart building business outcomes. This involved powering the building with PoE and integrating PoE powered lighting and shades. To further deliver on the Hybrid Work from Office user experience, they used Cisco native APIs/SDKs for lighting and shade controls (provided by 3rd party vendors) to integrate to collaboration in-room controls.

The implementation consisted of:

●   Deploying Webex devices (Webex Board, Webex Room and Webex Room Kit with touch panels, Desk Pros, etc.) found on projectworkplace.cisco.com/products

●   Leveraging macros to control lighting & shades

●   Leveraging RoomHub to implement middleware for communicating between video device and IoT)

●   Referencing documentation found on Igor light system for REST API integration

●   Referencing documentation found on SolarTrac shades for REST API integration

●   Leveraging Issue Reporting Tool (UI and macro) on all Webex devices and posting feedback to Webex spaces using Webex SDK / REST API

●   Enabling Voice-Assist for controlling lights and shades in a few demo rooms as a proof of concept of the capabilities of the Webex Assistant Skills features.

 

Connecting Systems and Data (API/SDK)

As part of connecting systems and data to make this possible, the team leveraged Cisco Spaces Meta API and open REST APIs to integrated with custom applications. This integration enabled them to provide presence, people count, and environmental data to 3rd party data consumers.

Lessons Learned and Recommendations

Operations: IT Observability, Management, and Assurance

Supporting Day 2 operations is critical, and the team is consistently looking for new ways to:

●   Improve visibility into the Health and Performance of Users, Devices, and Applications

●   Reduce Mean Time to Resolution of Reported Issues

●   Leverage alert notifications for network and client issues to proactively address impact to end user experiences

●   Ensure a clear line of sight between user locations and Webex services to optimize performance and ensure exceptional Webex user experiences

Monitoring and Troubleshooting

Identifying new ways to perform monitoring and troubleshooting using the tools available is critical, and the team is consistently working within Webex Control Hub, DNA Center, and ThousandEyes when troubleshooting site issues. They use Cisco DNA Center to discover wired and wireless network devices and immediately begin collecting telemetry. Cisco DNA Center automatically correlates telemetry data and creates automated baselines that provide context for network metrics and events. The team is effectively able to analyze the telemetry data to take advantage of insights and visibility into overall network, client, and application health. Identifying anomalous events by comparing telemetry data to established baselines has become a best practice. When device specific issues are encountered, the team uses the 360-degree views for each device connected to the network to understand anything that may potentially be impacting the device that is challenged. Comparing current and historical end-user location data enhances the team's ability to provide enhanced support.

Below is a table highlighting areas of consideration and the platforms used to support ongoing operations:

Lessons Learned and Recommendations

Voice Assistant

The team recognized the need to enable voice assistant functionality to enhance the user experience. They decided to create custom skills using the Webex Assistant SDK to further enhance integrations (voice activated lighting and shade control).  It was important to leverage 3rd party solution API documentation and work with the appropriate talent to develop the necessary Skills.

Conclusion

Cisco’s PENN 1 office was the first of many Cisco offices that have been transformed from traditional workspaces to smart workspaces tailored to the demands of hybrid work. The office provides a variety of spaces that draw employees to the office for collaborative experiences that might not be as effective remotely, while still being inclusive of remote participants and allowing those who could not make the commute the opportunity to participate. The site is a showcase of how technology can make the office welcoming to hybrid workers while providing the visibility to IT and facilities staff and helping the company get closer to our net-zero goals. This deployment was a learning experience, and we hope our journey helps you achieve your hybrid work from office outcomes.h