The Age of Full Stack Observability Solution Overview

 

Why technologists must act now to optimize application performance and deliver sustainable innovation within hybrid IT environments

Executive summary

Rapid adoption of cloud native technologies is enabling technologists to dramatically improve speed to innovation, while offering greater agility, reliability and scalability.

But modern application architectures are posing serious challenges for IT teams across all industries—expanded attack surfaces, soaring complexity, and overwhelming volumes of data. A lack of unified visibility into application paths spread across on-premises and cloud native technologies is making it harder for technologists to identify and resolve availability, performance, and security issues before they impact end-user experience.

Within the IT department, tension is growing as silos emerge between new teams that have been created to implement and manage cloud native technologies. Operations and security teams are working in isolation, implementing their own processes and tools, and relying on their own siloed data.

At the same time, IT leaders are coming under increased scrutiny to demonstrate the value that cloud investments are delivering. As costs for cloud computing rise and technologists struggle to validate their investments, many organizations are slowing down their cloud migration plans.

Indeed, there is now a growing realization that most organizations will be operating hybrid IT environments for many years to come. While cloud native technologies will steal the headlines, on-premises technology will continue to play a pivotal role.

The shift from monitoring to observability

This report sets out the challenges that IT departments are encountering as they attempt to manage an increasingly fragmented IT estate. In particular, it exposes the limited levels of visibility that are making it so difficult for IT teams to swiftly troubleshoot application performance issues.

The research highlights the need for new thinking within IT departments, and the need for technologists to develop new skills to operate effectively within a hybrid environment.

Significantly, the research reinforces the need for organizations to move beyond traditional approaches to application performance monitoring and to implement full-stack observability solutions. Full-stack observability provides a single source of truth for all technologists, enabling IT teams to integrate application availability and performance data with security throughout the application lifecycle, and making it easier for technologists to rapidly detect and remediate issues.

Across all industries, organizations are turning to full-stack observability to cut through IT complexity and deliver the seamless digital experiences that customers now demand at all times.

As such, full-stack observability has become the foundation for organizations to embed accelerated innovation into their operations and to achieve their business objectives moving forward.

78% of technologists state that the increased volume of data from multi-cloud and hybrid environments is making manual monitoring impossible.

85% of technologists state that observability is now a strategic priority for their organization.

Research methodology

Cisco has undertaken comprehensive global research, from board-level directors and CIOs, through to senior and mid-level IT management.

This research entailed:

●     Interviews with 1,140 IT professionals in organizations with a turnover of at least $500m (with the exception of Colombia, where organizations with a turnover of at least $100m were included in the sample)

●     Interviews were conducted in 13 markets– Australia, Brazil, Canada, Colombia, France, Germany, India, Japan, Mexico, Singapore, United Arab Emirates, United Kingdom and United States

●     Respondents worked across a range of industries, including financial services, retail, public sector, IT, manufacturing and automotive, and media and communications

●     All research was conducted by Insight Avenue in April 2023

Note:       Totals in charts/tables for single coded questions sometimes add up to more or less than 100% due to rounding.

Cloud native and the push for accelerated innovation

Organizations are embracing cloud native technologies as they look to embed speed and agility into their development processes.

On average, technologists report that 49% of their new innovation initiatives are being delivered with cloud native technologies, and they expect this figure to climb to 58% over the next five years. That means that the majority of new digital transformation programs will be built on cloud native technologies by 2028.

There are a wide range of perceived benefits for organizations migrating to cloud native technologies, including greater reliability, enhanced customer experience, and more robust application security.

The top five benefits of migrating to cloud native technologies

●     Improved reliability

●     Improved customer experience

●     Improved application security

●     Speed of application development

●     More responsiveness/agility

Almost all technologists believe that increased adoption of cloud native technologies will accelerate application development velocity within their organization, and the majority expect to deliver applications at least four times faster than with traditional, on-premises technology.

Expected increase in application development speeds due to migration to cloud native technologies

Shows how much faster technologists expect application development to be if using cloud native technologies.

The future is hybrid… which means increased complexity

Despite the advantages that cloud native technologies are already delivering, the economic slowdown and a tightening of IT budgets have pushed some organizations to take a more rigorous approach to cloud migration. For many IT leaders, the costs associated with cloud native technologies are higher than they had originally anticipated and difficult to budget for, while others are keen to maintain the additional control and visibility that on-premises IT provides.

Indeed, most technologists believe that on-premises IT still has an important role to play within their organization, and 92% state that multi-cloud and hybrid environments are here to stay.

However, this move towards hybrid environments is presenting IT departments with serious challenges. Technologists point to an expansion of attack surfaces and increased vulnerability to cybersecurity threats. With application components running across a mix of cloud native platforms and on-premises databases, visibility gaps are being exposed and the risk of a security event is becoming heightened.

The challenges of managing multi-cloud and hybrid IT environments

Shows how respondents rate the challenges of managing multi-cloud and hybrid IT environments.

83% of technologists state that adoption of cloud native technologies is leading to increased complexity within their IT departments, with microservices and containers spawning a huge volume of Metrics, Events, Logs and Traces (MELT) data.

Significantly, the move to hybrid environments isn’t just presenting technical challenges, it is also having a more personal impact on technologists. The majority report that silos are emerging within their IT department, and IT teams are coming under greater pressure. Ultimately, this is leading to increased stress and higher rates of churn.

The top five consequences of increased complexity across multi-cloud and hybrid environments

●     Increased vulnerability to security threats/ more security blind spots

●     Poor application performance impacting customers/employees

●     Slowdown in release velocity, impacting any competitive advantage

●     Erosion of team morale/increase in staff churn

●     Reduced efficiency and productivity within IT teams

Departmental silos and the need for greater collaboration

Modern applications are being built with a very different approach to traditional applications, by new teams with new skills and new ways of working. Over recent years, IT departments have introduced Site Reliability Engineers (SREs), DevOps and CloudOps teams to accelerate adoption of cloud native technologies. In the meantime, teams are also in place to maintain existing applications, while transformation projects are in motion.

But this has created new silos within IT departments, even more pronounced than those that existed before. 80% of technologists point to an increase in silos between IT teams as a result of managing multi-cloud and hybrid environments, and only a third (31%) report that there is ongoing collaboration between their IT Operations and security teams.

Almost all technologists point to at least one barrier to improved collaboration within their IT department, and many of these are cultural, such as skills gaps, management and reporting structures, and a lack of shared vision and objectives across the IT department. Interestingly, however, the biggest barrier to collaboration between IT teams is the use of technology and tools that reinforce these silos.

Shows how often IT operations and security teams collaborate.

The five biggest barriers to greater collaboration between IT teams

●     Technology/tools that reinforce silos

●     Skills gaps

●     Management and reporting structures

●     Lack of shared vision/objectives

●     Lack of unified data

Alarmingly for IT leaders, growing silos have the potential to cause a talent exodus from their IT departments if they are not addressed. 36% of technologists claim that silos and ineffective collaboration are already resulting in IT talent leaving their organization, and 46% predict that churn within their IT departments will increase if silos persist.

Most technologists report that they are open to working in a more collaborative way and sharing KPIs with other teams. The challenge, however, is for organizations to implement the right structures, processes and tools to embed more collaborative behavior in their IT departments.

Likelihood of IT talent leaving organizations if silos persist in the IT department

Shows how likely it is for IT talent to leave because of organizations silos.

The need to move beyond monitoring

Many IT departments are still deploying separate tools to monitor on-premises and cloud applications, and this means they have no clear line of sight of the entire application path across hybrid environments.

Technologists are having to operate a split screen mode and are therefore unable to see the complete path up and down the application stack. As a result, it is difficult to swiftly troubleshoot issues and key metrics such as Mean Time To Resolution (MTTR) inevitably go up.

At a broader level, less than half of technologists are fully confident that their organization currently has the right skills, processes and tools in place to manage application availability, performance, and security in a sustainable way.

78% of technologists state that the increased volume of data from multi-cloud and hybrid environments is making manual monitoring impossible.

Full-stack observability is now the foundation for sustainable innovation

As innovation velocity increases, IT teams will need to adopt new ways of working and embrace new tools and technologies to keep pace.

More than anything else, technologists point to observability across multi-cloud and hybrid environments as important for their organization to deliver accelerated and sustainable innovation.

Alongside this, organizations need to work with the right technology partners, develop the right skills within their IT department, and create a shared vision and execution plan across all IT teams to achieve their innovation goals.

Significantly, 97% of technologists point to a critical need for their organization to move from a monitoring approach to an observability solution in order to manage multi-cloud and hybrid environments. More than half (53%) of organizations are already exploring observability solutions, and 44% are planning to do so in the next 12 months.

The research uncovers a wide range of benefits that technologists believe full-stack observability will deliver to their organization. These include the ability to link IT performance to business metrics, deeper insight to detect and solve the root cause of issues, and the ability to bring together dispersed IT infrastructure.

Observability is also viewed as important in improving productivity and efficiency in the IT department, forging closer collaboration between teams and supporting the development of new skills and ways of working among technologists.

Organizational requirements to deliver accelerated and sustainable innovation

Shows how respondents rate needs for delivering accelerated and sustainable innovation.

85% of technologists state that observability is now a strategic priority for their organization.

While technologists understand the urgent need for full-stack observability, many still have work to do to convince business and IT leaders within their organization do not fully understand that modern applications need new approaches and tools to manage availability, performance, and security.

Evidently, technologists will need to focus on building strong business cases to secure investment in new full-stack observability solutions. And in order to do this, they themselves need to get to grips with this rapidly evolving market. The research uncovers demand for greater education and clarity within the full-stack observability market, and technologists will look to work with trusted partners to identify specific organizational requirements and analyze the capabilities of new solutions.

The top five benefits of observability over traditional monitoring solutions

●     Ability to link IT performance to business outcomes

●     Deeper insight/ability to detect and solve root cause of a problem

●     Improved logging, providing early warning of anomalies or unauthorized access

●     Works across dispersed IT infrastructure, multiple tools and applications

●     Improved end user experience

64% of technologists admit that they find it difficult to differentiate between different observability and monitoring solutions.

The critical elements of full-stack observability

Technologists have clear but rapidly evolving views on what full-stack observability must deliver for their organization.

Most of all, technologists want an observability solution that integrates application availability and performance data with security, and that enables them to link IT performance with business outcomes.

Other important elements are speed of deployment and the delivery of a single source of truth for all availability, performance, and security data. This is considered essential in breaking down silos and foster greater collaboration between teams.

As digital transformation budgets come under heightened scrutiny, technologists are under pressure to validate the impact of cloud investments. New teams, such as SREs, need to demonstrate the value they are adding so that they are seen as revenue generating, rather than an overhead.

The research shows that technologists regard full-stack observability as critical to achieving this. The ability to generate business transaction insights in real-time, and then to view them in business-level dashboards, is vital for IT teams to measure the value that their innovation programs are generating and focus their investments in the right places.

The key characteristics of an observability solution

Shows what organizations rank as top characteristics when evaluating observability solutions.

89% of technologists report that their organization’s expectations around observability solutions are increasing.

88% of technologists claim that observability with business context will enable them to be more strategic and spend more time on innovation.

The research highlights the role that Artificial Intelligence (AI) and automation must increasingly play within full-stack observability, to optimize performance and to identify and resolve security vulnerabilities at every stage of the application lifecycle. Automation is seen as important in areas such as user experience, security response, cost optimization, and workload optimization in order to handle the vast volumes of data and complexity that IT teams are now encountering. Full-stack observability needs to be plugged into the CI/CD application development pipeline and to provide log analytics.

Technologists also point to the need for their organizations to introduce digital experience monitoring over the next two years to measure the impact of their applications and digital services, and to deliver more seamless experiences to customers and employees.

Importance of new approaches within the IT department within next two years

Shows what use cases respondents think are most important in the next two years.

Conclusion - technologists must take a strategic approach to full-stack observability

IT teams find themselves trying to manage an ever more complex and fragmented IT estate. They’re struggling to get visibility into applications and underlying infrastructure across complex and fragmented hybrid environments.

Most technologists have no way to handle the huge volumes of data that are being spawned every second by containers and microservices within large, managed Kubernetes environments running on public clouds, or self managed on-premises.

At the same time, as many cloud migration programs slow down in light of challenging economic conditions, technologists cannot take their eye off the ball when it comes to on-premises technology.

Organizations therefore need to implement an full-stack observability solution that provides flexibility to span across both cloud native and on-premises environments —with telemetry data from cloud native environments and agent-based entities within legacy applications being ingested into the same platform.

Encouragingly, the research paints a positive picture. Momentum around observability is building, with more than half of organizations already exploring solutions, and many more planning to do so in the next 12 months. The challenge now is for technologists to turn interest and intent into investment and implementation, ensuring they have the knowledge and support to select the right full-stack observability solution for their organization. They need to lean on trusted partners to build a compelling business case to secure budget and sponsorship from senior leaders.

Technologists also recognize that the shift to an observability approach extends beyond the implementation of new tools and technologies. It requires a holistic strategy that combines cultural change, significant upskilling and new structures and operating models. IT teams need to leave behind entrenched mindsets and siloed ways of working. In a hybrid environment, technologists need to be brought together by common purpose and goals, developing a broader outlook that is focused on wider organizational objectives.

Full-stack observability is now regarded as the foundation for organizations in all sectors to deliver seamless digital experiences and to achieve their ambitions for rapid and sustainable innovation in the future.

About Cisco Full-Stack Observability

Cisco is a leading provider of Full-Stack Observability solutions. Cisco provides real-time visibility, insights, and recommended actions enriched with business context, enabling ops teams to proactively identify, prioritize, and resolve issues to deliver exceptional digital experiences. Underpinning the Full- Stack Observability portfolio is the Cisco Observability Platform which correlates real-time telemetry from data sources across multiple operations domains—including applications, multicloud infrastructure and cloud services, network, security, end users, the business, and more—to help teams understand risks and dependencies across environments, strengthen security measures, and optimize resources across the full technology stack before it impacts the business.

To learn more about the security solutions offered as part of the Cisco Full-Stack Observability portfolio click here.