Welcome to today's session, where we will delve into the validation, that's detection pillar of Cisco AI defense. We will explore how this component enhances security and ensures the robustness of AI applications. Let's begin our journey by understanding the foundational aspects of this system.00:04To start, let's define what validation means in this context. Validation involves automated algorithmic red teaming, which simulates adversarial attacks to identify security, privacy, and safety weaknesses before deployment. This process is crucial because it allows us to conduct continuous scalable testing reducing the extensive time typically required.00:22For manual testing, through automated algorithmic red teaming.00:42We can detect known vulnerabilities that every AI system should withstand continuously providing a measurable security baseline. We'll also discuss the key benefits of validation such as comprehensive security testing, automation, scalability, and the ability to align with industry standards. Throughout this session, we will examine the end to end setup of AI validation.00:45How to execute it, interpret results and apply best practices.01:06Now, let's dive into how validation is integrated within Cisco AI defense through its graphical user interface. Begin by navigating to Cisco Security Cloud control, a unified platform for management and policy orchestration across Cisco's security portfolio.01:11You can access it at HTTPS colon/slash security.cisco.com. Once logged in, we will move on to the next steps. Once inside the security cloud control, locate the AI defense section under the products category on the left pane.01:26This navigation will guide us to the necessary tools where we can manage and oversee AI defense operations efficiently. Next, click on the validation option under the AI defense section. This will take us to the interface where you can manage past validations and set up new ones.01:42Here at the validation page, you'll notice four tabs, overview, targets, profiles, and custom goals. The overview tab lists past validations while targets define the AI systems to test. Profiles offer reusable scanning configurations and custom goals let you test for application specific risks using natural language objectives.02:00These features ensure comprehensive and customizable validation processes. For configuring a target, which is essential for AI validation, go to add component and select add target. This step connects AI validation with your AI agent, model or application, setting the stage for thorough testing.02:19This section explains target setup, identifying the types of targets you can define, such as a model, application or agent. It also details the providers, including AWS bedrock, custom endpoints, and others, highlighting when to use each based on your system's location and needs.02:40When selecting AWS bedrock as your provider, you will need to configure specific connection details. These include setting the data plane, endpoint, and AWS account ID. This configuration ensures seamless integration and management across multiple AWS accounts.02:58Crucial for expansive enterprise deployments. Upon completion, clicking create target will save this setup for future validations. Validation profiles allow for configuring reusable scan settings, ensuring consistent testing parameters across different targets. Custom goals offer flexibility for testing specific scenarios.03:14Videos relevant to your application.03:34Helping identify unique vulnerabilities. To create a custom goal, navigate to the add component and then click add custom goal section. In this section, we discuss the setup and management of custom goals within the validation system. Start by assigning a short descriptive name to your goal such as unauthorized refund processing.03:36The name serves as a reference point for identification. Next, provide a detailed description of the attacker's objective. E.g., convince the agent to issue a refund without manager approval or proper verification. This goal description is pivotal as a specific description allows the system to generate more targeted attack scenarios.03:55Once you have detailed the goal, click Save to store your changes. It is important to note the behavior of custom goals in different modes. In multi turn mode, the system emulates a patient attacker, escalating threats gradually across multiple interactions. In single turn mode, the system evaluates responses to isolated prompts from its attack bank. Custom goals.04:13Are tenant wide, meaning all users in your organization can view and leverage them.04:33Ensuring comprehensive coverage of potential vulnerabilities. Now, we move on to running a validation test. There are two primary entry points to initiate a test run. 1st, you can select the run validation option for standalone test setup. Alternatively, use the validate link on the target detail page for a quick retest of a specific target.04:38These options provide flexibility based on your requirements, whether starting fresh or retesting a previous target with new settings.04:58It's crucial to understand your test objectives prior to selection as each entry point is suited to different scenarios. Next, we define the configuration for running a test. Begin by naming the test run appropriately such as validation for AWS novolite. This naming helps in identifying and organizing test runs later.05:05Choose the target AI system from your save list. Select the appropriate profile to determine the scope and depth of the test. The choice of a prompt bank is also crucial. You can choose between a default which is a larger scan for comprehensive testing or a quick scan that's focusing on top selected prompts for an expedited assessment.05:24Once these configurations are in place, click run validation to initiate the test. Keep in mind that the duration of the test could vary based on the size of your profile, target latency, the number of custom goals, and whether multi turn interactions are enabled among other factors.05:41Upon completion of a test run, click to view detailed results. Here, you'll find critical insights including the overall risk score, specific findings per category and custom goal, and mapping to standards such as a WASP, mitreatlas, and NIST. These results offer valuable feedback on your AI system security posture.05:58Detailing which areas are robust and which require attention. Ensure you thoroughly examine these insights to guide remediation efforts effectively. Navigating to the deep dive section provides options to filter results by technique, sub technique or procedure. By expanding the procedure box, you can see a comprehensive list of over 55.06:16Procedures used in crafting attacks.06:36Two display options are available, a graphical view demonstrating attacks blocked by sub technique at a list view of each test run attack. This dual perspective allows for easy analysis of your systems defenses against various threat vectors. Switch to the list view by clicking the respective icon below overview.06:39In this view, each test result is displayed in detail. Pay particular attention to the severity column where an alert indicates a failed test. Understanding these failures is crucial in identifying weaknesses and focusing remediation efforts where they are most needed. This insight is instrumental in maintaining robust security defenses.06:57Click on any row in the list view to open the detailed side drawer. Here, expand the results section to review the full prompt used in the model's response. This enables you to see the range of attacks tested from prompt injection to harmful content generation and how effectively the model responded. Reviewing these interactions provides a clear picture of potential vulnerabilities and.07:16And the effectiveness of current security measures guiding future improvements.07:36In our final section, we focus on establishing a reliable validation cadence to maintain security consistently. It is critical not to view validation as a one time process, instead, integrated into regular workflows. With every code change, use quick standard validation profiles.07:42For model changes, employ full standard validation profiles prior to a launch, in corporate standard validation, adaptive red teaming, and custom goals. Perform a full audit quarterly including all custom goals. Post incident, run targeted custom goals relevant to the incident.07:59Strategically use different profiles tailored for fast checks, comprehensive pre launch scanning, audits, and regression testing. Maintain a library of custom goals, beginning with essential business risks and adapting as your environment evolves. Ensure findings from validation drive actionable workflows.08:15From triaging to remediation and verification, applying these best practices ensures efficient and effective security measures across all facets of your AI systems, maintaining alignment with both internal and industry standards.08:31
