Mastering Regex with GitHub Copilot for Enhanced Azure WAF Security
Written in collaboration with davidfrazee Introduction Azure Web Application Firewall (WAF) is a cloud native security service that provides protection for web applications from common exploits and vulnerabilities. It provides centralized protection for applications hosted on Azure Front Door and Azure Application Gateway ensuring that malicious traffic is detected and blocked before reaching the application backend. Azure WAF leverages managed rulesets to actively protect web applications from threats and attacks. These rule sets are maintained by Azure, with the Default Ruleset (DRS) including rules from the Microsoft Threat Intelligence Collection, ensuring enhanced coverage, specific vulnerability patches, and improved false positive reduction. In addition to the managed rulesets, Azure WAF offers custom rules that enable you to create your own rules. With custom rules, you can set conditions based on attributes such as IP addresses, HTTP headers, and query strings to precisely control which traffic is allowed or blocked, providing flexibility and granularity. Within the custom rules, you can incorporate regex, which offers enhanced accuracy when matching patterns in your traffic. Regex (regular expressions) enable you to define complex conditions, allowing for highly specific filtering of incoming requests. Working with regex can sometimes be challenging due to its non-intuitive syntax. In this blog, we will demonstrate a practical, step-by-step approach for generating regex patterns using GitHub Copilot, refining them on Regex101, and validating their effectiveness in Azure WAF. This process helps ensure that your custom rules with regex work as intended, thereby enhancing your overall security effectiveness. GitHub Copilot GitHub Copilot is an AI-powered code completion tool developed by GitHub in collaboration with OpenAI. It assists developers by suggesting code snippets, functions, and even entire blocks of code as they type. By leveraging machine learning models trained on a vast amount of public code, GitHub Copilot can understand the context of the code being written and provide relevant suggestions, making the coding process faster and more efficient. Prompting GitHub Copilot can be particularly useful for security professionals. Enhanced code quality is one of the benefits, as GitHub Copilot can help security professionals write cleaner and more secure code by identifying potential vulnerabilities and suggesting best practices for secure coding, thus reducing the risk of introducing security flaws. Additionally, it offers time efficiency, as security professionals often need to write scripts or tools to automate security tasks, and GitHub Copilot can speed up this process by generating code snippets based on the prompts provided, allowing professionals to focus on more critical aspects of their work. GitHub Copilot can also assist in creating regex code, which is often complex and challenging to write. By providing accurate regex patterns based on prompts, it can help security professionals quickly develop effective text manipulation and pattern matching solutions. While AI-generated content can significantly streamline the process of creating regex patterns, it is important to verify the accuracy of these patterns to ensure they work as intended. Tools such as Regex101 provide a valuable platform for refining and validating regex patterns, helping to identify and correct any errors before implementation. What is Regex? Regular expressions, commonly known as regex, are sequences of characters that define search patterns. They are used in various programming languages and tools to match, locate, and manage text. Regex is incredibly powerful because it allows for complex text manipulation and pattern matching, making it an essential tool for tasks such as data validation, parsing, and transformation. However, regex can be difficult to learn due to its intricate syntax and the need for precise pattern construction. The learning curve is steep because even small mistakes can lead to unexpected results or errors. Additionally, regex patterns can become quite complex, requiring a deep understanding of both the syntax and the specific text being manipulated. Despite these challenges, regex is invaluable in the field of security. It can be used to detect and prevent malicious activities by identifying patterns associated with attacks, such as SQL injections or cross-site scripting. Regex can also help in filtering user inputs, ensuring that only safe and valid data is processed. Its ability to analyze large volumes of text swiftly and accurately makes it a powerful tool for security professionals aiming to protect systems and data from threats. Creating & Validating Regex Patterns Now that we have explored the capabilities of GitHub Copilot and the intricacies of regex, it is time to put this knowledge into practice. In the following steps, we will guide you through the process of leveraging GitHub Copilot to create and validate regex patterns effectively. By combining the power of AI-driven code suggestions with the precision of regex, you will be able to enhance your security tasks, automate complex text manipulations, and ensure robust data validation. Let us dive into the practical steps to get this done and see how these tools can streamline your workflow and bolster your security measures. To get started with GitHub Copilot, navigate here. We are going to leverage GitHub Copilot through Visual Studio Code (VSC). Once you have completed the requirements from the link above and you have enabled the extension, you can start prompting GitHub Copilot Chat. Prompt: Create two unique regex patterns. The first pattern should match any of the following words: "l33t", "h@x0r", "kewl", and "n00b", but the characters can be case insensitive. The second pattern should match the string of special characters "!@#$" followed by any integer that comes after the symbol "&". Response: Here are the two regex patterns: To match any of the words "l33t", "h@x0r", "kewl", and "n00b" in a case-insensitive manner: (?i)\b(l33t|h@x0r|kewl|n00b)\b To match the string of special characters "!@#$" followed by any integer that comes after the symbol "&": !@#\$&\d+ Now that GitHub Copilot has delivered 2 unique regex strings for use in our Azure WAF, let’s verify if the regex patterns work with the use of https://regex101.com. Regex101.com is an online platform designed to help users create, test, and debug Regex. It provides a user-friendly interface where you can input your regex patterns and evaluate them against sample text to see how they match. The site offers real-time feedback, highlighting matches and providing detailed explanations of each part of the regex pattern. This makes it an invaluable tool for both beginners learning regex and experienced users fine-tuning their patterns. Additionally, Regex101.com supports multiple regex flavors, including PCRE, JavaScript, and Python, allowing users to work with the syntax specific to their needs. Above, we see how we are validating the regex pattern at Regex101.com. I paste my regex pattern provided by GitHub Copilot at the top and then enter a JSON test body to match against the pattern. The tool verifies that the first regex pattern captures malicious attempts without case sensitivity and provides a detailed breakdown on the right side. This breakdown includes explanations of each part of the regex, helping to ensure that the pattern is correctly identifying the intended matches and highlighting any potential issues. In another example, we are using Regex101.com to validate a regex pattern aimed at identifying strings of unique characters. The tool verifies that the regex pattern successfully captures the string where each character appears only once and in order, followed by an integer. On the right side, Regex101.com provides a detailed breakdown of the regex pattern, explaining how each part contributes to the overall match. Now that we have validated the regex patterns with Regex101.com, let us implement them into Custom rules for Azure WAF. Using Regex with Azure WAF Having validated the regex patterns with Regex101.com, we can now proceed to implement these patterns into Custom rules for Azure WAF. This section provides a guide on integrating the validated regex patterns into your Azure WAF configuration to enhance web application security. By establishing these custom rules, you can tailor protection to meet specific requirements, ensuring malicious attempts are effectively intercepted and blocked. First, we will navigate to the Custom rules section of our Azure WAF policy, and author the two regex rules that we want to use to identify special patterns in request bodies going through our WAF. What is unique about using regex in Custom rules, is that you select Regex as an Operator in the Condition. From there, you will enter your regex pattern in the Match values section, select the action and the Custom rule is complete. After implementing the custom regex rules into Azure WAF, we executed a simulated malicious attempt to evaluate their effectiveness. The WAF, equipped with our regex patterns, successfully detected and intercepted the attack. The custom rules accurately identified the malicious activity and promptly blocked it, demonstrating the power and precision of using AI-generated regex patterns to enhance security measures. After executing the simulated malicious attempt, we examined the Azure WAF logs to confirm the effectiveness of our custom regex rules. The logs clearly indicated that the attack was intercepted, with the highlighted rule name and match value providing specific details about the block. This information is crucial for verifying that the custom rules are functioning as intended and accurately identifying malicious activities. By reviewing these logs, we can ensure that our security measures are robust and capable of protecting against potential threats. The detailed log entries not only confirm the success of our regex patterns but also offer insights into further refining and optimizing our security configurations. Conclusion Leveraging GitHub Copilot to generate regex patterns and validating them on Regex101.com before applying them to Azure WAF showcases the remarkable synergy between AI and security practices. By utilizing GitHub Copilot's intelligent code suggestions, we can efficiently create complex regex patterns tailored to our specific needs. Validating these patterns on Regex101.com ensures their accuracy and effectiveness in capturing malicious attempts. Once applied to Azure WAF, these regex patterns enhance our security measures, providing robust protection against potential threats. Testing and observing the impact of these AI-generated regex strings highlight the power and value of integrating AI into our security workflows. This approach not only streamlines the process but also demonstrates how AI can significantly contribute to hardening security, making it an efficient and worthwhile endeavor. References Introduction to Azure Web Application Firewall | Microsoft Learn What is Azure Web Application Firewall on Azure Application Gateway? | Microsoft Learn What is Azure Web Application Firewall on Azure Front Door? | Microsoft Learn Create and use v2 custom rules - Azure Web Application Firewall | Microsoft Learn GitHub CopilotAzure Firewall integration in Security Copilot: protect networks at machine speed with gen AI
Today, at Microsoft Build, we are excited to announce the availability of the Azure Firewall integration in Security Copilot. It helps analysts perform detailed investigations of the malicious traffic intercepted by the IDPS feature of their firewalls across their entire fleet using natural language questions in the Security Copilot portal. Azure Firewall is a cloud-native and intelligent network firewall security service that provides best of breed threat protection for your cloud workloads running in Azure. It's a fully stateful firewall as a service with built-in high availability and unrestricted cloud scalability. In this blog we will walk through the newly announced Azure Firewall integration in Security Copilot. These capabilities were announced at RSA. Take a look at this blog to learn more about the user journey and value that Copilot can deliver: Bringing generative AI to Azure network security with new Microsoft Copilot integrations. There are four primary capabilities now in preview which are outlined below. Get top IDPS signature hits This capability retrieves the top IDPS signature hits for an Azure Firewall. It helps the user get information about the traffic intercepted by the IDPS feature by simply asking natural language questions instead of the user having to construct KQL queries manually. Get details on an IDPS signature This capability enriches the threat profile of an IDPS signature beyond the information found in logs. It helps the user get additional details about an IDPS signature instead of requiring them to manually source this information. Search across firewalls for an IDPS signature This capability looks for a given IDPS signature across your tenant, subscription or resource group. It helps users perform a fleet-wide search (over any scope) for a threat across all their Firewalls instead of searching for the threat manually. Secure your environment using IDPS This capability generates recommendations to secure your environment using Azure Firewall’s IDPS feature. It helps users get information from documentation about using Azure Firewall’s IDPS feature to secure their environment instead of having to look up this information manually. Get started Learn more in our documentation about these capabilities and how to access them today!Copilot in Azure embedded experience for Azure Firewall integration in Security Copilot
Today, at Microsoft Ignite, we are excited to announce that we are building on our announcement at RSA and Microsoft Build earlier this year and integrating Security Copilot even more closely with our Network Security products. These capabilities were launched on the Security Copilot portal (also called the standalone experience) earlier this year. The Security Copilot attack investigation capabilities for Azure Firewall can now be queried via the Copilot in Azure experience (also called the embedded experience) directly on the Azure portal where you regularly interact with your Azure Firewalls, bringing interactive, generative AI-powered capabilities even closer to where you work. To learn more about the user journey and value that Copilot can deliver, see Bringing generative AI to Azure network security with new Microsoft Copilot integrations | Microsoft Azure Blog. Investigating Azure Firewall IDPS attacks using Copilot As a member of your organization’s network security team, it is imperative that you understand the kinds of threats your network security devices are intercepting. Azure Firewall intercepts and blocks malicious traffic using a deep-packet inspection technology called IDPS (Intrusion Detection and Prevention System) today. However, when you need to perform a deeper investigation of the threats that Firewall catches using IDPS, you need to do this manually - which is a non-trivial and time-consuming task. The Azure Firewall integration in Security Copilot helps analysts perform these investigations with the speed and scale of AI. Retrieve the top IDPS signature hits for an Azure Firewall The first step in an investigation is to pick a specific Azure Firewall and see the threats it has intercepted. Analysts today spend hours writing custom queries or navigating through several manual steps to retrieve threat information from Log Analytics workspaces. With Copilot, you just need to ask about the threats you'd like to see, and Copilot will present you with the requested information. Azure Copilot presents the top 5 IDPS signatures flagged in the requested time period along with a brief summary of why each signature is a threat and volumetric information on the number of flows associated with each signature. Enrich the threat profile of an IDPS signature beyond log information The next step in an investigation is to better understand the nature and impact of these threats. Today, analysts must retrieve additional contextual information such as geographical location of IPs, threat rating of a fully qualified domain name (FQDN), details of common vulnerabilities and exposures (CVEs) associated with an IDPS signature and more, manually from various sources. This process is slow and involves a lot of effort. Copilot pulls information from the relevant sources to enrich your threat data in a fraction of the time. Copilot provides a brief summary of both the attacks as well as a comparison between the two, emphasizing the differing severities and why it is crucial to respond to these threats quickly. You can drill deeper if you’d like by asking follow up questions or by asking the same question again to get Copilot to reinforce or substantiate any of its answers. For example, reinforcing that you’d like to understand how much of a threat an attack is, even though Copilot may have already provided an answer to that question, is a great way for junior analysts, or anyone operating outside of their normal area of focus to truly understand what is happening. Look for a given IDPS signature across your environment Once a detailed investigation has been performed for a single Azure Firewall and single threat, analysts would like to determine if these threats were seen elsewhere in their environment. All the manual work you performed for an investigation for a single Azure Firewall is something you would have to repeat fleet wide. Copilot can do this at machine speed and help correlate this information with other security products to better understand how attackers are targeting your entire infrastructure. Copilot searches across your entire tenant and finds that another Firewall also saw one of the attacks over the timespan you defined. Your suspicions are unfortunately confirmed. This is a threat that is targeting multiple points of entry in your environment. You can ask a follow up question to search for another high severity threat as well, now that you know that at least one threat was not contained to a single Firewall and has proliferated across your environment. Secure your environment using IDPS Now that you are convinced this attack warrants attention from your organization, as a first step, you can ask Copilot for some recommendations on how to better use your Firewall to protect against these kinds of attacks. Copilot produces a response that combines contextual information from your conversation alongside general network security best practices and specific guidance from Azure Firewall documentation to produce a response that is informative. Looking forward In addition to the open prompting experience covered in this blog. We are also working on embedding Copilot directly into the Firewall portal so that you can simply click buttons with suggested prompts that automatically pull relevant information from context and generate helpful responses – making it easy to invoke Copilot when contextually relevant while still giving you complete control over its usage. embedded experiences on the Azure portal We are also excited to share a sneak peek of upcoming capabilities like Natural language to KQL for IDPS that are currently being tested by a small group of customers. This capability can act as a query assistant helping you craft complex queries but can also help find answers to questions you have by running automatically generated queries on the appropriate Log Analytics workspaces to retrieve the relevant data. This is only the start of our journey toward infusing AI into every aspect of our Network Security offerings making it easier for our customers to be more productive and quickly analyze threats and mitigate vulnerabilities to stay ahead of their adversaries. These capabilities are in preview and over the coming weeks we look forward to adding new capabilities and making improvements based on your feedback. Get started Learn more in our documentation about these capabilities and how to use them today!Azure Firewall Protection Against Apache Struts Vulnerability - CVE-2023-50164
In this blog, we delve into the critical Apache Struts vulnerability CVE-2023-50164, demonstrating how Azure Firewall Premium can effectively block this high-severity threat. Learn about our detailed proof-of-concept setup, see how Azure Firewall Premium's Intrusion Detection and Prevention System (IDPS) protects against such vulnerabilities, and explore how Copilot for Security enhances the investigation process, ensuring your systems remain secure.
