Mastering Regex with GitHub Copilot for Enhanced Azure WAF Security
Written in collaboration with davidfrazee Introduction Azure Web Application Firewall (WAF) is a cloud native security service that provides protection for web applications from common exploits and vulnerabilities. It provides centralized protection for applications hosted on Azure Front Door and Azure Application Gateway ensuring that malicious traffic is detected and blocked before reaching the application backend. Azure WAF leverages managed rulesets to actively protect web applications from threats and attacks. These rule sets are maintained by Azure, with the Default Ruleset (DRS) including rules from the Microsoft Threat Intelligence Collection, ensuring enhanced coverage, specific vulnerability patches, and improved false positive reduction. In addition to the managed rulesets, Azure WAF offers custom rules that enable you to create your own rules. With custom rules, you can set conditions based on attributes such as IP addresses, HTTP headers, and query strings to precisely control which traffic is allowed or blocked, providing flexibility and granularity. Within the custom rules, you can incorporate regex, which offers enhanced accuracy when matching patterns in your traffic. Regex (regular expressions) enable you to define complex conditions, allowing for highly specific filtering of incoming requests. Working with regex can sometimes be challenging due to its non-intuitive syntax. In this blog, we will demonstrate a practical, step-by-step approach for generating regex patterns using GitHub Copilot, refining them on Regex101, and validating their effectiveness in Azure WAF. This process helps ensure that your custom rules with regex work as intended, thereby enhancing your overall security effectiveness. GitHub Copilot GitHub Copilot is an AI-powered code completion tool developed by GitHub in collaboration with OpenAI. It assists developers by suggesting code snippets, functions, and even entire blocks of code as they type. By leveraging machine learning models trained on a vast amount of public code, GitHub Copilot can understand the context of the code being written and provide relevant suggestions, making the coding process faster and more efficient. Prompting GitHub Copilot can be particularly useful for security professionals. Enhanced code quality is one of the benefits, as GitHub Copilot can help security professionals write cleaner and more secure code by identifying potential vulnerabilities and suggesting best practices for secure coding, thus reducing the risk of introducing security flaws. Additionally, it offers time efficiency, as security professionals often need to write scripts or tools to automate security tasks, and GitHub Copilot can speed up this process by generating code snippets based on the prompts provided, allowing professionals to focus on more critical aspects of their work. GitHub Copilot can also assist in creating regex code, which is often complex and challenging to write. By providing accurate regex patterns based on prompts, it can help security professionals quickly develop effective text manipulation and pattern matching solutions. While AI-generated content can significantly streamline the process of creating regex patterns, it is important to verify the accuracy of these patterns to ensure they work as intended. Tools such as Regex101 provide a valuable platform for refining and validating regex patterns, helping to identify and correct any errors before implementation. What is Regex? Regular expressions, commonly known as regex, are sequences of characters that define search patterns. They are used in various programming languages and tools to match, locate, and manage text. Regex is incredibly powerful because it allows for complex text manipulation and pattern matching, making it an essential tool for tasks such as data validation, parsing, and transformation. However, regex can be difficult to learn due to its intricate syntax and the need for precise pattern construction. The learning curve is steep because even small mistakes can lead to unexpected results or errors. Additionally, regex patterns can become quite complex, requiring a deep understanding of both the syntax and the specific text being manipulated. Despite these challenges, regex is invaluable in the field of security. It can be used to detect and prevent malicious activities by identifying patterns associated with attacks, such as SQL injections or cross-site scripting. Regex can also help in filtering user inputs, ensuring that only safe and valid data is processed. Its ability to analyze large volumes of text swiftly and accurately makes it a powerful tool for security professionals aiming to protect systems and data from threats. Creating & Validating Regex Patterns Now that we have explored the capabilities of GitHub Copilot and the intricacies of regex, it is time to put this knowledge into practice. In the following steps, we will guide you through the process of leveraging GitHub Copilot to create and validate regex patterns effectively. By combining the power of AI-driven code suggestions with the precision of regex, you will be able to enhance your security tasks, automate complex text manipulations, and ensure robust data validation. Let us dive into the practical steps to get this done and see how these tools can streamline your workflow and bolster your security measures. To get started with GitHub Copilot, navigate here. We are going to leverage GitHub Copilot through Visual Studio Code (VSC). Once you have completed the requirements from the link above and you have enabled the extension, you can start prompting GitHub Copilot Chat. Prompt: Create two unique regex patterns. The first pattern should match any of the following words: "l33t", "h@x0r", "kewl", and "n00b", but the characters can be case insensitive. The second pattern should match the string of special characters "!@#$" followed by any integer that comes after the symbol "&". Response: Here are the two regex patterns: To match any of the words "l33t", "h@x0r", "kewl", and "n00b" in a case-insensitive manner: (?i)\b(l33t|h@x0r|kewl|n00b)\b To match the string of special characters "!@#$" followed by any integer that comes after the symbol "&": !@#\$&\d+ Now that GitHub Copilot has delivered 2 unique regex strings for use in our Azure WAF, let’s verify if the regex patterns work with the use of https://regex101.com. Regex101.com is an online platform designed to help users create, test, and debug Regex. It provides a user-friendly interface where you can input your regex patterns and evaluate them against sample text to see how they match. The site offers real-time feedback, highlighting matches and providing detailed explanations of each part of the regex pattern. This makes it an invaluable tool for both beginners learning regex and experienced users fine-tuning their patterns. Additionally, Regex101.com supports multiple regex flavors, including PCRE, JavaScript, and Python, allowing users to work with the syntax specific to their needs. Above, we see how we are validating the regex pattern at Regex101.com. I paste my regex pattern provided by GitHub Copilot at the top and then enter a JSON test body to match against the pattern. The tool verifies that the first regex pattern captures malicious attempts without case sensitivity and provides a detailed breakdown on the right side. This breakdown includes explanations of each part of the regex, helping to ensure that the pattern is correctly identifying the intended matches and highlighting any potential issues. In another example, we are using Regex101.com to validate a regex pattern aimed at identifying strings of unique characters. The tool verifies that the regex pattern successfully captures the string where each character appears only once and in order, followed by an integer. On the right side, Regex101.com provides a detailed breakdown of the regex pattern, explaining how each part contributes to the overall match. Now that we have validated the regex patterns with Regex101.com, let us implement them into Custom rules for Azure WAF. Using Regex with Azure WAF Having validated the regex patterns with Regex101.com, we can now proceed to implement these patterns into Custom rules for Azure WAF. This section provides a guide on integrating the validated regex patterns into your Azure WAF configuration to enhance web application security. By establishing these custom rules, you can tailor protection to meet specific requirements, ensuring malicious attempts are effectively intercepted and blocked. First, we will navigate to the Custom rules section of our Azure WAF policy, and author the two regex rules that we want to use to identify special patterns in request bodies going through our WAF. What is unique about using regex in Custom rules, is that you select Regex as an Operator in the Condition. From there, you will enter your regex pattern in the Match values section, select the action and the Custom rule is complete. After implementing the custom regex rules into Azure WAF, we executed a simulated malicious attempt to evaluate their effectiveness. The WAF, equipped with our regex patterns, successfully detected and intercepted the attack. The custom rules accurately identified the malicious activity and promptly blocked it, demonstrating the power and precision of using AI-generated regex patterns to enhance security measures. After executing the simulated malicious attempt, we examined the Azure WAF logs to confirm the effectiveness of our custom regex rules. The logs clearly indicated that the attack was intercepted, with the highlighted rule name and match value providing specific details about the block. This information is crucial for verifying that the custom rules are functioning as intended and accurately identifying malicious activities. By reviewing these logs, we can ensure that our security measures are robust and capable of protecting against potential threats. The detailed log entries not only confirm the success of our regex patterns but also offer insights into further refining and optimizing our security configurations. Conclusion Leveraging GitHub Copilot to generate regex patterns and validating them on Regex101.com before applying them to Azure WAF showcases the remarkable synergy between AI and security practices. By utilizing GitHub Copilot's intelligent code suggestions, we can efficiently create complex regex patterns tailored to our specific needs. Validating these patterns on Regex101.com ensures their accuracy and effectiveness in capturing malicious attempts. Once applied to Azure WAF, these regex patterns enhance our security measures, providing robust protection against potential threats. Testing and observing the impact of these AI-generated regex strings highlight the power and value of integrating AI into our security workflows. This approach not only streamlines the process but also demonstrates how AI can significantly contribute to hardening security, making it an efficient and worthwhile endeavor. References Introduction to Azure Web Application Firewall | Microsoft Learn What is Azure Web Application Firewall on Azure Application Gateway? | Microsoft Learn What is Azure Web Application Firewall on Azure Front Door? | Microsoft Learn Create and use v2 custom rules - Azure Web Application Firewall | Microsoft Learn GitHub CopilotAzure WAF’s Bot Manager 1.1 and JavaScript Challenge: Navigating the Bot Threat Terrain
Discover how Azure WAF’s Bot Manager 1.1 and the JavaScript Challenge elevate your bot protection strategy. In this blog, we explore how these advanced features in Azure Front Door and Application Gateway provide a robust defense against malicious bots while ensuring legitimate traffic flows smoothly. Learn how to fine-tune your web application’s security to safeguard against evolving bot threats without compromising on performance.
Get ClientIp in backend service (behind FrontDoor and APIM)
Context: Azure Front Door connects to APIM (external mode) that forwards the requests to services inside AKS. Services are monitored with Application Insights. The target is to be able to retrieve client IP in our backend service thanks to an Header. Situation: We have tested multiple ways to do so: - via the header X-Forwarded-For - via the header X-Azure-ClientIP - via a policy in APIM with the following field: "<value>@(context.Request.IpAddress)</value>" None of these methods works to retrieve the clientIP, we get Microsoft datacenters IPs instead. Do you have an idea to be able to get this clientIP in backend service? Thank you for any tips!
Issue with Azure VM Conditional Access for Office 365 and Dynamic Public IP Detection
Hi all, I have a VM in Azure where I need to allow an account with MFA to bypass the requirement on this specific server when using Office 365. I've tried to achieve this using Conditional Access by excluding locations, specifically the IP range of my Azure environment. Although I’ve disconnected any public IPs from this server, the Conditional Access policy still isn’t working as intended. The issue seems to be that it continues to detect a public IP, which changes frequently, making it impossible to exclude. What am I doing wrong?
