Today, we are launching the general availability of Bot Manager1.1 ruleset in Azure WAF integrated with Azure Front Door.
Bot Manager1.1 extends all the rules in the existing Bot Manager1.0 ruleset and adds multiple new rules to provide comprehensive bot management capabilities to web applications. The new capabilities introduced in this ruleset include new Goodbots rules and a new Badbots rule.
The main value prop of the new ruleset is to reduce false positives in good bot detections and increase true positives in malicious bot detections.
Benefits of the new rules in the Goodbots rule group:
- Improving SEO rankings due to good bots crawling websites and reducing FP (false positive) seen by customers.
Customer websites are crawled by good bots which results in increased SEO (search engine optimization) rankings. With Bot Manager 1.1 ruleset, a comprehensive set of rules are added to the Goodbots rule group which allows a larger set of legitimate published bots. Examples of such Goodbots include Googlebot, Bingbot etc.
As a real-life scenario, we encountered an issue with the Bot Manager1.0 ruleset where certain Goodbots were absent, leading to blocked requests to web applications. For example, a valid Google crawler bot was getting blocked by the Bot Manager1.0 100200 rule, which resulted in lower SEO rankings for the customer and eventually disappearing from the SEO rankings. As a workaround, the customer disabled rule 100200 which brought their SEO rankings up but resulted in lowered protection from true malicious bots that have falsified their identities. Prior to implementing the Bot Manager1.1 ruleset, the only other alternative to allow legitimate crawlers was to add custom rules to allowlist their IP addresses. However, this approach posed challenges due to the dynamic nature of crawler IPs, which change frequently.
With the new updates to Bot Manager1.1, a comprehensive list of good bot IPs is added to the existing rule 200100 which results in lower false positive detections by the Bot Manager ruleset. The 200100 rule from Bot Manager1.0 ruleset is now revamped to only include good bots in the search engine crawler category.
- Bringing clarity to the Goodbots rule group
With Bot Manager 1.1 ruleset, many new verified good bot rules have been added that target different categories of good bots. These new rules include the link checker, social media, content fetchers, feed fetcher and advertising bots. Additional bots that don’t fit into any particular category are added to 200200 as verified miscellaneous bots. This empowers customers to have granular control over their WAF policy. For example, if a customer does not wish to have social media bots crawling their sites, they can achieve this by changing the action associated with the social media rule.
Benefits of the new rule in the Badbots rule group:
Today customers see malicious bots perpetuating many malicious attacks. Examples includes:
- Scraping websites and spreading dis-information, executing targeted phishing attacks and social engineering attacks.
- Spamming customer websites with form submission pages.
- Manipulating rankings of content tooling websites’ analytics pages.
- Launching denial-of-inventory attacks.
- and many others.
The new Bot Manager1.1 ruleset incorporates a novel rule, Bot100300, complemented by the existing rules in the Badbots rule group rules, effectively mitigates malicious bot attacks.
Let’s take a closer look at the Bot Manager1.1 ruleset:
Goodbots rule group
The following screenshot describes the new good bot rules added to the new ruleset
Read the full post here: General availability of Azure WAF Bot Manager1.1 Ruleset