4.2 User Agent Filtering

📖 Overview The User Agent Filtering module allows you to control access to your website or application based on the User-Agent header included in HTTP requests. This header typically identifies the browser, tool, bot, or crawler making the request. By allowing or blocking specific User Agents, you gain powerful control over how your application interacts with browsers, bots, and potential attackers.

---

✅ Allowed User Agent List This section enables you to define trusted User Agents that should always be allowed access — even if other security mechanisms are in place.

Use Cases:

• Allowing access to legitimate bots:

  • Googlebot

  • Bingbot

  • Slackbot

• Permitting internal testing tools or scanners

Configuration Options:

• Enter a regex pattern to match the User-Agent string

• Select a sensitivity level from the dropdown Example: Allowing ^Googlebot.* ensures only Google’s official crawler is accepted.

---

🚫 Blocked User Agent List This section allows you to define unauthorized User Agents that should be completely blocked from accessing your system.

Use Cases:

• Blocking known scraping tools or headless browsers:

  • curl

  • python-requests

  • Scrapy

• Preventing spam bots or fake search engine crawlers

Configuration Options:

• Define a regex pattern for partial or full User-Agent strings

• Select a sensitivity level from the dropdown Example: Blocking .*bot.* will catch most generic bots and automated tools.

---

🔍 Regex Matching & Sensitivity Options When creating a rule, you must select a sensitivity level. The following four options are available:

1. Case Insensitive Matches plain text without considering letter case. Example: curl will match Curl, cURL, or CURL.

2. Case Sensitive Matches plain text exactly as typed, including letter case. Example: curl matches only curl and not CURL or Curl.

3. Regex Regular Expression Case Sensitive Allows full regex usage and respects letter casing. Example: ^SlackBot.* matches SlackBot/1.0, but not slackbot/1.0.

4. Regex Case Insensitive Enables full regex support while ignoring letter case. Example: ^slackbot.* matches SlackBot/1.0, SLACKBOT/2.0, etc.

---

⚙️ How to Add a Rule

1. Navigate to Security Rules > User Agent Filtering

2. Choose either the Allowed or Blocked tab

3. Click Add User Agent

4. In the popup:

   • Enter a regex pattern to match the User-Agent string

   • Choose a sensitivity level from the dropdown

5. Click Save — your rule is instantly applied

You can manage existing rules, search by regex, or delete them at any time.

---

🚨 Warning User-Agent headers can be easily spoofed. While User Agent Filtering is effective against basic bots and misconfigured clients, advanced attackers may forge trusted User-Agent strings.

For enhanced security:

• Combine this feature with IP Reputation databases

• Enforce rate limiting

• Use Web Application Firewall (WAF) rules

---

🎯 Conclusion User Agent Filtering is a lightweight, efficient, and flexible way to block malicious bots, reduce system noise, and protect public-facing endpoints — especially APIs and marketing pages.