# 4.2 User Agent Filtering
π **Overview**\
The **User Agent Filtering** module allows you to control access to your website or application based on the `User-Agent` header included in HTTP requests. This header typically identifies the browser, tool, bot, or crawler making the request.\
By allowing or blocking specific User Agents, you gain powerful control over how your application interacts with browsers, bots, and potential attackers.
***
β **Allowed User Agent List**\
This section enables you to define trusted User Agents that should always be allowed access β even if other security mechanisms are in place.
**Use Cases:**
* Allowing access to legitimate bots:
* `Googlebot`
* `Bingbot`
* `Slackbot`
* Permitting internal testing tools or scanners
**Configuration Options:**
* Enter a regex pattern to match the User-Agent string
* Select a sensitivity level from the dropdown\
**Example:** Allowing `^Googlebot.*` ensures only Googleβs official crawler is accepted.
***
π« **Blocked User Agent List**\
This section allows you to define unauthorized User Agents that should be completely blocked from accessing your system.
**Use Cases:**
* Blocking known scraping tools or headless browsers:
* `curl`
* `python-requests`
* `Scrapy`
* Preventing spam bots or fake search engine crawlers
**Configuration Options:**
* Define a regex pattern for partial or full User-Agent strings
* Select a sensitivity level from the dropdown\
**Example:** Blocking `.*bot.*` will catch most generic bots and automated tools.
***
π **Regex Matching & Sensitivity Options**\
When creating a rule, you must select a sensitivity level. The following four options are available:
1. **Case Insensitive**\
Matches plain text without considering letter case.\
Example: `curl` will match `Curl`, `cURL`, or `CURL`.
2. **Case Sensitive**\
Matches plain text exactly as typed, including letter case.\
Example: `curl` matches only `curl` and not `CURL` or `Curl`.
3. **Regex Regular Expression Case Sensitive**\
Allows full regex usage and respects letter casing.\
Example: `^SlackBot.*` matches `SlackBot/1.0`, but not `slackbot/1.0`.
4. **Regex Case Insensitive**\
Enables full regex support while ignoring letter case.\
Example: `^slackbot.*` matches `SlackBot/1.0`, `SLACKBOT/2.0`, etc.
***
βοΈ **How to Add a Rule**
1. Navigate to **Security Rules > User Agent Filtering**
2. Choose either the **Allowed** or **Blocked** tab
3. Click **Add User Agent**
4. In the popup:
* Enter a regex pattern to match the User-Agent string
* Choose a sensitivity level from the dropdown
5. Click **Save** β your rule is instantly applied
You can manage existing rules, search by regex, or delete them at any time.
***
π¨ **Warning**\
User-Agent headers can be **easily spoofed**. While User Agent Filtering is effective against basic bots and misconfigured clients, **advanced attackers may forge trusted User-Agent strings**.
For enhanced security:
* Combine this feature with IP Reputation databases
* Enforce rate limiting
* Use Web Application Firewall (WAF) rules
***
π― **Conclusion**\
User Agent Filtering is a lightweight, efficient, and flexible way to block malicious bots, reduce system noise, and protect public-facing endpoints β especially APIs and marketing pages.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.shieldsguard.com/getting-started/4.-security-rules/4.2-user-agent-filtering.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.