Crawler Shield for OpenCart — protects the catalogue from AI crawlers, meta-externalagent and filter scrapers

Crawler Shield for OpenCart is a module that keeps bots from taking the shop down on filter pages. Every combination of filter, sorting, items per page and page number is a URL of its own, and each one is a heavy database query. A person opens a few. AI crawlers (meta-externalagent, GPTBot, ClaudeBot, Bytespider), SEO tools and price scrapers open all of them — and a shop on ordinary hosting hits the limits of its CPU and database.

The module recognises a bot by User-Agent and refuses it before the first catalogue query: in our test a refused filter page made 0 product queries, while the same page for a shopper made 43–45. Search engines and your ad previews go through, shoppers notice nothing. There is also a WooCommerce version.

What the shopper sees

Nothing new. In the test on all four versions a shopper in a real browser went through a category, a filter, sorting, six more filter pages, a product, the cart and guest checkout — without a single refusal or JavaScript error, and the order was saved with the right total. The module never touches checkout, account, payment callbacks, feeds or cron, and does not check POST requests at all.

The only limit that can touch a person is the Pro limit of filter pages from one IP (60 a minute by default). If many shoppers share one address (a mobile carrier, an office), raise it or add the address to “Never touch these IPs / networks”.

What the shop gets

• A catalogue that bots do not bring down. 300 meta-externalagent requests to filter pages (10 in parallel) were refused with a median of 12–25 ms, while a filter page for a shopper took 80–137 ms to build.
• Rules for each family. By default: search engines — let through; Meta ad previews — let through; AI crawlers, SEO tools and scrapers — refused on filter pages, other pages let through (limited in Pro); AI assistants on a person’s request (ChatGPT-User and others) — refused only on filters, products stay open.
• Less junk in the index. X-Robots-Tag: noindex, follow on filter, sorting and search pages — categories and products stay open.
• Meta ads keep working. facebookexternalhit is let through by default and only limited on filter pages in Pro.

What the admin gets

Settings are in Extensions → Extensions → Modules → Crawler Shield, four tabs: “Rules”, “Bot report”, “Check and files”, “Licence”. “Rules” has the mode, noindex, where the visitor’s IP comes from and a table of bot families with two rules each: for filter pages and for other pages. Unsure — choose “Watch only” for a day or two and look at the report.

noindex for filter pages is on by default. If you deliberately rank filter pages with parameters in the URL, untick “Tell search engines not to index filter pages” in the “General” block. Below are the filter parameters in the URL (you can add your own), routes where every page counts (search), the page number from which a list counts as a filter page for bots, and routes the module never touches.

“Check and files” shows what the module does with any visitor: pick a User-Agent from the examples or paste your own, add the page URL — and you see the decision, family, route and reason. The same tab has robots.txt lines worth adding for honest bots.

The “Bot report” shows requests by family, day and individual bot: how many in total, how many to filters, how many let through, served from the cache and refused.

Pro: limits, IPs and bots that pretend

The “Limit” rule gives a family a limit of requests a minute (20 on filter pages and 120 on other pages by default) — then 429 with Retry-After. The limit of filter pages from one IP and, optionally, from a /24 network catches a scraper calling itself a browser. IP and network lists — “always refuse” and “never touch”, with “Add Alibaba Cloud networks” (in September 2026 a scraper with a browser User-Agent walked the filters of our demo shop from there) and “Add my IP” buttons.

Googlebot, bingbot and Applebot are checked by reverse DNS once a day per address: a “Googlebot” from someone else’s network lands in “Fake search engines” and is refused. The filter page cache serves recognised bots a file instead of catalogue queries (a page lives 60 minutes — configurable; a bot may see a price that old), shoppers always get a live page. The journal of refusals shows top IPs, networks and pages for 24 hours with a “Block the network” button. The “Check and files” tab generates nginx and Apache rules and llms.txt.

How it works — step by step

1. A storefront request. The module runs on the event OpenCart fires before the page controller (in 2.3 — on the first language load of the page) — before catalogue queries.
2. GET and HEAD are checked, POST is not. Routes from “Never touch” (checkout, account, payments, feeds, cron) pass right away.
3. The bot family is identified by User-Agent; in Pro also by reverse DNS for search engines and by IP lists.
4. A filter page is identified by URL parameters, the search route and the page number for bots.
5. The family rule: let through, refuse (403) or limit (429 with Retry-After, Pro). A refusal is immediate, without building the page.
6. The decision goes into the statistics, in Pro also into the journal of refusals.

Under the hood

• One event, no modifications. OpenCart 4 and 3 — catalog/controller/*/before, 2.3 — catalog/language/*/before. In 2.3 pages whose controller queries the database before loading its language (some AJAX endpoints of filter modules) are not checked.
• A shopper costs zero queries — string checks only (in Pro the per-IP limit on filter pages is 2 queries). A bot costs one statistics update.
• Tables: per-minute counters, daily totals (60 days), the Pro journal (up to 5,000 rows, 14 days), DNS verdicts (one day).
• The bot cache lives in system/storage/cache/cc_crawler/, is filled only from 200 responses to recognised bots and is never served to shoppers.
• llms.txt on Apache. The stock OpenCart .htaccess refuses every .txt except robots.txt — the admin detects it and shows the one line to change.
• Uninstall removes the event, tables, settings (including the licence key — activate it again after reinstalling), our llms.txt and the cache.

How to install

1. OpenCart 4: Extensions → Installer → upload cc_crawler.ocmod.zip (keep the archive name) and click “Install”. OpenCart 3: Extensions → Installer → cc_crawler-oc3.ocmod.zip. OpenCart 2.3: upload the contents of the upload/ folder from cc_crawler-oc2.ocmod.zip to the shop root (FTP or the hosting file manager) — the 2.3 Installer does not copy files without FTP configured.
2. Extensions → Extensions → Modules → Crawler Shield → “Install”, then “Edit”.
3. Tip: first day — “Watch only” mode, then “Protect”. Behind Cloudflare — choose “Cloudflare header CF-Connecting-IP”.
4. For Pro: paste the licence key from the e-mail after purchase or get a trial on the “Licence” tab.