=== AI Crawler Log ===
Contributors: catcodestudio
Tags: ai crawlers, gptbot, claudebot, robots.txt, indexnow
Requires at least: 6.2
Tested up to: 7.1
Requires PHP: 7.4
Stable tag: 1.0.0
License: GPLv2 or later
License URI: https://www.gnu.org/licenses/gpl-2.0.html

See which AI crawlers visit your site, what they read and what they get. Open a page as a bot, allow or refuse each one, send IndexNow.

== Description ==

GPTBot, ClaudeBot, PerplexityBot and others read your site every day. Your analytics does not show them, because they run no scripts. This plugin writes them down, on any WordPress site, with or without a shop.

= The journal =

* Every visit of a known AI crawler: time, bot, address, the status your server really answered, network address.
* Who visits: visits in 24 hours, 7 days and 30 days per bot, with the owner and what the bot is for: model training, AI search, or a live fetch because a person asked.
* Most read pages, and the errors bots are served, so you see the 404s and 500s they run into.
* Ordinary visitors are not recorded and pay for one regular expression, nothing else.
* During a crawl wave the counters stay exact, while the journal thins out to every 20th visit above 300 an hour, so the load does not move into your database.
* Retention in days, an option not to keep network addresses, one button to delete everything.

= See as a bot =

Open any page of your site the way a crawler gets it: with its name, without cookies, without JavaScript. The plugin compares the answer with what a browser gets and tells you when:

* the bot gets an error, or a different answer than a browser, for example from a firewall or security plugin;
* robots.txt forbids the page for this bot, and which line does it, whoever wrote that line;
* the page asks not to be used (noindex, noai, X-Robots-Tag);
* hardly any text arrives before scripts run, so a crawler that runs no JavaScript sees next to nothing;
* the answer came from a cache.

You see the title, description, canonical address, structured data blocks, and the text itself.

= Allow or refuse =

* A rule for each bot: Allowed, Asked to stay away (robots.txt), or Refused (robots.txt and a 403).
* One click to refuse training and undocumented crawlers while AI search and live fetches stay allowed.
* Robots-only words such as Google-Extended and Applebot-Extended are covered and explained: they control how content is used, and never appear as visitors.
* The screen shows what the robots.txt your site really serves says about each bot, and warns when it disagrees with your rule, for example when a real robots.txt file is in the way.
* A bot told to stay away that keeps coming is shown as such, three days after the rule, so that it had time to read it.
* Ready lines for .htaccess and nginx if you want to refuse before WordPress and before the cache.
* Nothing is refused until you say so. robots.txt itself always stays readable.

= IndexNow =

Off by default. Switch it on and the addresses of published, changed, moved and removed pages go to IndexNow in the background. Password-protected pages, pages marked noindex in Yoast SEO, Rank Math, SEOPress or The SEO Framework, sites hidden from search engines and test copies stay out. The key file is created and checked for you, and the latest sends are listed. If CatCode Instant Index, the IndexNow plugin from Microsoft Bing or Rank Math's Instant Indexing is active, this plugin stays quiet to avoid double sending.

Sending an address only says that a page changed. The plugin does not promise that any search engine or AI service will visit, use or show your pages.

// @pro-start
= Pro =

* **Real or fake.** A bot name is free for anyone to use. Pro compares each visit with the address lists published by OpenAI, Anthropic, Perplexity, Amazon, DuckDuckGo, Mistral AI, Common Crawl and Google for its agent. Bots whose owner publishes no list stay "Not checked" rather than guessed at.
* **Fakes refused**, on request: a request with the name of a bot from someone else's address gets a 403.
* **Server log import.** Pages answered by a cache never reach WordPress. Upload an Apache or nginx access log, or use WP-CLI, and those visits join the journal without doubles.
* **Mail** when a new bot appears, and a summary on Mondays.
* **CSV export** of the journal with the filters of the screen.

7-day trial on request. A licence covers 1 to 5 years of updates and support, and a key works on two sites: the main one and a test copy.
// @pro-end

OpenAI, GPTBot, Anthropic, Claude, Google, Apple, Meta, Amazon, Perplexity, DuckDuckGo, Mistral AI, ByteDance, Cohere and other names are trademarks of their owners. This plugin is independent and is not affiliated with or endorsed by them.

== Installation ==

1. Install and activate the plugin.
2. Open Tools → AI Crawler Log. Recording starts at once and nothing is refused.
3. Open See as a bot to check a page today. Crawlers come when they come, and a small site can wait days for the first visit.
4. Set your rules on the Access tab when you know who visits.

== Frequently Asked Questions ==

= Do I need WooCommerce? =

No. The plugin works on any WordPress site.

= Does this make my site appear in AI answers? =

The plugin shows and controls access. Whether an AI service reads, uses or quotes a page is its owner's decision, and no plugin can promise that.

= Does this add llms.txt? =

No. This plugin looks at what crawlers really do.

= Why are Googlebot and Bingbot not on the list? =

They are search engines first, and refusing them takes the site out of search. The rules here are for bots you can refuse without that. To opt out of Gemini training without touching Search, use Google-Extended on the Access tab.

= My page cache is on and the journal is thin. Why? =

A page answered by a cache never reaches WordPress, so no plugin running inside WordPress sees that visit or can refuse it. The plugin tells you when it finds a cache. Exclude the crawlers from the cache, or add the server rules from the Access tab.

// @pro-start
Pro can also read those visits from the server log.
// @pro-end

= Does robots.txt stop a bot? =

It is a request. Documented crawlers say they follow it, others do not, and live fetches made for a person may not apply it. Refused adds a 403 for bots that reach WordPress.

= Can I trust the bot name in the journal? =

It is the name the visitor gives, and anyone can use it. Treat it as a claim.

= Does See as a bot show exactly what the real bot gets? =

Nearly. The check runs from your own server. A firewall or CDN that goes by the network address rather than the name can answer the real bot differently.

= What is stored? =

Time, bot name, address, status and, unless switched off, the network address of requests that name a known AI crawler. It stays in your database. The suggested text for your privacy policy is in Settings → Privacy.

= Does it work on multisite? =

Each site has its own journal and its own rules. robots.txt rules work for a site at the root of its domain, where crawlers look for that file.

== External services ==

The journal, the rules and See as a bot talk only to your own site.

**IndexNow.** Only when you switch it on. After a page is published, changed or removed, the plugin sends the host name, the page addresses and your site's IndexNow key to https://api.indexnow.org/indexnow. The service then opens the key file on your site. Terms: https://www.indexnow.org/terms Documentation: https://www.indexnow.org/documentation

// @pro-start
**Address lists.** With an active licence, once a day the plugin downloads the lists published at openai.com, claude.com, perplexity.ai, developer.amazon.com, duckduckgo.com, mistral.ai, index.commoncrawl.org and developers.google.com. Only a normal download: your site address appears in the User-Agent of the request.

**catcode.com.ua.** The licence key, your site address and, for a trial, your e-mail go to catcode.com.ua to activate the licence and to check for updates. Terms: https://catcode.com.ua/terms/
// @pro-end

== Changelog ==

= 1.0.0 =
* First release.
