SmallFishBot

It reads a small number of pages of a business website and records facts about what the business offers — whether there is a way to book online, request a quote, or start a chat. People searching for suppliers use those facts to find businesses that match what they need.

It stores extracted facts, not copies of your pages. No page is republished, mirrored or cached for readers.

• robots.txt is fetched first and honoured. A disallow for SmallFishBot/0.1 or * stops the fetch.
• At most a homepage plus three linked pages per business, chosen by what the search is actually asking about.
• One request at a time per host, with at least 1.5 seconds between them.
• Retry-After is honoured on 429 and 503, and a host that refuses gets exponentially longer gaps before any further request.
• No login is attempted, no form is submitted, no paywall or block is worked around. A site that declines to be read stays unread, and says so to our users as “this site blocks automated reading”.

Add this to your robots.txt and it will stop on the next visit:

User-agent: SmallFishBot/0.1
Disallow: /

We do not disguise ourselves as a browser to get around this. That was tested and measured: it would make one site in 78 readable, which is not worth being the kind of crawler that does it.

Email [email protected] from an address at the business’s own domain, or from the phone number listed on its site, and it will be suppressed from results.

Being honest about the limits. A self-serve opt-out form with ownership verification is planned and not yet built, so removal is a manual request today. And removal cannot reach rows a user has already exported — nothing can. What it does do is stop the business appearing in any future result or export.

Business names, addresses and phone numbers come from Overture Maps, an open dataset. Everything else is read from the business’s own public website.