NebulaBot
If a request from NebulaBot appears in your server logs, it came
from us and this page explains it. We are
Nebula Design, a web studio in Bristol, and we
run a small research project measuring the state of UK charity websites in
aggregate — how many use HTTPS, which platforms the sector builds on, how email
security is configured, and similar sector-level questions. Findings are
published at nebula.design/research.
Its user agent string is:
Mozilla/5.0 (compatible; NebulaBot/1.0; +https://nebula.design/bot)
The short version: we read the pages any visitor can read,
slowly and politely, and publish only aggregate statistics — never a list of
sites, and never anything that identifies an individual organisation. If you
would rather we did not visit your site at all, one email or one line of
robots.txt ends it permanently.
What the crawler does
- Reads your
robots.txt first and honours it.
- Requests at most a handful of pages per site — typically the homepage and,
where linked from it, a donations page — never more than a few pages total.
- Identifies itself honestly in every request, with this page's address in
the user agent string.
- Waits several seconds between any two requests to the same site.
- Reads public DNS records (MX, SPF, DMARC, NS) — the same records every
mail server on the internet reads on every message.
What it never does
- No security scanning or probing of any kind — no login pages, no admin
paths, no vulnerability checks. This is enforced by a denylist in the
crawler's code, not just by policy.
- No personal data. We collect organisation-level facts only. Email
addresses, names of individuals, and anything similar are neither
collected nor stored.
- No forms are submitted, nothing is downloaded except the page HTML, and
no JavaScript is executed.
What we publish
Aggregate figures only — for example, "41% of crawled charity websites run
WordPress". Published statistics are checked before release so that no charity's
name or web address appears, and small groups are suppressed so no organisation
can be identified by elimination. The underlying observation data is not
published, sold or shared.
How to opt out
Either of these works, permanently:
- Email [email protected] with your
domain name — no explanation needed. We add it to a permanent exclusion
list honoured by every future run.
- Or add this to your
robots.txt:
User-agent: NebulaBot
Disallow: /
Data sources and licensing
The list of charity websites comes from the public registers of the
Charity Commission
for England and Wales, OSCR and the
Charity Commission for Northern
Ireland, used under the Open Government Licence v3.0. Register data is Crown
Copyright.
Questions? [email protected]