Our crawlers

Who they are

PropertyExtra runs two web crawlers, which identify themselves in the user-agent header with a link to this page:

PropertyExtraBot checks property pages on estate agents’ own websites, so that the listings we show stay accurate — for example, whether a property is still available and at what price.

PropertyExtraResearchBot reads estate agents’ public pages, such as their home, contact and branch pages, to learn how an agency is organised: its branches, office contact details, and the property software its website runs on.

How they behave

Both obey robots.txt, including any crawl delay you set. They make one request at a time to a site, with a pause between requests, and they do not attempt to get around blocking, logins or paywalls.

We keep office contact details, not personal ones: we discard email addresses that appear to belong to a named individual, and our crawlers do not visit team or staff pages. The privacy policy (section 11) explains how we use information about estate agents and your rights over it.

Opting out and removal

To stop either crawler, add User-agent: PropertyExtraBot or User-agent: PropertyExtraResearchBot followed by Disallow: / to your robots.txt, or email hello+privacy@propertyextra.ai with your domain. We act on either the same day. To have information we already hold about your agency removed, email the same address.