RockethorseResearchBot

Crawler identification and contact page · rockethorseresearch.com

What this is

RockethorseResearchBot is an automated web crawler operated by the registrant of this domain. If you found this page from a User-Agent string in your server logs, you are in the right place: this page identifies the crawler, describes how it behaves, and tells you how to reach a person or exclude the crawler from your site.

Purpose

The crawler gathers publicly available information, primarily product and catalog pages, for private market and product research connected with prospective commercial projects. Retrieved pages are stored and analyzed internally. Page content is not republished, and access to the collected material is not sold or provided to third parties.

How it behaves

• It always identifies itself. Every request it makes carries the User-Agent string shown below, including requests for robots.txt. It never disguises itself as a browser or another crawler, and does not rotate identities or proxy addresses.
• It obeys robots.txt, including Crawl-delay, and including rules addressed to generic groups (a group such as User-agent: bot is treated as applying to it). Where a rule is ambiguous, it is interpreted in the site’s favor.
• It reads at a human scale. Requests to a given site are made one at a time, never in parallel, in short sessions of a few dozen pages with randomized delays of several seconds to minutes between requests (never faster than one request per five seconds), followed by cooldowns of several hours. A hard cap limits requests to any one site to a few hundred pages per day; typical days are far below that.
• It fetches public pages only. It does not log in to anything, does not bypass paywalls, does not solve or circumvent CAPTCHAs, and does not evade rate limiting or other technical access controls.
• It leaves when asked. An HTTP 403 or 429 response ends the session immediately, suspends crawling of that site for at least 24 hours, and alerts the crawler’s operator for review.
• It keeps records. A complete log of every request the crawler has ever made is retained, including the URL, timestamp, and outcome of each.

User-Agent string

RockethorseResearchBot (+https://rockethorseresearch.com/bot; contact@rockethorseresearch.com)

Excluding the crawler

To exclude it from your entire site, add this to your robots.txt:

User-agent: RockethorseResearchBot
Disallow: /

Path-specific Disallow rules and Crawl-delay are honored as well. Robots rules are rechecked regularly and a change is normally picked up within 24 hours; while a site’s rules are cached, the crawler will not knowingly act against a published change.

Contact and opt-out

contact@rockethorseresearch.com is a monitored inbox read by a person. Site operators may request exclusion by email instead of (or in addition to) robots.txt; such requests are honored and confirmed by reply, normally within ten business days.