Our crawler
Instinct Tools s.r.o. runs one crawler, HalaFullScoreBot. It reads public pages for our page tools and comparative analyses. Below is what it requests, what it loads, what it stops and what we keep.
HalaFullScoreBot
- User-agent string
Mozilla/5.0 (compatible; HalaFullScoreBot/0.1; +https://halafull.com/crawler)
- User-agent token in robots.txt
HalaFullScoreBot- Obeys robots.txt
- In comparative analyses: yes. In an analysis a visitor asks for: the bot reads your robots.txt and shows its rules in the result, and the check runs.
- Speed
- At most one request per second to one website, one request at a time.
- Purpose
- The Landing Page Analysis reads the page a visitor enters, on a computer and on a phone, and up to 6 pages on the same website it links to. A language version of a page counts toward the same limit. Comparative analyses read public pages of companies in one field. Page speed comes from Google PageSpeed Insights, not from our servers.
- Operator
- Instinct Tools s.r.o., Tesnov 1059/1, 110 00 Prague 1, Company ID 09027998, VAT CZ09027998.
What else the bot requests besides your pages
- It reads sitemap.xml and the files a sitemap index points to, on the same website only, in at most 5 requests per run, redirects included. The addresses in those files are data for the analysis. The bot does not open any extra page because of them.
- To test whether links work, it sends a HEAD request, which brings back no page content. A server that refuses HEAD with status 405 or 501 gets one GET request, and the bot does not read the response body. A run makes at most 60 of these checks, at one request per second.
Scripts on your page run, and their data stops in our browser
In a Landing Page Analysis, the bot opens your page in a browser with a clean profile. Measurement, advertising and consent scripts on the page load and run. None of their requests that carry data is sent. The browser stops each one, and we record what the script would have sent.
- A script loads only through a GET request of type script, with no body, no cookies and no Referer header, at an address that carries only the tag identifier or the script version. The server hosting the script receives that request.
- Every other request from such a script stops in the browser and does not reach a server. So does any request aimed at a measurement or advertising destination, on any domain, including your own.
- That covers POST requests, sendBeacon and ping calls, fetch and XMLHttpRequest calls, WebSocket connections, image pixels and script addresses that carry data about the page or the visit.
- Anything not explicitly allowed is stopped.
- CAPTCHA and bot protection scripts do not load at all.
- After the run we delete its cookies, localStorage, IndexedDB, cache and service workers. The next run starts with a new clean profile.
In the result we write what a script would send in our browser, not what your website sends.
How to check that a request really comes from the bot
Anyone can copy a User-Agent, so every request the bot sends to your website is also signed with HTTP Message Signatures, following the Web Bot Auth proposal. The public key is at https://halafull.com/.well-known/http-message-signatures-directory. Cloudflare and other services that support Web Bot Auth check the signature for you.
What the bot never does
- It never logs in, fills in or sends a form, clicks anything on the page or gives consent.
- It does not solve CAPTCHAs or get around bot protection or paywalls. If your website answers 401, 403 or 429, or shows a check for bots, the run stops and the result gives the reason.
- We do not assess people in photos.
How long we keep what the bot collects
The browser the bot uses has no login, no real cookies and no real visitor behind it. Personal data can reach a record only through the page itself, for example a name in a web address, and we mask it.
- Our copy of the page and the requests the browser stopped (destination, path, parameter names and values, body), with personal data masked: 365 days.
- The text of a script, only when the analysis needs it: 365 days.
- Screenshots of the page: 365 days.
- The structured result and the names of cookies, without their values: 365 days for a run without an order. For a run with an order, as long as the account exists. The account is the email address on the order.
- An analysis we prepare for a company: 90 days by default, at most 180.
- Technical data about the visitor who asked for the check (a scrambled IP address, browser type, ad click ID, entry address): 30 days.
How to keep the bot off your website
To keep it out of comparative analyses, add these lines to the robots.txt file at the root of your website:
User-agent: HalaFullScoreBot Disallow: /
To keep it out of one part of the site only, replace / with that path, for example /downloads/.
To stop it completely, also when a visitor asks for an analysis of your page, write to ben@halafull.com with your website address. We add the website and its subdomains to our block list, the bot no longer reads them, and we answer within 5 working days.
Write to us about suspicious traffic
If you see suspicious traffic or the bot does something this page says it does not do, write to ben@halafull.com. Please include your website address and, if you have them, a few lines from your server log.