About CenvianBot

Identity and purpose

CenvianBot belongs to Cenvian. Version 1.0 collects limited HTTP, HTML, robots.txt, Sitemap, and public-resource evidence for a user-requested public or verified-owner website check.

Public and verified-owner scans

A public scan obeys robots.txt. CenvianBot prefers a CenvianBot group, otherwise uses the wildcard group, and treats an unreachable robots.txt or a server error as disallowing public page requests for that scan.

A verified-owner scan checks ownership again before it starts. For that exact verified hostname, it records robots.txt evidence but does not use Allow or Disallow rules to limit public HTTP resources. External hostnames and unverified subdomains remain under public-scan behavior.

What CenvianBot requests

• CenvianBot sends GET requests over HTTP or HTTPS on public ports 80 and 443. It does not send Cookie, Authorization, client certificates, request bodies, or user-supplied sensitive headers, and it does not submit forms. It does not log in, make purchases, publish content, or impersonate another crawler.
• It can request the website root, robots.txt, Sitemap files, sampled public HTML pages, public links needed for discovery, and a limited external URL when a current rule requires a status check.
• It follows at most 5 redirects and repeats protocol, hostname, DNS, IP-address, port, and TLS safety checks on every hop. A redirect does not expand recursive scanning to another hostname.
• The current V0.1 detailed sample analyzes at most 25 HTML pages and does not execute browser JavaScript or collect a rendered DOM.

Default request rate

At most 1 request starts per second for a hostname, with 1 concurrent HTTP request per hostname and 4 concurrent HTTP requests per worker. Tasks for the same hostname share the same rate bucket.

CenvianBot slows or stops after timeouts, connection resets, HTTP 429, or HTTP 503. It honors a Retry-After value only within the current 15-second wait window; otherwise it preserves available evidence and exits as partial or retryable instead of increasing concurrency.

Safety boundaries

Before every request and redirect, Cenvian resolves all A and AAAA results, rejects any non-public or prohibited address, connects through a validated public address, and keeps normal TLS certificate and hostname verification. It does not use CenvianBot to enter private networks or cloud metadata services.

Report an issue or request a stop

To report unexpected CenvianBot behavior or request that public scans stop for a hostname, email support@cenvian.com.

Include the hostname, approximate UTC time, and the relevant request details. Do not send passwords, session cookies, authorization values, verification tokens, or other secrets. No response-time commitment is promised.