Research bot

RodrigoSilvanoResearchBot is a program that reads public pages of .pt websites for a study on how Portuguese sites are prepared for search engines and AI assistants. Results are published as aggregated figures on rodrigosilvano.pt.

1. How to identify it

Requests carry this User-Agent:

RodrigoSilvanoResearchBot/1.0 (+https://rodrigosilvano.pt/en/legal/research-bot)

2. What it reads

• The domain home page (one GET request).
• /robots.txt.
• /llms.txt, on domains that return a valid page.

It does not follow links to other pages, fill in forms, log in or run JavaScript.

3. What it stores

• Domain, HTTP response code and page language.
• Home page title and description.
• robots.txt rules addressed to AI agents (for example GPTBot, ClaudeBot, PerplexityBot and Google-Extended).
• Whether llms.txt exists and which structured data types (JSON-LD) are declared.
• The platform the site is built on, when the page code identifies it.

It does not store full page content, email addresses or personal data of visitors.

4. Rate and robots.txt compliance

The bot makes at most one request per second to each domain and reads each file once. If the domain robots.txt blocks the RodrigoSilvanoResearchBot token or all agents (User-agent: * with Disallow: /), the domain is excluded from the study and only the block is recorded.

5. How results are published

The study presents totals and percentages by sector. It does not identify sites with weak results. Only sites with favourable results are named, with the owner's permission.

6. How to exclude your domain

• Add to your robots.txt: User-agent: RodrigoSilvanoResearchBot followed by Disallow: /.
• Or write to info@rodrigosilvano.pt with the domain. Data for that domain is deleted and the domain is not read again.

7. Contact

Rodrigo Silvano, info@rodrigosilvano.pt.