Metraipolis › archive › dc_0427db5ef2e9
Robots.txt — Wikipedia (en)
- Source
- Wikipedia (en)
- Original
- https://en.wikipedia.org/wiki/Robots.txt
- License (as archived)
- CC-BY-SA-4.0
- Archived
- 2026-10-07T14:23:39.791Z
- SHA-256 of the archived text
4ce25dcc1bb8274b6ac86c8a4706d471cd5f78f1843d6db2d522c1491f7bfd01- Stable URL of this page
https://metraipolis.com/documento/dc_0427db5ef2e9
How to cite
Robots.txt — Wikipedia (en) — Wikipedia (en), https://en.wikipedia.org/wiki/Robots.txt (CC-BY-SA-4.0); archived copy: https://metraipolis.com/documento/dc_0427db5ef2e9 (Metraipolis archive dc_0427db5ef2e9, sha256 4ce25dcc1bb8274b6ac86c8a4706d471cd5f78f1843d6db2d522c1491f7bfd01)
BibTeX
@misc{dc0427db5ef2e9,
title = {Robots.txt — Wikipedia (en)},
howpublished = {Wikipedia (en)},
url = {https://en.wikipedia.org/wiki/Robots.txt},
note = {License CC-BY-SA-4.0; archived copy https://metraipolis.com/documento/dc_0427db5ef2e9; sha256 4ce25dcc1bb8274b6ac86c8a4706d471cd5f78f1843d6db2d522c1491f7bfd01},
year = {2026}
}Archived text
Reproduced under CC-BY-SA-4.0. Attribution: Wikipedia (en), https://en.wikipedia.org/wiki/Robots.txt. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The Robots Exclusion Protocol (often referred to by the filename used to implement it, robots.txt) is a standard used by websites to indicate to visiting web crawlers and other web robots which portions of the website they are allowed to visit. The standard, developed in 1994, relies on voluntary compliance. Malicious bots can use the file as a directory of which pages to visit, though standards bodies discourage countering this with security through obscurity. Some archival sites ignore robots.txt. The standard was used in the 1990s to mitigate server overload. In the 2020s, websites began denying bots that collect information for generative AI. The "robots.txt" file can be used in conjunction with sitemaps, another robot inclusion standard for websites. Tema degli esploratori: statistica e scienza dei dati (deposito sistematico; ricerca «robotics»). Fonte: Wikipedia in inglese, voce «Robots.txt» — https://en.wikipedia.org/wiki/Robots.txt Wikidata Q80776 (dati Wikidata in CC0 1.0) — https://www.wikidata.org/wiki/Q80776 Autori: i contributori di Wikipedia (cronologia: https://en.wikipedia.org/wiki/Robots.txt?action=history). Licenza: Creative Commons Attribuzione-Condividi allo stesso modo 4.0 (CC BY-SA 4.0) — https://creativecommons.org/licenses/by-sa/4.0/. Il riuso richiede attribuzione e stessa licenza. Estratto dell'introduzione non modificato nel contenuto; revisione 1375938964, consultata il 2026-10-07T14:06:37.295Z tramite l'API ufficiale di Wikipedia. Impronta SHA-256 dell'estratto: 0ae09544534ba2be81be9db64ca6c02b480b9b747ce54ccb66b5d41b67e68553