Talk:Robots.txt

This is the talk page for discussing improvements to the Robots.txt article.
This is not a forum for general discussion of the article's subject.

Put new text under old text. Click here to start a new topic.
New to Wikipedia? Welcome! Learn to edit; get help.

Article policies

Find sources: Google (books · news · scholar · free images · WP refs) · FENS · JSTOR · TWL

Archives: 1: 6 months

Computing Mid‑importance

	This article is within the scope of WikiProject Computing, a collaborative effort to improve the coverage of computers, computing, and information technology on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.ComputingWikipedia:WikiProject ComputingTemplate:WikiProject ComputingComputing
Mid	This article has been rated as Mid-importance on the project's importance scale.

Internet High‑importance

	Internet portal This article is within the scope of WikiProject Internet, a collaborative effort to improve the coverage of the Internet on Wikipedia. If you would like to participate, please visit the project page, where you can join the discussion and see a list of open tasks.InternetWikipedia:WikiProject InternetTemplate:WikiProject InternetInternet
High	This article has been rated as High-importance on the project's importance scale.

On 18 December 2022, it was proposed that this article be moved from Robots exclusion standard to robots.txt. The result of the discussion was moved.

Proposal to Include an Additional Resource on robots.txt

Hello,

I would like to propose the inclusion of a free tool as an additional resource on the robots.txt Wikipedia page. The tool, available at generaterobotstxt.com, helps users easily create and optimize their robots.txt files.

The purpose of this tool aligns with the educational and practical goals of the Wikipedia page. It allows users to:

Automatically generate valid robots.txt files tailored to their needs.
Avoid common errors that may inadvertently block or allow undesired web and AI crawlers.
Learn about the purpose and structure of this file through a user-friendly interface.

This tool is freely accessible and designed to benefit individuals with varying levels of technical expertise. It has received positive feedback for its simplicity and usefulness.

I believe including this resource would complement the information provided in the article and support users looking for practical guidance.

Looking forward to the community's thoughts on this proposal.

Thank you! 83.54.17.114 (talk) 21:54, 19 December 2024 (UTC)[reply]

I would say no because of the WP:ELNO policy. And also because it would be unfair to other tool developers. PhotographyEdits (talk) 19:48, 20 December 2024 (UTC)[reply]