Robots exclusion protocol know as simply robots.txt.
We use robots.txt in websites to communicate with web crawlers and other web robots.
we manipulate crawlers boat through this files. it the main purpose of this files.
Few important things for robots.txt file.
-> Robots can ignore your robots.txt. Especially malware robots that scan the web for security vulnerabilities.
-> Anyone can see robots.txt file of server and it’s publicly available file. So don’t hide information.
-> we need to put this files on our root repository.
An example of robots.txt file content.
The “User-agent: *” means this section applies to all robots.
The “Disallow: /” tells the robot that it should not visit any pages on the site.