InterviewSolution
Saved Bookmarks
InterviewSolution
| 1. |
What Is Robot.txt & Why We Use It? |
|
Answer» Web ROBOT are known as crawlers, spiders. And robot.txt USE to give instructions to these Web ROBOTS. Using this file a website owner can block robots to follow some LINK in website. Ex: User-agent: * Disallow: / Above code means website owner RESTRICTING all (“*”) to visit whole website(“/”). Web Robot are known as crawlers, spiders. And robot.txt use to give instructions to these Web robots. Using this file a website owner can block robots to follow some link in website. Ex: User-agent: * Disallow: / Above code means website owner restricting all (“*”) to visit whole website(“/”). |
|