Robots.txt: A file which sits in the root of a site and tells search engines which files not to crawl. Some search engines will still list your URLs as URL only listings even if you block them using a robots.txt file. Do not put files on a public server if you do not want search engines to index them!

 

Related Words

Robots.txt: A file which sits in the root of a site and tells search engines which files not to crawl. Some search engines will still list your URLs as URL only listings even if you block them using a r . . . View Full Definition

Yahoo Slurp: Yahoo web crawler. Instructions to Yahoo Slurp can be directed in the robots.txt file. Googlebot statistics can be viewed with web analytics software, on webserver logs, and in Google webmas . . . View Full Definition

Spider: The rear suspension element on the cone of a loudspeaker. All the moving parts of the speaker are suspended by the spider in the rear and the Surround in the front. The spider is so named be . . . View Full Definition