Crawler: (robot, spider, crawler) A program which performs a task more or less autonomously. Search engines use bots to find and add web pages to their search indexes. Spammers often use bots to “scrape” content for the purpose of plagiarizing it for exploitation by the Spammer.

 

Related Words

Crawler: A strike on which the ball misses the head pin. So called because the 4, 2, and 1 pins usually fall slowly, like dominos, after the rest of the pins are down.

Canonical Tag: Code used by search engine crawlers/spiders to tell search engines what URL is the original version of your webpage.

Yahoo Slurp: Yahoo web crawler. Instructions to Yahoo Slurp can be directed in the robots.txt file. Googlebot statistics can be viewed with web analytics software, on webserver logs, and in Google webmas . . . View Full Definition

Spider: The rear suspension element on the cone of a loudspeaker. All the moving parts of the speaker are suspended by the spider in the rear and the Surround in the front. The spider is so named be . . . View Full Definition

Invisible Web: Portions of the web which are not easily accessible to crawlers due to search technology limitations, copyright issues, or information architecture issues.

Robot: (robot, spider, crawler) A program which performs a task more or less autonomously. Search engines use bots to find and add web pages to their search indexes. Spammers often use bots to “scr . . . View Full Definition

Bot: An autonomous computer controlled character or opponent, used to mimic the effect of other human players participating. Bots are most often seen in first person shooters, and are often used . . . View Full Definition