BotScout helps prevent automated web scripts, known as "bots", from registering on forums, polluting databases, spreading spam, and abusing forms on web sites. We do this by tracking the names, IPs, and email addresses that bots use and logging them for future comparison. We also provide a simple API that you can use to test forms when they're submitted on your site.

This is the main source for information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.

Bots vs Browsers - Public Bots and User Agents Database and Commentary