How to Avoid Spambots

How to Avoid Being Harvested by Spambots

Identifying Spambots

Unfortunately, while munging is effective at hiding email addresses from most spambots today, it is likely spambots will adjust and defeat most munging techniques. The next step is to identify visitors that are likely to be spambots and either not display email addresses on your site to them, or completely deny their access to your pages. Two techniques are generally useful for telling real visitors from spambots: checking the referer string, and seeing if the visitor accepts cookies.

The referer string typically indicates where a visitor to your page linked from. Robots, including most spambots, visiting your website generally do not set the referer string, or they set it to some third-party's URL. One strategy to spot robots is to look for referer strings that come from outside your website. If, for example, someone arrives at your contact page from a remote site, you should consider limiting access to any email addresses that page may contain.

Similarly, robots typically do not handle cookies. While it would be possible for spambots to deal with cookies as they traverse the web, it would add substantially to their overhead and, in turn, increase the costs to spammers stealing addresses. Again, we suggest that if a visitor to your site does not accept cookies you consider hiding the addresses displayed or restricting access to your contact page.

In considering these two techniques, it's important to note that they both work best only under certain limited circumstances. For example, there are perfectly legitimate reasons to disable cookies, and you do not want to block all access to your website because a user has turned off their cookies. However, you may want to block access to your contact page, or alternatively, hide the email addresses displayed on that page if the user does not accept cookies or comes directly to the contact page from outside your site. We suggest, wherever possible, to also include a "contact us" form, in addition to any email addresses, so real users who are accidentally caught by the anti-spambot measures can still contact you.

Finally, Project Honey Pot is gathering a list of the IP addresses of known spambots. Over time, as the list becomes more robust, we will publish this list to assist webmasters in keeping spambots off their sites. For example, you can use the list of IP addresses to direct visitors to a page containing a CAPTCHA or other instructions only readable by a human being. We believe this will help keep the addresses on your site from being stolen while minimizing the risk of legitimate users being blocked.

On the next page we will discuss measures you can put in place to increase the legal risk to spammers who harvest addresses from your site. If these risks are raised high enough, spammers may program their spambots to avoid sites protected by these measures.