A few days ago I came to recognize that email hiding technique I use won’t work anymore. However I still think it would be a great idea if I could remove a part of a page from search index, even if it won’t help me hide from spammers in the long run.

What I’d really like is to specify which parts of a page are not its content and can be safely ignored by search engines. Ignoring it would also mean absence of those parts from search indexes. Spiders could still follow links inside those parts, because I can still use robots.txt file if I really want to block their traversal. In effect, I’d like a more fine-grained approach to blocking, which doesn’t force the layout of my pages to even slightly favor spiders over people.

The way I imagine it would work is by simply applying a class name to the parts of the page I’d want shielded. I don’t care much (yet) if such class name would need to be specified in robots file or if it would be some name achieved by a web consensus. As far as I can see both approaches would work equally well.

The benefit of this would also be better search results. Every now and then I follow a search hit that fit my query because of a combination of site content and its navigation. Judging by my logs I’m not the only deceived soul. So, wouldn’t it be nice to be able to tell which parts are content and which aren’t?

I’m not sure how I missed the original announcement of Simple List Extensions (SLE) for feeds, but better late than never.

SLE is a very simple specification to extend feeds syntax to better support some common scenarios that weren’t covered with original specifications of RSS and Atom. Previously there was no way to tell feed readers/aggregators to display feed items in fixed order (e.g. feed with most popular stories of the day) or delete items not present in feed anymore (e.g. auction site feed with items matching a query).

SLE also adds support for grouping and filtering of feed items, which might not be very useful with you average 10-item feed, but will be, if feeds will indeed become a general mechanism for transporting data and not only a tool to highlight most recent bunch of updates. In any case, you can read a very short specification or visit latest announcement which includes more explanatory links to why and how they can and already are used.

By the way, I’m not sure if any reader apart from beta of IE7 already supports SLE, but I expect that most of them will soon. There’s little reason not to.

One of more popular javascript scripts I’ve written is a simple function that hides an email address from spiders by constructing a mailto link on page load. So far it has worked quite well.

Lately I started to have my doubts about this approach though. There was a series of articles a while ago about Google’s new Mozilla-based spider that I didn’t take too seriously at the time. However, even if not true then, it’s still only a matter of time before a spider like this will show up. The new found popularity of AJAX/Javascript simply guarantees that, since search engines can’t and won’t give up indexing content hidden behind fancy scripts.

Which means that my script will stop working in not so distant future as will all email obfuscating scripts out there. If it can be seen by a human, then it will be seen by a spider. So is there a way to publish my email address without it becoming public through search engines indexes, which is where most spammers seem to get our emails?

I could write a robots.txt file, which would exempt my contact page from being indexed at all. But this is a rather crude approach, since it means nothing on that page will get indexed. I could move email information to a separate page to let other contact data get indexed, but this is hardly any nicer.

What I’d really want is to be able to tell to search engines that only a part of my document is off limits. I don’t think there’s currently a way to do this, but if anyone has an idea how to do it, I’d really like to hear it.

Update: I think all efforts, including mine, to prevent spammers from collecting published addresses are ultimately doomed. The basic premise of all such approaches is to cloak an address in a way that spider can’t see it or can’t recognize it if it does. By basing spider on something like Mozilla, there won’t be any difference between what spider sees and what user does. There’s a similar problem with recognition. As spiders gets smarter, as they invariably do, you’ll be getting ever growing overlap between smartest spiders and stupidest users until it’s big enough to be unacceptable.

So we’ll either put up contact forms or I hope move to defending our inboxes with smart spam filters (those who haven’t yet).