GNU/Linux

This bot has been hammering my websites, I have denied access in cpanel. GO AWAY CUILL

This accursed software mess has hit the same page on my web site over 9,000 times in 5 days. I've emailed CUILL several times with no response and have today faxed a strong letter of complaint to their office. I've also sent a copy to the registrars that manage their IP addresses in the vain hope that I can get this feral thing stopped. If I get no satisfaction I intend to invoice them $1 per hit..

Peter,

I like your idea for invoicing (or just suing) them for any hit after their robot/spider/crawler or other's automated indexer encounters its first 403, preferably with a redirect to a file containing a message confirming to the offending robot/spider/crawler it's trespass¹ on your computer.

If your ISP provides access to mod_rewrite (assuming you do not have direct server access) you might try something like this in .htaccess:
RewriteRule ^YourRudeRobotIsTresspassingMyCompter\.txt - [L] #Don't want to loop
RewriteCond %{HTTP_USER_AGENT} ^[\(\).]+[Tt][Ww][Ii][Cc][Ee][Ll][Ee][Rr][\-\(\).]+ [OR]²
RewriteCond %{HTTP_USER_AGENT} ^[Tt]wiceler
RewriteRule ^.* YourRudeRobotIsTresspassingMyCompter.txt [R=301,L]

Eventually we should automate this along the lines of the up and coming machine-readable privacy. Then browsers and bots can just access the Webmaster's policy and unlike robot.txt, automatically enforce it.

Of course this needs to be thoroughly discussed so the law of unintended consequences does not inadvertently add unwanted or unnecessary friction to Internet information flows!

We also want to balance stopping the rudeness with encouraging the incredible innovative minds, such as those behind Twiceler. Who knows maybe Twiceler can ensure its nextgen indexer incorporates such a machine readable and enforceable access policy? (Oppressive, repressive, and thug governments and some "mature" democracies everywhere are lining up for a centralized machine readable and enforceable access policy)

Hope it helps, stay cool, and think technically, I have reason to believe the folks at Cuill are well, cool people³. Some of Toocan’s postings, particularly in controversial areas have caused horrendous and persistent attacks, ostensibly to change the post (the posts never change as a result of such abuses). After you get the hang of it, it’s actually fun analyzing the attacks, if sometime irritating and always a waste of time and bandwidth.

-----notes-----

1. If you sue it's always good to take notice, particularly when using a novel approach—you could even include a counter++ so the message read "Twiceler has trespassed my computer 9,321 times in 17 seconds".

You could also just deny from [All Twiceler IPs]; the above rewrite is less rude since it just silently and politely returns a text message which I like to be a big white page of nothing. Also if browsers happen upon the text page, the modern ones will eventually just return a 304 and get the white page from their cache, saving bandwidth.

2. There are more elegant regexs but I always use one that helps me remember what I blocked, believe it or not you can forget even a Twiceler. Most virtual irritants, like their physical counterparts quickly go away when ignored.

3. It also helps to remember that some of these people are the kind of people that think society is somehow bettered because they move from California to New York to shave 30 nano-seconds off the time it takes the computer to execute a buy or sell order!

Hi!
It's been monthes that Twiceler breaks down our servers scannig our sites, and we have spent days trying to understand why we passed from 256kb/s to 72Mb/s of bandwitdh usage!
Our hoster invoices us 6,000 EUROS for the bandwidth usage, our clients says "No more hosting with you!", our business is compromised. And we are a new small company trying to start with lots of difficults.
Ok, new companies that wants to research on a new search engines are welcome, but why such a abnormal attack? Who is going to pay for the bandwidth usage?
We had to ban their IP adresses last night, our major client dosen't have anymore faith in our work.

Reading between the lines its quite obvious twicelers screwy design wasnt by accident. You have an Irish Techie who convinced VC's to dump 33 million in small change into his concept, which was based mainly on their massive index size. But guess what, when the VC auditors came to see the index size for real Mr. Tom had to fill it with something. So he came up with a spur of the moment idea to run the system dictionary agaisnt every sites web directory, and presto, Cuil now has generated millions if not billions of indexes to unique ip addresses, ofcourse they are all 40(1,2,3) errors, but who cares, since he was selling index size to his VC's not content. He passes the auditors test, they open up the bank account, and its muffins and chocolates forever (or at least until the money runs out and the VC's try to sell a goldmine full of fools gold.

This bot has been hitting my server like crazy.