Originally published April 23, 2008: Updated and Republished May 18, 2008; Updated and Republished June 27, 2008:
UPDATE 06/27/2008:Twiceler is still behaving, entering the site at reasonable intervals by reading robot.txt; crawling like a spider—…
more »