We mentioned yesterday that WWW::SimpleRobot did not respect the robots.txt file. Well, there’s always WWW::RobotRules, which was easily dropped into place providing me a simple robot that followed the rules of a robots.txt file, if one exists.
My “outlinebot” is working quite well now, and I’m sure I’ll be tweaking it for the next few weeks or months… If I haven’t said this in a while, I really like Perl…