bots.htm: How to search the web, by fravia+ (¯`·.¸(¯`·.¸ Bots ¸.·´¯)¸.·´¯)

~ Bots lore ~

				Bots lore

Version April 2001

BOT WRITING, BOT TRAPPING & BOT WARS

This is a 'living' workshop on bots trapping and reversing, you will find elsewhere on my site other "broad" site protection (and de-protection :-) lore. As ~S~ deep writes in his bot-essay: "There are many Perl bots available on the net, but I'm fairly certain that you will not find one that does exactly what you want. There's also a "convention" amoung bot writers not to give bots source code to people who do not understand them - it's considered irresponsible. Of course, once you've learned how to build bots, you can be as irresponsible as you like". This is exact. Anyway we will find more knowledge only if we create it ourself at the same time. Your own contributes and work are necessary. The material presented here should be more than enough to "get you started". Write your own bots, publish the code so that others may improve them. Reverse the code of the existing bots. Awaiting your own contributions

Skip introduction and go directly to
[Iliad fetchbot] [Juno autoresponder] ["our" essays!] [our friends' essays!]

An introduction - explanation for those that do not know...

The term "bot" is, according to DeadelviS, a short for "robot", which sounds much cooler than "program"

As Andrew Leonard explains, like mechanical robots, bots are guided by algorithmic rules of behavior - if this happens, do that; if that happens, do this. But instead of clanking around a laboratory bumping into walls, software robots are executable programs that maneuver through cyberspace bouncing off communications protocols. Strings of code written by everyone from teenage chat-room lurkers to top-flight computer scientists, bots are variously designed to carry on conversations, act as human surrogates, or achieve specific tasks - such as seeking out and retrieving information. And... bots can also be used as weapons.

These pages of mine may regards all sort of web robots: spiders, wanderers, and worms. Cancelbots, Lazarus, Automoose. Chatterbots, softbots, userbots, taskbots, knowbots, mailbots. MrBot and MrsBot. Warbots, clonebots, floodbots, annoybots, hackbots, and Vladbots. Gaybots, gossipbots, gamebots. Skeleton bots, spybots, and sloth bots. Xbots, meta-bots. Eggdrop bots... as you can see the terminology is far from being simple... basically, though, the idea is to allow you to learn enough to WRITE your own search bots. Information searching through commercial engines is less and less efficient, as you'll soon realize it is of paramount importance, for a ~S~, to learn how to bulid his OWN 'home made and tailored" specific bots. You'll be amazed at the whealth of really strong signals you'll be able to find among the noise as soon as you use your own bots, even if they will be imperfect and surely wont be 100% state of the art...

It's up to you to help us with your own work or to keep what you'll dvelop for yourself: it is my intention to offer you enough material on this page to allow you to start. Choose the konwledge path or choose the dark path, it's up to you.
Since I'll NEVER charge money for accessing my site I would like to ask you the only "money" that's worth something on this web of ours: knowledge for all!

I want you to contribute with YOUR knowledge!
Remember, if you build on other people's shoulders, you must offer your own shoulders for others to build upon!

The powerful Iliad fetchbot

Hey! I wanna see a real bot in action before joining this section!

Yessir! Here is a good and powerful 'fetchbot', very useful for seekers and searchers alike. And if you knew nothing of this stuff you'll be fascinated (and even if you already knew... :-)
You'll now (at once if I were you) approach the "iliad" Searchbot (a very useful one, btw, was at [email protected], is now at [email protected]):
Send an email to:
[email protected]
write into the SUBJECT part of your email (into the subject field, duh!):
iliad query
write into the TEXT part of your email (that's your letter, duh!):
?Q: internet bots automated retrieval
(for instance... and you'll -most probably- get quite a lot of interesting material about bots from this mighty useful Searchbot... whatt'd'ya say?
If you're stuck email the same address with the word help both in Subject and in text (a pretty poor help will you get :-(

The powerful Juno autoresponder

Hey, this is great! I wanna taste another email-bot, just for fun!

Yessir! Please go ahead: have a look at friend autobot: Send an email to:
[email protected]
write into the SUBJECT part of your email (into the subject field, duh!):
send index
or if you want to have a laugh at some 'scarecrow' copyright propaganda, write - always in the SUBJECT field - the following:
send Copyrights

Hey, this is gorgeous! Now, before I start working on my own, let me please see and touch the code of a "real" bot!
Yessir! Please go ahead: enjoy the following essays!
All the code you may need to start working on your own!

Essays: how to build your own bots

PHASE ONE (16 July 1999)

this essay (perl_es1.htm): [email protected] ~ How to reverse a "free" service has been written by [blue] in July 1999 for the removing banners section, read and enjoy, let's hope you'll write afterwards your own perl-bots and send them here so that others can ameliorate and give feedback...

PHASE TWO (22 July 1999)

this essay (rt_bot1.htm):The HCUbot: a simple Web Retrieval Bot in Perl has been written by deep in July 1999, read and enjoy! Let's hope you'll write afterwards your own perl-bots and send them here so that others can ameliorate and give feedback...

PHASE THREE (14 September 1999)

this essay (botcgi.htm):Mirbot 1.0: a very special kind of a Robot has been written by The Mystical Friend in September 1999, read and enjoy! Let's hope you'll write afterwards your own perl-bots and send them here so that others can ameliorate and give feedback...

PHASE FOUR (14 September 1999)

this essay (rt_bot2.htm):The HCUbot (Version 2.0): a simple Web Retrieval Bot in Perl has been written by deep in July 1999 and updated and ameliorated in September 1999, read and enjoy! Let's hope you'll write afterwards your own perl-bots and send them here so that others can ameliorate and give feedback...

PHASE FIVE (21 September 1999)

this essay (sono_bot.htm):spider.r: a handy search tool and intro to REBOL has been written by sonofsamiam in September 1999, read and enjoy! Let's hope you'll write afterwards your own rebol-bots and send them over here so that others can ameliorate and give feedback...

PHASE SIX (March 2000)

[ftpbot1.htm]: A small ftp fetcher bot
by DarkWyrm
This bot searches a FTP site for a particular file (in Perl)

PHASE SEVEN (May 2000)

[plbtgrab.htm]: Source code for a spam bot (Kevin's spider) (in Perl)
by Kevin Jobson
Automatical link searching

PHASE EIGHT (September 2000)

[scan_reb.htm]: A simple REBOL scanner ways to retrieve hidden files, pages, zips, images
by -Sp!ke
Automatical link sniffing

PHASE NINE (October 2000)

Check [mysearch.zip]: ~ 20233 bytes A search bot in Visual Basic by Shoki (see [shokiwcd.htm])

PHASE TEN (February 2001)

Check [wf_add.htm]: Adding engines to WebFerret by Laurent (The guts of a search engines parser) Advanced

PHASE ELEVEN (April 2001)
[perlbot.htm]: HOW TO FOOL SSL DOWNLOAD OBSTACLES (spelunking into https "secure" servers)
by DigJim, Very Advanced essay

PHASE TWELVE (April 2001)

[lexi_wot.htm]: the lexibot essay (600 engines for next to nothing - part ONE - first steps) by WayOutThere, Advanced essay, part of the [bots], and of the [Essays] sections.
[lexi_lau.htm]: the lexibot essay (600 engines for next to nothing - part TWO - delving deeper) by Laurent, Advanced essay, part of the [bots], and of the [Essays] sections.

Friends' essays: how to build your own bots
(www::search modules in perl)

Advanced ~ For programmer-seekers only.
Added in July 2000

If you are really serious about advanced searching the following www::search modules in perl, which are constantly tuned to return results from the major search engines and news indexes, are a (basic) MUST READ. It is difficoult to underestimate how important this stuff is for any Seeker. Believe me, the time you'll invest reading their code will be WELL SPENT... oh yeah, actually very well spent. Pay special attention to some of the comments to the code :-)
Feedback with your own bots built on this stuff is mandatory.

[AdvancedWeb.pm.txt]
by Jim Smyser & USC/ISI, v 2.02 2000/04/04

[AlltheWeb.pm.txt]
by Jim Smyser v 1.4 2000/04/03

[AltaVista.pm.txt]
by John Heidemann v 1.6 2000/05/03

[Deja.pm.txt]
by Martin Thurn v 1.2 2000/02/24

[Dejanews.pm.txt]
by Martin Thurn v 1.25 2000/06/23

[Excite.pm.txt]
by Martin Thurn v 1.24 2000/06/19

[Google.pm.txt]
by Jim Smyser v 2.20 2000/06/08

[HotBot.pm.txt]
by Wm. L. Scheding and Martin Thurn v 1.58 2000/06/26

[Infoseek.pm.txt]
by Martin Thurn v 1.26 1999/12/10

[Lycos.pm.txt]
by Wm. L. Scheding and Martin Thurn v 1.18 2000/06/15

[Magellan.pm.txt]
by Martin Thurn v 1.19 2000/05/22

[NorthernLight.pm.txt]
by Jim Smyser, v 2.06 2000/06/16

[Opendirectory.pm.txt]
by Jim Smyser, v 1.4 2000/02/04

[Yahoo.pm.txt]
by Wm. L. Scheding and Martin Thurn, v 1.29 2000/05/10

Conclusion

Ehm, yes, you should learn [Perl] , by all means... but if you prefer to learn [PHP] instead, all these classes can be easily ported there, of course...

Err, this could be quite useful as well, come to think of it :-)
[X-Search.pl.txt]
by Jim Smyser, v 1.06 2000/06/14

Helping hands needed!
Enjoy! Just take your time, there is no hurry whatsoever, reverse some of the script above, understand their working, try some slightly different models... implement your own ideas... and finally write some (good) essays on this stuff yourself!