Slashdot Log In
How To Build a Web Spider On Linux
Posted by
kdawson
on Wed Nov 15, 2006 03:13 AM
from the five-eyes dept.
from the five-eyes dept.
IdaAshley writes, "Web spiders are software agents that traverse the Internet gathering, filtering, and potentially aggregating information for a user. This article shows you how to build spiders and scrapers for Linux to crawl a Web site and gather information, stock data, in this case. Using common scripting languages and their collection of Web modules, you can easily develop Web spiders."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Hmm... (Score:5, Funny)
Re: (Score:3, Interesting)
The PHP interpreter is over 5 megabytes in size. And it isn't thread-safe. That's a lot of memory overhead for a program that's going to be blocking on I/O most of the time, seeing how you'll have to fork() a new process for each new "thread" you want.
Also, languages like Perl and Python have binaries that are about 1 megabyte in size. Now, while they'll probably need to load in extra files for most practical applications, these extra files are typically small. Most importantly, Per
Crawling efficiently (Score:5, Informative)
Better to use an associative array to cache the links since lookup is O(1). The Queue's lookup time is O(n) and if n gets large, so does the lookup time, not to mention that since you are checking each link the worst case scenario is a lookup time of O(n^2). A hash (associative array) will perform the same check in O(n).
Re: (Score:3, Interesting)
The 90s called (Score:5, Funny)
What's the point? (Score:2)
Actually... (Score:4, Interesting)
Regardless, I do, in fact, build spiders. For instance, in an MMO I play, all users can have webpages, so it's very useful to have a spider as part of a clan/guild/whatever to crawl the webpages looking for users who have illegal items and such. In a more general way, there is a third-party site which collects vital statistics of everyone who puts those in their user page, so you can get lists of the most powerful people in the game, the richest people, etc.
Parent
downloads (Score:5, Informative)
for those of us who don't have them, here are the basics:
Wget: http://www.gnu.org/software/wget/ [gnu.org].
Curl http://curl.haxx.se/ [curl.haxx.se]
yes, I did RTFA (Score:2)
Re:yes, I did RTFA (Score:4, Funny)
Parent
Hardly linux-specific (Score:5, Insightful)
some points (Score:5, Interesting)
Re: (Score:2)
Firefox's automation capabilities don't need to match those of IE, for pretty much the same reason. The only thing Mechanize can't do is JavaScript, and there are vague plans about that.
Re: (Score:2)
Re:some points (Score:4, Informative)
It's always had it. Look up XUL some day. The entire browser is written in xul.
Parent
'Steve? Send the web spiders.' (Score:2)
Oh sweet Jesus! (Score:3, Insightful)
crawling is not so trivial (Score:2, Interesting)
That reminds me. (Score:3, Informative)
Unfortunately, many web developers still ignore the inevitable, leaving their sites vulnerable to the dreaded Googlebot "attack". While most of the spider developer manuals (TFA included) stress the importance of being polite (respect robots.txt & friend
Quality of article? (Score:2, Insightful)
"Iterate through response hash"
Why would somebody want to do that?
A quick net search "reveals": A simple resp["server"] is all you need.
Maybe the article was meant to be posted on thedailywtf.com?
Re-inventing a square wheel (Score:5, Insightful)
Basically, the article gives you ruby and python examples of how to get web pages, and (badly) parse them for information. The same thing everyone has been doing for at least a decade with Perl and the appropriate modules, or whatever other tools, except that most know how to do it correctly.
The first script is merely ridiculous: 12 lines of code (not counting empty and comment lines) to do:
HEAD slashdot.org | grep 'Server: '
But it gets worse. To extract a quote from a page, the second script suggests this:
You don't need to know ruby to see what it does: it looks for the first occurence of 'class="price">' and just takes the 10 characters that follow. The author obviously never used that sort of thing for more than a couple of days, or he would know how quickly that will break and spit out rubbish.
Finally, there is a Python script. At first glance, it looks slightly better. It uses what appears to be the Python equivalent of HTML::Parse to get links. But a closer look reveals that, to find links, it just gets the first attribute of any a tag and uses that as the link. Never mind if the 1st attribute doesn't happen to be "href".
I suppose the only point of that article were the IBM links at the end:
And that is in a section for Linux developers on the IBM site? Maybe the did copy stuff from SCO after all?...
Okay kids... (Score:5, Informative)
Example to find all links in a document:Yes, it's that simple. For an URL opener that also handles proxies, cookies, HTTP auth, SSL and so on, look into the urllib2 module that ships natively with Python.
Parent
Re:Re-inventing a square wheel (Score:5, Insightful)
It's a (perl) script which comes with libwww-perl [linpro.no] which either is now part of the standard Perl distribution, or is installed by default in any decent Linux distribution.
If you don't have HEAD, you can type a bit more and get the server with LWP::Simple's head() method (then you don't need grep):
$ perl -MLWP::Simple -e '$s=(head "http://slashdot.org/" )[4]; print $s'
Either way is better than those useless 12 lines of ruby (I'm sure ruby can also do the same in a similarly simple way, but that author just doesn't have a clue)
Parent
Re: (Score:3, Informative)
Re: (Score:2)