How To Build a Web Spider On Linux

IdaAshley writes, "Web spiders are software agents that traverse the Internet gathering, filtering, and potentially aggregating information for a user. This article shows you how to build spiders and scrapers for Linux to crawl a Web site and gather information, stock data, in this case. Using common scripting languages and their collection of Web modules, you can easily develop Web spiders."

Hmm...

Yes, but does it run on ... damn.

Crawling efficiently

Their example of a web crawler uses a queue to hold links. Since a link may appear twice, they use a lookup to scan the queue to see if the link is already loaded, and discard it if so.

Better to use an associative array to cache the links since lookup is O(1). The Queue's lookup time is O(n) and if n gets large, so does the lookup time, not to mention that since you are checking each link the worst case scenario is a lookup time of O(n^2). A hash (associative array) will perform the same check in O(n). /([\W_\-]@\W+)/gs

The 90s called

They want their technology back.

What's the point?

Why would anyone have a need to write a simple spider nowadays? In 2006, there has to be a better way than just following links. For example, it would be interesting to see something that crawled the various social bookmarking sites and corelated the various terms. For example, User A on Delicious and User B on Stumble Upon both bookmark a link about Pink Floyd and another one about Led Zep. If I'm searching for something about Floyd, the system could recommend some cool info about Led Zep too. (Email me if you need to know where to send my royality checks).

Actually...

Some websites do not have good search functionality. Sometimes it's an area that Google doesn't crawl (robots.txt and such), and sometimes I'm looking for something very, very specific.

Regardless, I do, in fact, build spiders. For instance, in an MMO I play, all users can have webpages, so it's very useful to have a spider as part of a clan/guild/whatever to crawl the webpages looking for users who have illegal items and such. In a more general way, there is a third-party site which collects vital statistics of everyone who puts those in their user page, so you can get lists of the most powerful people in the game, the richest people, etc.

downloads

for those of us who don't have them, here are the basics:

Wget: http://www.gnu.org/software/wget/ [gnu.org].

Curl http://curl.haxx.se/ [curl.haxx.se]

Re:yes, I did RTFA

You RTFA? Are you sure you're in the right place?

Hardly linux-specific

All my love for linux aside, this has to do nothing with linux, the kernel (or even the GNU/Linux, the OS). It works just as well on any other unix-derivate or even windows.

some points

• Don't forget to check and respect robots.txt [robotstxt.org]. Python [python.org] has a module [python.org] that helps you parse that file
  
• Samie [sourceforge.net] and its Python port Pamie [sourceforge.net] are your friends. You can automate IE so your script is treated as an human and not discriminated as a robot.
  
• I use such beasts to do one-click time reporting at work and one-click cartoon collecting in my favorite newspaper.
  
• And once I even repeatedly voted on an online poll and changed the course of history.
  
• Ah, yes, TFA was about building a spider on Linux. I didn't check if my one-click IE scripts work on IE/Wine/Linux.
  
• If I write an one-click script for online shopping, does it infringe the infamous Amazon patent?
  
• When will Firefox's automation capabilities match those of IE?
  

Re:some points

"When will Firefox's automation capabilities match those of IE?"

It's always had it. Look up XUL some day. The entire browser is written in xul.

'Steve? Send the web spiders.'

Dammit, I was hoping this was article was about the evolution of Dr Weird's phone spiders, mechanical creatures that could be sent down your cable line to maul anyone sending you phishing emails and spam.

Oh sweet Jesus!

Pull the article out. The last thing we need is more indexing bots.

crawling is not so trivial

As the two students who started a little web search company, crawling the web is not trivial: http://infolab.stanford.edu/~backrub/google.html [stanford.edu]. An excerpt follows.

Running a web crawler is a challenging task. There are tricky performance and reliability issues and even more importantly, there are social issues. Crawling is the most fragile application since it involves interacting with hundreds of thousands of web servers and various name servers which are all beyond the control of the system.

In order to scale to hundreds of millions of web pages, Google has a fast distributed crawling system. A single URLserver serves lists of URLs to a number of crawlers (we typically ran about 3). Both the URLserver and the crawlers are implemented in Python. Each crawler keeps roughly 300 connections open at once. This is necessary to retrieve web pages at a fast enough pace. At peak speeds, the system can crawl over 100 web pages per second using four crawlers. This amounts to roughly 600K per second of data. A major performance stress is DNS lookup. Each crawler maintains a its own DNS cache so it does not need to do a DNS lookup before crawling each document. Each of the hundreds of connections can be in a number of different states: looking up DNS, connecting to host, sending request, and receiving response. These factors make the crawler a complex component of the system. It uses asynchronous IO to manage events, and a number of queues to move page fetches from state to state.

It turns out that running a crawler which connects to more than half a million servers, and generates tens of millions of log entries generates a fair amount of email and phone calls. Because of the vast number of people coming on line, there are always those who do not know what a crawler is, because this is the first one they have seen. Almost daily, we receive an email something like, "Wow, you looked at a lot of pages from my web site. How did you like it?" There are also some people who do not know about the robots exclusion protocol, and think their page should be protected from indexing by a statement like, "This page is copyrighted and should not be indexed", which needless to say is difficult for web crawlers to understand. Also, because of the huge amount of data involved, unexpected things will happen. For example, our system tried to crawl an online game. This resulted in lots of garbage messages in the middle of their game! It turns out this was an easy problem to fix. But this problem had not come up until we had downloaded tens of millions of pages. Because of the immense variation in web pages and servers, it is virtually impossible to test a crawler without running it on large part of the Internet. Invariably, there are hundreds of obscure problems which may only occur on one page out of the whole web and cause the crawler to crash, or worse, cause unpredictable or incorrect behavior. Systems which access large parts of the Internet need to be designed to be very robust and carefully tested. Since large complex systems such as crawlers will invariably cause problems, there needs to be significant resources devoted to reading the email and solving these problems as they come up.

Quality of article?

I've never programmed in Ruby, but I think the comment in Listing 1 says it all:
"Iterate through response hash"

Why would somebody want to do that?
A quick net search "reveals": A simple resp["server"] is all you need.
Maybe the article was meant to be posted on thedailywtf.com?

Re-inventing a square wheel

Basically, the article gives you ruby and python examples of how to get web pages, and (badly) parse them for information. The same thing everyone has been doing for at least a decade with Perl and the appropriate modules, or whatever other tools, except that most know how to do it correctly.

The first script is merely ridiculous: 12 lines of code (not counting empty and comment lines) to do:

HEAD slashdot.org | grep 'Server: '

But it gets worse. To extract a quote from a page, the second script suggests this:

stroffset = resp.body =~ /class="price">/
subset = resp.body.slice(stroffset+14, 10)
limit = subset.index('<')
print ARGV[0] + " current stock price " + subset[0..limit-1] +
" (from stockmoney.com)\n"

You don't need to know ruby to see what it does: it looks for the first occurence of 'class="price">' and just takes the 10 characters that follow. The author obviously never used that sort of thing for more than a couple of days, or he would know how quickly that will break and spit out rubbish.

Finally, there is a Python script. At first glance, it looks slightly better. It uses what appears to be the Python equivalent of HTML::Parse to get links. But a closer look reveals that, to find links, it just gets the first attribute of any a tag and uses that as the link. Never mind if the 1st attribute doesn't happen to be "href".

I suppose the only point of that article were the IBM links at the end:

Order the SEK for Linux, a two-DVD set containing the latest IBM trial software for Linux from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

And that is in a section for Linux developers on the IBM site? Maybe the did copy stuff from SCO after all?...

Re:Re-inventing a square wheel

what exactly is HEAD slashdot.org

It's a (perl) script which comes with libwww-perl [linpro.no] which either is now part of the standard Perl distribution, or is installed by default in any decent Linux distribution.

If you don't have HEAD, you can type a bit more and get the server with LWP::Simple's head() method (then you don't need grep):

$ perl -MLWP::Simple -e '$s=(head "http://slashdot.org/" )[4]; print $s'

Either way is better than those useless 12 lines of ruby (I'm sure ruby can also do the same in a similarly simple way, but that author just doesn't have a clue)

Okay kids...

Just so people who may come across this know, if you're going to do some HTML or XHTML parsing in Python, you'd be insane not to use BeautifulSoup [crummy.com] or a similar tool.

Example to find all links in a document:

from BeautifulSoup import BeautifulSoup
for tag in BeautifulSoup(html_document).findAll("a"):
  print tag["href"]

Yes, it's that simple. For an URL opener that also handles proxies, cookies, HTTP auth, SSL and so on, look into the urllib2 module that ships natively with Python.

It's a trap!

Ah, I can see it clearly now!

1. Post to Slashdot a decoy article(it includes Linux in the subjest) with new spam tricks
2. Watch if spam increases 30% next days
3. Bribe Cowboy Neal with 10G midget lesbian pr0n and get IP adresses of the art. readers
4. Load shotgun and make the world a better place!

Been there, done that

I guess most male CS students will have coded something similar at least once to D/L pr0n.

I did one in shell and one in TCL/TK.

User-Agent

They forgot the set the User-Agent header to IE.

A similar application

An app to find broken links on your web site.

Checking links with LinkCheck
http://world.std.com/~swmcd/steven/perl/pm/lc/link check.html [std.com]

Reinventing the wheel

I know, I know. Flame me. But I found Heritrix http://crawler.archive.org/ [archive.org] is a very polished package. Used it for my Masters research, and found that it is very extensible. Useful if you are doing real crawling, ie not concentrating on one site.

Incorrect Title

Should be: "How Not ..."

I don't think I am alone in my thinking

Nostarch press book

Nostarch press are releasing a book about this soon, they had a mockup on display at the Frankfurt book fair.

Nutch

Why not Nutch?

http://lucene.apache.org/nutch/ [apache.org]

Walk the dom directly

Once i had to collect a lot of info from a website. I used java and wget and some java html parser library (possibly JTidy). anyway the code was very short and clean. I'd recommend DOM walking to other solutions when the data isn't trivial.

screen-scraper

screen-scraper (http://www.screen-scraper.com/ [screen-scraper.com]) runs fabulously on Linux, and integrates well with most modern programming languages. It can save all kinds of time over writing Perl and Python scripts. There's a free (as in beer) version available, and a pro version if more features are wanted.

I did similar things in college...

I did similar things in college with Perl. (shudders*) The programs were OS-neutral; I think I developed mine in Windows under Cygwin.

*Yes, I know Slashdot is written in Perl.

Re:Just what the internet needs...

I think that's robots.txt, *not* spider.txt

Re:Just what the internet needs...

How does "spider.txt" get an Insightful when it's "robots.txt"? Sheesh, bump the Mods Roster.

Re:I hate

Just because you're paranoid doesn't mean people aren't crawling your site.

KFG