Putting Sphider to the Test
I had an extra computer laying around and wanted to use it for something, so I decided to test out the search engine / indexing power of Shpider. I highly doubt I will ever use this for anything but it should be a fun experiment.
Tonight I am going to begin indexing digg.com using Sphider php search engine and see just how long it takes using a Pentium 3 512 MB box running minimal Debian Lenny. I set the indexer to parse all links so it has its work cut out for it. We shall see how long it takes before either a) the box croaks, b) I run out of disc space (28 GB) or c) a full index is generated.
By the way it is 1:13 AM on Thursday May 22, 2008
I will keep you updated on its progress.
:: Update ::
Was just playing with the search engine and searched the word “shpider” to see if my new article would appear - found a link to diggs terms of use that state that what I am doing is bad…
I am going to proceed seeing as this is not for monetary gain and I am pretty sure no one cares too much.
// what digg says about spiders…
With the exception of accessing RSS feeds, you will not use any robot, spider, scraper or other automated means to access the Site for any purpose without our express written permission. Additionally, you agree that you will not: (i) take any action that imposes, or may impose in our sole discretion an unreasonable or disproportionately large load on our infrastructure; (ii) interfere or attempt to interfere with the proper working of the Site or any activities conducted on the Site; or (iii) bypass any measures we may use to prevent or restrict access to the Site; //