The Bucket and the Firehose
We PubSub staffers are often asked what it is, exactly, that sets our service apart from search engines like Google. It’s an essential difference – what we do is something that nobody else is doing, or at least doing at the scale we are — and it’s worth clarifying here.
Conventional (“Retrospective”) Search: the “Bucket”
Search engines have been around for a while now, and a lot of effort has gone into refining their design. The best ones now work very well indeed. They all do pretty much the same thing:
Crawling the Web
The World Wide Web is enormous; nobody knows just how enormous, but the “surface” of the Web – documents that are directly linked to – comprises, conservatively, at least ten billion pages, and it is growing very, very rapidly. Finding relevant documents in this ocean of information would be a hopeless task without search engines.
Search engines do a simple but labor-intensive job. They visit a Web page, make note of what is there, and then follow the links on the page, wherever they might lead. Everywhere they go, they record information about the current page before moving on. All of this is stored in a vast, indexed data repository (let’s call it “the bucket”). When a user comes to the search engine’s website, query in hand, the search engine looks around in the bucket, and returns everything it finds there that seems to satisfy the query. This, of course, is not as simple as it sounds, and doing it well – that is, returning results that are both inclusive and relevant, and doing so quickly – requires great expertise, as well as an enormous investment in equipment.
But what’s in the bucket?
As mentioned above, the World Wide Web is huge, and is getting bigger every second. As fast as the best search engines can “crawl” through its links (and they can do so very rapidly indeed), it can still take a long time – weeks – before a page, once indexed, is revisited to see if it has changed. This is fine for documents that don’t change very often, but many pages are updated frequently, and the versions of them returned by a conventional search engine might well be so old as to have lost all their value. The best search engines, aware of this problem, do mark many URLs for more frequent indexing, but even at best, for any query there are going to be a lot of results returned that are old enough to be considered “stale”. If a site is updated daily, but the search engine only visits and indexes it every twenty days, then 95% of the site’s content will be unavailable to the search engine’s users.
Furthermore, there is a lot of the Web that is invisible to robotic “crawlers”. Many websites, rather than being static sets of documents, are dynamic pages, built on underlying data stores, that return different documents depending on a request by the user. This request can be in the form of queries typed into a text field, or can be part of the URL itself, but in either case the visible “surface” document, which serves only as a gateway, is just the tip of a much larger iceberg. All of this “unlinked” data, which cannot be reached by search engines, makes up what is called the “deep” or “hidden” Web, and in aggregate is many, many times larger than the visible Web.
So the “bucket” model, while it is enormously useful, and has made the Web into the indispensable practical tool it has become, has two major drawbacks: what is in the bucket might well be days or weeks old, and a lot of the Web (the vast majority of it, in fact!) is never going to find its way into the bucket in the first place.
We call this model “retrospective search”. The name reflects the nature of the typical interaction a user has with a search engine: the user goes to the search engine’s website and submits a query, whereupon the search engine looks through its collection of indexed Web pages and returns a collection of links. It’s a one-off deal; once the results are returned, the user’s query is discarded, and the transaction is over. The user has taken a look back over whatever the search engine has already found, and that’s why it’s called “retrospective”.
So what does PubSub do that’s so different?
“Prospective” Search: the “Firehose”
PubSub’s approach is to turn the conventional search model upside down. We don’t crawl the Web, and we don’t maintain a gigantic repository of indexed pages. What we store, instead, are our users’ queries. We aren’t concerned with the parts of the Web that don’t change over time; what we are interested in is the “changing web”: those pages that are updated all the time – every day, or even every minute. And we don’t crawl to get them; they come to us.
Pings
Here’s how that works: When a website updates its content, it can send a notification packet, called a “ping”, to anyone who is interested. Well, we’re interested, and through these ping notifications we are now able to detect, in real time, whenever changes or updates appear in (as of this writing) over 13 million different sources, including blogs, news services, press releases, and a variety of other things as well. This translates, at the moment, to over 2500 new items arriving at our doorstep every minute. This is an awfully dense stream of data, sort of an information “firehose”, if you will.
So, if we aren’t storing all this data, then what are we doing with it? That’s the really tricky part.
Matching
As I mentioned above, PubSub doesn’t store indexed pages; what we store is our users’ queries. So what do we do with all of the data pouring in through the “firehose”? We do matching, and to do this we use our own state-of-the-art matching engine. As each item comes down the pipe, the matching engine examines its content, then checks it against each of our users’ queries. If there is a match – in other words, if this new item satisfies the terms of the query – then the matched item is delivered to the user. Once the item has been compared to all of the stored queries, we’re done with it, and move on to the next one. In effect, what we are doing is taking the “firehose” of data and filtering it, pulling out only those items that our users want to know about.
If this seems like a lot of work, well, it is – but the PubSub matching engine is so fast that it is able to handle our current load of over a trillion matches a day without breaking a sweat.
You might have noticed one fundamental difference between our model and the traditional search engine (or even the “blog search” services like Technorati, Feedster, or BlogPulse): when you use a search engine, you submit your query, get the results, and you’re done. With PubSub, though, your query is persistent – we keep notifying you as long as you keep the query registered with us. That’s why this model is sometimes referred to as persistent search. We prefer to call it prospective search, though, because that neatly captures the way it complements traditional, retrospective search. We refer to our users’ stored queries as “subscriptions”, and that’s where our name comes from – when new or changed items are published to us from the dynamic Web, we evaluate them with our matching engine, and deliver the filtered results to our subscribers, in much the same way that magazine subscribers have the latest issues brought to their doorsteps.
The best of both worlds
So how are we different? We at PubSub feel that traditional search engines do a very good job, but only half the job. What we are offering is the other half of search: not discovery of the tremendous amount of data that has already been published, but real-time, ongoing notification of relevant future data, as soon as it appears, and whenever it changes. By complementing a traditional search with a PubSub subscription you will not only know what has already been said, but can also be sure you won’t miss what is about to be said.