Do we really need another web crawler when most enterprise search vendors include their own with the software? The folks at Norconex thought that the world did, because they could not find one that measured up to their standards. If you visit Norconex and hit the introduction to the web crawler: “Welcome To Norconex HTTP Collector” you will be given a detailed rundown about how the company’s web crawler makes up for the poor quality it finds in other products:
At Norconex, we have extensive experience with different crawler implementations as Enterprise Search integrators. While they all have their strength and weaknesses, we always wished we could get our hands on one that combines all the things we like, while minimizing many of the recurrent pain points we kept experiencing. After years of waiting for it we took matters into our own hands and the results is here. While at first its main goal was to facilitate our own job as integrators, we now hope it can benefit you too. Please be vocal about things you would like to see included in future releases.
They also include a list of Norconex’s features that appear to be standard, except for a few. Norconex makes a point of noting its web crawler is easy to for non-developers as well as the inclusion of commercial and community support. It also does not lock you into a specific vendor or platform and it has been tested on prove technology. What sets Norconex apart from other rivals is the company actually wants you to experiment with the software and it knows that it does not have all the answers. Norconex does not focus on Big Data or preserving document structure, but that is when you pull in a search vendor that does such as LucidWorks.