To search... or not to search

There is no time like the New Year to rethink everything and take a step back in an attempt to see the proverbial forest from the trees. Often this comes to me in the form of wondering where the words we use come from.

The verb "search" comes through the Old French circare, meaning to "go about, wander, traverse," from the Latin circus or circle - A very fitting description indeed. The term comes to be known in the early fourteenth century and would exactly describe the process of looking for something or someone. An individual actually had to go about and wander around looking for what they sought.

Contrast this with the expectations placed on search engines today. Users expect the engine to know immediately upon asking, often using a query of less than two words, where the exact piece of content is they are looking for. If it does require a bit of wandering or traversing it seems to immediately frustrate the user. The desire is that the document most relevant to them is returned in the top results every time. Very little wandering or going about is expected by the user.

In reality the user is not performing a search but instructing the search engine to do so - yet we say "I am going to search for X." We query but the engine does all the "going about, wandering and traversing." This usage is very telling - the engine and the process of its searching has become an extension of the user. The expectation is that the engine, being a natural extension of themselves, knows their every desire and what they consider important.

In light of this should we should not be surprised at user's constant complaints regarding their search experience. Yet the industry seems to keep churning out more and more algorithms that focus on natural language processing, semantic search and other content focused approaches. Vendors seem to neglect and purchasers of enterprise engines keep pushing back deploying any sort of relevance methods that actually focus on understanding the user and fall for the newest vendor jargon year after year.

In this coming year I do not doubt we will see some very interesting technologies brought to market. They will undoubtedly allow us to find experts, tag results, star them and move them around, share them and socialize them - but will they seek to understand what is relevant to an individual searcher? Search profiles on a individual level do exist in some engines but usually remain fixed and static - ignoring context and behavior altogether.

I am putting in an early request - all I want for Christmas is my enterprise search engine to pay attention to me this year.

Search User Interface and User Experience

Just found a webpage that I want to share with you: http://www.searchtools.com/info/user-interface.html 

Just one page but with a treasure of information about designs and patterns made for search.

IT has not enabled counter terrorisme

The article on The Standard today about IT failing in helping intelligence agencies with discovering relations between pieces of information and “connecting the dots” pretty much amazed me.

As we all (at least us information access professionals) know, the intelligence agencies in the US (and other countries) use search engine software intensively.
The homepage of Autonomy states clearly that the Department of Homeland security uses their Intelligent Data Operating Layer (IDOL). I think it’s safe to presume that the agencies not only use Autonomy software exclusively. Home grown solitions are of course combined with the best of the breed search engines and discovery tools.

Clearly the companies that are implementing the IT solutions have not done their work well or the major search vendors are promising more than they can live up to.

From experience I know that a top search engine can be made useless because of a bad implementation. On the other hand I must say that the vendors almost always paint an over simplifide picture when it comes to getting the most out of their software.

 

The field of search and information discovery is a daring one. One thing I hope is that the experiences and insights that are given on past events are being used to find the wholes and causes for not seeing connections in advance. That kind of knowledge can help everyone in building better solutions and information processes that feed the systems.

I know a company that can help intelligence agencies getting the most out of their search and discovery environment ;-)

LucidImagination releases LucidWorks for Solr 1.4

Finally a couple of months after the release of the Solr 1.4 distribution by Apache, LucidImagination – a company that deliveres commercial level support for the open source Solr engine – releases their certified version of the Solr 1.4 Enterprise Search Server.

This distribution contains a comprehensive manual with lots of information on how to setup and use the search server.

The documentation is outstanding in quality and completeness. But that’s not all.

LucidImagination has succeeded in wrapping the Solr software in a user-friendly installer that makes it possible to get the search environment up and running in no time.

The thing that is missing at this moment is a “ready to run” indexing solution for a filesystem with binary documents like PDF’s and Office-type documents.
When they succeed in packaging that type of solution, combined with a ready to run HTTP crawling environment, LucidImagination has the potential of competing with the “big” and expensive enterprise search providers like Autonomy, Microsoft / FAST and Exalead.

 

Open source search is starting to pick up on the big pie of enterprise search solutions, but lacks the “click and run” possibility that other software vendors offer.

The complexity of getting a Solr solution up and running with real life data sources is what’s holding back the large adoption. It is still too much a toolbox.

Missing vendors in article on Network computing

In the article “Best and worst of times for enterprise search” the author, Paul Korzeniowski, mentions some vendors of enterprise search / eDiscovery solutions.

The list of vendors is missing some very important players though, like Autonomy, Attivio en Microsoft / FAST.

I would advise the author to be more thorough the next time