Blending "Real-time" results in resultlist

Today I had a meeting with a client and an external party that is going to develop the internal and external websites (intranet and internet) for them.

Subject of the meeting was the relation that the (re)development of the websites have with "search" functionality. We tried to get a vision on how the sites could evolve with the use of search technology.

Of course the issue of relevancy came to speak and I explained that relevancy is subjective and influenced or even driven by the context of the user and even of the context of the information it self (the semantics of information but it would be off-topic to get into that in this post).

One aspect of relevancy, "freshness" or "new information" triggered the person from the webdesign / webbuilding company to refer to Google and the integration of real-time result into their search results. He presented this as if current information came op automatically when searching for something and that this proved that the freshness of information was taken into account when calculating the relevance of all the results.

I would like to differ on this statement. When I look at the integration of real-time information in search results, Google has made a seperate section to show results from real-time sources like Twitter. This is the only way to do this, because there is a difference between "freshness" of a result and the "information value" of other results. You can not mix 140 character messages about, for instance, the islam with a profound description of what the Islam is al about.

Just watch this video and you will see what I mean. The Google interface serves 2 information needs by "clustering" the results in a "real-time"-section and a "contents / value based"-section.

Search initiative Newssift is no more

Just read an article about pulling the plug on the advanced search feature of the Financial Times: Newssift.
About one year ago the interface got some pretty positive reactions, due to the fact that this kind of user interface was fearly new on a public website.
Powered by Endeca the functionality heavily made use of facets. All categories, subjects etc. were made-up of metadata in the content, were the values are used to generate the lists of choices.

It's a shame that FT has no logging or other results as to the way the service was used and if the users liked or disliked this new way of searching FT content.

As a search professional I am always looking out for experiences with non-conventional search interfaces. Could this experiment have been more succesfull when guided and analyzed?

See: http://paidcontent.co.uk/article/419-financial-times-shuts-semantic-search-service-newsift/

Excellent clarification on semantic search

Today I received an update on one of te discussions on Semantic search on LinkedIn. Charlie Hull put up an excellent example on how semantic search works. This has to do with the capabilities of the search technology that is used in a specific situation, but also with the fact that a search application has to engage in a dialog with the user to assess his meaning or context. This has to do with the fact that most users just use 1 to 3 words to formulate a query. There's not much you can do with such a query in the first try. But... the search application has to pick up on those keywords and try to make something out of it.
The next step is to try to ask the user what he means.

Semantic search technology - does it actually exist? 33 comments »

Started by Charlie Hull

At Expert System we have been building semantic search systems for 20 years. Here is what we learned in serving 100's of corporate customers. A semantic search system must establish and store the CONTEXT of content. Then you need an interface to choose the CONTEXT you would like so a match can be made.

Establishing CONTEXT means the following processes must be followed. 1). word morphology (e.g. stems), 2). word roles (e.g. nouns, verbs, etc.) 3). word logic (e.g. subject - verb - object reduction) and 4). sense disambiguation (e.g. assignment of a definition for each word based on the best fit from available alternatives and in the context of the rest of the sentence(s). All 4 of these methods require the use of a semantic network that is both broad - covers the majority of the language to be used and deep - has many ways in which words relate to one another.

With the above approach will you reach a precision ("accuracy") and recall ("completeness") in search beyond the 80% mark. With further customization a 90% mark is easily achieved. Systems that rely on statistical / heuristic methods typically fall far short of these benchmarks. This is true since statistical / heuristic methods cannot fully establish logic and disambiguation.

Finally the interface must be constructed in a way that allows the user to tell the system what CONTEXT the query is in. Full natural language questions using the above methods can do this automatically. But the reality is we live in a 1-3 query word world. So allowing the user to select the word sense of one or more of the query words gives the system much more "to chew on" and is not generally an intrusion for the user. Similar user interface interactions include showing categorical, domain, people, places, organization outcomes from a search which are clickable, showing lists of semantic triples (subject-predicate-object) from which to choose, etc. All of these are at most 1-2 more clicks than a normal keyword based search but improve the experience immensely.

Such interfaces also allow what we call a 3-step walk through search where step 1 is about precision - less of a list, step 2 is an expansion of concepts - to include things related but that you did not know about, and step 3 another step of precision. This "ratcheting" effect therefore begins to bring into the Enterprise Search function other important aspect of corporate work like discovery, exploration and analysis. http://www.expertsystem.net By Brooke Aker

Semantic Search Engine: Inbeta... as in "not Alpha"?

Through a discussion on LinkedIn about real life examples of semantic search, I was pointed to the existence of Inbeta. I company with a curious name because it says that the company has a "Bèta"-status. I can't image what that says about there offerings.
But now for the offerings of the company. On their product page they have many products listed.
Of course the first one caught my eye because a "semantic search engine" is something that everybody dreams of. Imagine a search engine that gives you insight and context regarding the query of the user in relation to the information at hand and maybe also on external resources by using the sematic relations between information....

But wait... Before you think I found the holy grail of search, The sentence

"Natural Language: user will not need to search for keywords anymore, our Semantic Search understands the aim of every search query and suggests results that are relevant, thus increasing cross-selling and saving customer care costs"

had my feet put on the ground again.

This proposition on using natural language as query input and giving back relevant results based on the combinaton of words that most likely exist in the available search index, is something that has been here for years. Autonomy has marketed that concept with the name Meaning Based Computing. It all revolves around the concept of terms and weights withing documents and in relation to the words in the entire index (corpus) and matching the queried words to these calculated figures.

For a serious search engine a regard this technique almost as a must have.

But, back to the semantic side of this... Where is it?

When you want a good example of what semantics can do within a search application, take look at http://www.freebase.com/view/en/barack_obama.

It has everything to do with the context of the concepts that can be derived from a query. People have roles and jobs, names can be linked to artists, historical data etc.

Query-time JOIN operator

Everyone who is active in the information access business knows that it is sometimes very necessary to combine the data from two resultsets into one.

Example:
The main search focusses on finding information from within a document. A document can have relations with many other datatype like geographical data like authors. Authors can have metadata themselves, like age, hobbies etc.

Now let say you want to find documents that contain the keyword "snow" and that are written by authors that have the hobby "skydiving", or you want to show the hobby of the author of a book in the result list.

For this to be searched in a search engine that doesn't have the possibility to combine the two types of information, you have two options:
1. to make this kind of search the data has to be flattened. With this we mean that all the information that can be related to a document must be indexed along with that document. This means that the fact that author X has the hobby skydiving, this must be stored with every instance where the author is X, while we already know this. This can lead to a dramatic expanding index.
2. If you want to show information from another recordset, then you have to make 1 extra query for each result in your main resultset (documents and authors) to find the hobby of an author.

In this day and age we are trying to make information more accessible and usefull by showing relations that search results have with other types of information so that users get more insight.
Especially within BI applications this functionality is needed because the type of systems that have to be connected is very divers and the data can not always be flattened for reasons of diversity or just because this would mean that lot's of information has to be duplicated...

The query-time JOIN function is very powerfull to make this possible.

Not very many search vendors have this function in there product. I know that Attivio and Exalead are capable of doing this.

Misconception on Google's revenues

In the article Is Google moving too far (from search) too fast? there is a misconception:

That expansion has some analysts wondering whether Google is in danger of losing focus on what made it such a profitable company, even as those same analysts say it can't rely on search as its only avenue for making money. Right now, Google relies on search for 95% of its revenue, according to Karsten Weide, an analyst with IDC.

Google does not rely on search for it's 95% revenue. They rely on advertisements (Adwords / Adsense) around there search product on the internet. They do not sell their search service, they sell space for advertisements.
Of course they are trying to get into the enterprise where they will be making money with their search product, the GSA. But for now, those sales are just a fraction of the rest.

Autonomy acquires MicroLink

Yesterday the guys from New Idea Engineering came with the rumour of a "large search vendor" buying in on a "partner" that has big business within the intelligence industry...

Today the Guardian releases the news that Autonomy has acquired Microlink.

MicroLink is a very valued Autonomy Partner that only 2 years ago was "partner of the year" because of their big revenues on selling IDOL licenses. MicroLink is the partner that implements Autonomy (IDOL) software within American intelligence organizations.
Big business for Autonomy and strategically important.

The question of the acquisition is: "Why, Why, Why?".

This move is not in line with the buying of Interwoven or Zantaz. Those companies / technologies added a complete suite of functionalities to penetrate other markets than just the enterprise search one.

Could it be that Autonomy wants to have more influence on the implementations with those important clients? Or could it be that MicroLink has some brilliant technological invention???