Onderscheidend vermogen facetted search vervaagd

Dankzij een bezoeker van een "round table" sessie van search vendors kunnen we smullen van de opmerkingen van de grote zoekmachines zoals Autonomy, Google en Endeca:

"I was at a search vendor round table today...". Ook op de blog Beyond Search is dit artikel besproken.

In eerdere jaren was het feit dat een search engine die facetted search in zich had, een reden om die oplossing te kiezen boven andere.

In het artikel gaat het met name om de vraag "waarom moeten klanten veel geld betalen voor een technologie die door open source (Solr) oplossingen gratis worden aangeboden?".

Alle commerciële aanbieders van search software waren niet in staat om aan de lezer een goed argument te geven van het verschil...

Voor de afnemers van een zoekoplossing voegt dit aan keuze veel toe. Het maakt immers niet veel uit welke aanbieder gekozen moet worden voor dit aspect op basis van functionele wensen. Search wordt daarmee inderdaad steeds meer een "commodity".

Uiteraard moet de keuze voor een zoekoplossing op meer dan alleen dit aspect worden beoordeeeld, maar het geeft wel aan dat "speciale features" minder discriminerend worden.

Zoekfunctie in Sharepoint voldoet niet

Vandaag blijkt uit een bericht op fiercecontentmanagement dat gebruikers niet tevreden zijn met de zoekfuncties in Sharepoint.

Stephen Arnold haalt dit onderzoek aan in zijn weblog "Fierce critisism of Sharepoint".

Ik onderschrijf deze kritiek. Dit is nou juist waarom zowel Google als Autonomy gebruikers van Sharepoint zien als een potentiele klant. Het IDOL-platform én de Google Search Appliance hebben bewezen dat ze betere resultaten kunnen bereiken met het implementeren van een zoekoplossing op Sharepoint.

Sharepoint is (op dit moment) een goede oplossing voor content en (beperkt) document management. De zoekfunctie kan echter beter worden overgelaten aan andere, in search gespecialiseerde, zoekmachines.

Zoeken met een tilde (~) op Google

Een nieuwe(?) operator bij Google is de tilde (~). Door dit teken voor een zoekterm te plaatsen zoek je niet alleen naar het betreffende woord, maar ook naar synoniemen of vergelijkbare betekenissen van het woord:

If you use Google to navigate the Internet, this just might be the coolest thing you read today: There’s a simple operator that lets you search for a word and all of its synonyms. If you place a tilde (~) before the word or phrase you’re searching with no spaces between the tilde and its associated word, you’ll conduct a search for the word, its synonyms, and terms with alternate endings.

What Some people think of Enterprise search...

Is sometimes not what you think they mean.

Read the Article "Google Dominates Enterprise Level Search" (http://www.marketingpilgrim.com/2010/03/google-dominates-enterprise-level-search.html).

No... you have not been sleeping for 3 year while Google has penetrated the Enterprise. It is just a wrong use of the words "Enterprise search". The author means "the use of web search from within the enterprise to fullfil a business information need".

Here is my comment on the Article I posted on the site:

Hi,

When I saw the title of this article I was suprised. How can it be that, as an expert on enterprise search, I missed this disruptive news?

On the other hand I thought this must be a Google Sponsored Article because Google is just starting its search business in the enterprise.

Reading the article it came clear that you don't use the concept "Enterprise search" as we search experts do. I will make myself clear.

Enterprise search is all about making the information that is stored in databases, CMS's, DMS's, Fileshares etc. etc. WITHIN the ENTERPRISE accessible for SEARCHin and data discovery. Also see the definition on Wikipedia: http://en.wikipedia.org/wiki/Enterprise_search.
Players in this field are for instance Autonomy, Exalead, FAST (Microsoft), Attivio, Coveo and Endeca.
Google is big in internet search but still building its enterprise search proposition and market share. The Google Search Appliance (GSA) lacks some important functional aspects that are need in the enterprise search environment. Yahoo simple doesn't exist in the enterprise domain and neither is Bing.

What you mean in your article is the share that web search engines like Google, Yahoo and Bing have when searching business related information. When someone in a company while doing his or her work, searches for information on the internet, you call that enterprise search.

You should call it business-use of web search engines.

Blending "Real-time" results in resultlist

Today I had a meeting with a client and an external party that is going to develop the internal and external websites (intranet and internet) for them.

Subject of the meeting was the relation that the (re)development of the websites have with "search" functionality. We tried to get a vision on how the sites could evolve with the use of search technology.

Of course the issue of relevancy came to speak and I explained that relevancy is subjective and influenced or even driven by the context of the user and even of the context of the information it self (the semantics of information but it would be off-topic to get into that in this post).

One aspect of relevancy, "freshness" or "new information" triggered the person from the webdesign / webbuilding company to refer to Google and the integration of real-time result into their search results. He presented this as if current information came op automatically when searching for something and that this proved that the freshness of information was taken into account when calculating the relevance of all the results.

I would like to differ on this statement. When I look at the integration of real-time information in search results, Google has made a seperate section to show results from real-time sources like Twitter. This is the only way to do this, because there is a difference between "freshness" of a result and the "information value" of other results. You can not mix 140 character messages about, for instance, the islam with a profound description of what the Islam is al about.

Just watch this video and you will see what I mean. The Google interface serves 2 information needs by "clustering" the results in a "real-time"-section and a "contents / value based"-section.

Misconception on Google's revenues

In the article Is Google moving too far (from search) too fast? there is a misconception:

That expansion has some analysts wondering whether Google is in danger of losing focus on what made it such a profitable company, even as those same analysts say it can't rely on search as its only avenue for making money. Right now, Google relies on search for 95% of its revenue, according to Karsten Weide, an analyst with IDC.

Google does not rely on search for it's 95% revenue. They rely on advertisements (Adwords / Adsense) around there search product on the internet. They do not sell their search service, they sell space for advertisements.
Of course they are trying to get into the enterprise where they will be making money with their search product, the GSA. But for now, those sales are just a fraction of the rest.

To search... or not to search

There is no time like the New Year to rethink everything and take a step back in an attempt to see the proverbial forest from the trees. Often this comes to me in the form of wondering where the words we use come from.

The verb "search" comes through the Old French circare, meaning to "go about, wander, traverse," from the Latin circus or circle - A very fitting description indeed. The term comes to be known in the early fourteenth century and would exactly describe the process of looking for something or someone. An individual actually had to go about and wander around looking for what they sought.

Contrast this with the expectations placed on search engines today. Users expect the engine to know immediately upon asking, often using a query of less than two words, where the exact piece of content is they are looking for. If it does require a bit of wandering or traversing it seems to immediately frustrate the user. The desire is that the document most relevant to them is returned in the top results every time. Very little wandering or going about is expected by the user.

In reality the user is not performing a search but instructing the search engine to do so - yet we say "I am going to search for X." We query but the engine does all the "going about, wandering and traversing." This usage is very telling - the engine and the process of its searching has become an extension of the user. The expectation is that the engine, being a natural extension of themselves, knows their every desire and what they consider important.

In light of this should we should not be surprised at user's constant complaints regarding their search experience. Yet the industry seems to keep churning out more and more algorithms that focus on natural language processing, semantic search and other content focused approaches. Vendors seem to neglect and purchasers of enterprise engines keep pushing back deploying any sort of relevance methods that actually focus on understanding the user and fall for the newest vendor jargon year after year.

In this coming year I do not doubt we will see some very interesting technologies brought to market. They will undoubtedly allow us to find experts, tag results, star them and move them around, share them and socialize them - but will they seek to understand what is relevant to an individual searcher? Search profiles on a individual level do exist in some engines but usually remain fixed and static - ignoring context and behavior altogether.

I am putting in an early request - all I want for Christmas is my enterprise search engine to pay attention to me this year.

Google Wave and Microsoft Sharepoint... NOT alike

Today I read an article with the title "Can Google succeed outside of search?"

One piece of information got my attention a little bit more:

But Google keeps pushing: Its still-in-beta Google Wave promises to attack one of Microsoft's most beloved products, its SharePoint collaboration software.

To my knowlegde and experience with Google Wave, Wave is an online community platform and not so much a collaboration platform. Yes people can talk to each other, send messages, maybe even share a document through some link, but that is to simple to be compared with sharepoint.

The review of Google Wave vs. Microsoft Sharepoint on the Bamboo team blog makes the difference even clearer:

First of all, the question, "Is Google Wave a SharePoint Killer" is a completely artificial, nonsense argument dreamed up by either the Google PR team and/or the media to generate buzz. Wave is an email client (very tempted to say "consumer email client")... an exciting and innovative one, but it's not a collaboration platform... at least if your definition of collaboration is more sophisticated than sharing pictures from a boating trip. Wave doesn't compete with SharePoint, it competes with Outlook.

That being said, it has occurred to me recently that one of the weaknesses of SharePoint is its failure to account for the fact that email is and will continue to be the fundamental central connective tissue to all collaborative activies. As you might imagine, Bamboo has one of the most sophisticated, tricked-out SharePoint portals in the world, and yet even here, people continue to exchange documents in email and communicate issues in email that *should* be written into a wiki or discussion board. Why? Convenience. So long as it is more convenient to fire off an email than to upload a document or navigate to a message board, that is what people will do. Wave does address this problem head on, and I think introduces a new paradigm that Outlook will have to adopt, quickly.

But an enterprise collaboration platform? Hardly. There is no sign of document management, content management, workflow, data integration, dashboarding etc. etc. If Google truly believes that Wave is a platform for enterprise collaboration, it only serves as proof that they really don't understand the business. Maybe if they fully integrated Wave, Google Sites, Google Apps, Google Gears, Google Calendar in a comprehensive platform... then you might have something. But Microsoft has a big head start in lashing together all of these disparate capabilities and a huge installed base of customers to drive innovation based on real world feedback.

I sometimes wonder if the "reporters" who send out this kind of information to the world, just lack somekind of insight or that thay deliberately say things that are just not correct. Double check your sources and trust your own knowlegde.

phpQuery: using Google as an information source

For some project I have to get search results from Google and follow the links on the result page to index those valuable resources.
Google doesn't supply an XML-interface to it's search results.

Bing (www.bing.com) and Yahoo (www.yahoo.com through an API key) do. Aside to the subject of this blog, I find this monopolistic behaviour of Google repulsive... They index al OUR content and resources, make money of it through advertising, and then they stop the "open source" character of their activities. Once our content enters their databases, the content is theirs!
Start using Bing more often!

Because of Google does not support an XML-response interface, I have find a way to "crawl" our own content.

I have found the "phpQuery" framework, ironicaly hosting on Google code.

What I wanted to do is get all the relevant links from the search results page and follow those links to make the information aivailable within the context of the search application.

I want to share the basic code that I have come with to establish getting the links from a webpage through the use of phpQuery:

require_once('phpQuery/phpQuery.php');

//The URL of the webpage to fetch
//$link = "http://www.emidconsult.com";
$link = "http://www.google.nl/search?q=multiple+sclerose";

//Get the whole HTML contents
$page = file_get_contents($link);
//Put the HTML content in a phpQuery object
phpQuery::newDocumentHTML($page, $charset = 'utf-8');

//Get the [body] contents out of the phpQuery object
//$body = pq('body');
//print $body;

//iterate through every link (a element) in the webpage.
foreach(pq('a') as $ref) {
//Get the link
$href = pq($ref)->attr('href');
//Show the url in the a element, but only if it contains external links
$pos1 = strpos($href, 'http://');
//and has no reference to Google own search commands
$pos2 = strpos($href, '/search?q');

if (($pos1 === false) || ($pos2 > 0)) {
//DO NOTHING
} else {
echo $href."\n";
//Now do some things with the href...
}
}
?>

Google and making multimedia searchable

Today I saw the announcement "Google adds automatic captions to YouTube". I really like this addition to the possibilities of Youtube.

But as we all know Google as a gigantic eposure in all thing they do.

This feature seems a neat thing that makes it easier for consumers to upload there video's to Youtube regardless of language and eliminating the need for adding manual captions or descriptions to there video's: Google converts the speech to text automaticaly...

As i am more involved in information access and enterprise search I place this news in another context: The possibility of making video and audio searchable.

Within the enterprise this issue of making video and audio searchable is getting more attention. Autonomy as well as Exalead are also focussing on this part of enterprise search.

To see the capabilities of Exalead you can visit their labs on http://labs.exalead.com/experiments/voxalead.html.

Autonomy also is very active on the market of media indexing. They have solutions like Virage that to the same thing. I find it very disapointing that Autonomy has no demo of their capabilities.

Bottom line?
Google is doing the same thing as the large search technology providers. The "public" will say that Google is "ahead of the pack". The advantage that Google has is a large community that follows everything that they do and spread the word fast.
Other search providers like Autonomy and Exalead can do and are doing the same thing as Google is doing now (and are maybe better), but are not in the position to reach the audience that Google can.

Google GSA gets smart (and smarter automatically)

Google announced some updates for the generation 6 of the GSA. They added a functionality that boosts relevance for documents / terms automatically when the link is clicked in the search results.

From InformationWeek:

Chief among the additions is a new algorithm
called Self-Learning Scorer, which analyzes employee clicks and behavior to
improve the relevance of search results.
If, for example, most employees searching for a given term click on the third search result, the GSA will place that term higher on the search results page for future searches.
On the Internet, Google gets highly relevant search results in part because of its
PageRank algorithm, which analyzes Web links and treats them as votes for
relevance. On corporate intranets, a rich link structure is absent, which makes search relevance harder.
The Self-Learning Scorer system should help compensate for the absence of Web links inside corporate firewalls.
[...]
"In other words, the GSA is moving upstream, while keeping the price low, and continuing to disrupt the confused search market," says Feldman in the report.

Google makes good use of "User Generated Content", in this case the fact that a clicked result is the "proof" of the document being relevant for that particular search.

It is good to see that the difference between internet search and enterprise search is becoming clearer.

More Google browser specific choices

From http://www.washingtonpost.com/wp-dyn/content/article/2009/10/14/AR2009101400362.html

Our tipster says that he's only seeing the new ads in the developer version
of Chrome, but I'm seeing them as well in Safari, though some TechCrunch staff
aren't seeing them in any browser. Google is always switching up ad placement
and formats in various bucket tests, some of which are browser-specific, so the
inconsistency isn't surprising.

It seems that Google has some favorites when it comes to testing ;-).

Visualisation of Enterprise Search ROI with Google GSA

The dutch company VLC has a great little application on their website that lets you interactively determine the ROI you will have when investing in a Google Search Appliance (GSA). They make use of the following factors:

• Documents
• Employees
• % of knowledge workers
• Cost per employee
  

Below is the text on the website of VLC (in dutch):

Bereken de voordelen van een goede zoekoplossing

Met een simpele formule is te berekenen hoeveel geld bespaard kan worden bij het goed doorzoekbaar maken van informatie met een Enterprise Search oplossing.

# medewerkers × (gemiddeld jaarsalaris / 2,080 uur) × (besparende minuten / medewerker) = € besparing per jaar.

Onderstaande applicatie voert deze berekening uit. Sleep de balletjes op de balkjes heen en weer en stel dit in op de informatie van de organisatie. Zo is te zien dat er snel een hoge productiviteitsbesparing te behalen valt bij het inzetten van een goede zoekoplossing.

Going from one monopolist to another

http://blog.seattlepi.com/microsoft/archives/180794.asp

Google is offering Office productivity tools in de cloud through Google Apps. To my knowlegde they are the only one with such a rich online solution.

When more and more people are going to use Google Apps, the suite is going to be a new standard. That will attract more users because it easier to exchange files in the same format.

And what about the "open-ness" of the Google Apps document format. Sure you can save a document in another format and then open it in that program (let say "Word"). But when you have edited that document with Word, you can not save it as an Google Apps document without trouble. You can upload the file as a new document, but that's not what you want. Also the layout of the document will be messed up...
I say: when you chose for Google Apps you are bound by it, as is the case with the choice for another office suite.

When Google has the biggest user base in the world, will they be not the next monopolist in the market?

Consider this:
The fact that all your data is on their servers is even more creepier.

With the offline Microsoft Office solution your data is still safe in your own network.

At the moment we use Google Apps only for "work in progress" and notes that must be shared or colaborated on.
The final documents and reports we still make and save with Microsoft Office (and Open Office although that causes compatibility issues). The files are safely kept under our own influence.

In the field of defying standards, Google is following Microsoft

I have said it before, and I will say it again: Google is growing out te be the next Microsoft. Driven by shareholders value the company is no longer the underdog. They are starting to set their own standard. The public opinion is changing...

By implementing currently non-standard features on their homepage, Google are
sending out a strong message on what they believe the new standard features
should be, and not coincidently, it is the features that their own browser
implements and supports. This is not the first time Google has sent a
wrecking-ball into the standards process. Google Gears was launched long before
Chrome as a way to implement proposed HTML5 standards, such as offline caching,
into browsers (see my NextGenWeb series from last year). It was born out of frustration
for the slow and beurocratic standardization process ¿ something that Google
couldn't afford to wait for as their web applications could not advance further
without a non-aligned platform to build them on.
A large part of the
anti-trust case against Microsoft was that with combined desktop, browser and server
market dominance the company could abuse that position to make the web a
Microsoft web by implementing Microsoft-only features. Google are using their
dominance to force an issue that has been stalled for far too long ¿ but the
difference is that they are using their force for potentially a greater good (I
hope). The theoretical Microsoft web would have been "this website only supports
Internet Explorer", whereas with Google so far we have "this website is a lot
better, and has sexy buttons, if you use Chrome (which btw is open source)".

source: http://www.washingtonpost.com/wp-dyn/content/article/2009/10/08/AR2009100800192.html

Fuzz about Google search options

The past few weeks I see so much news and blog articles about the "Search options" that Google has rolled out on their image search application. They added this feature a couple of months ago for there regular websearch to.
They are following the example of Bing who has had this from the start.

I don't see the big news here because "search options" are a standard feature of any enterprise search solution (Autonomy, Solr, Exalead). If you don't have faceted / parametric search the the search solution is useless.

The fact that so many websites see this feature as news has opened my eyes to the fact that so few people are used to real and usable search solutions at work. This strikes me as odd, because how are they able to find the info within their organisation then?

No wonder that the enterprise search / information access market has an estimate share of billions of dollars.

Chome OS... De Google val slaat dicht

Much fuzz about the announcement of the Google Chrome OS the past few days.

I must say that I am intrigued by the concept myself. My first reaction was like "Off course, after the Google Apps, Chrome Browser and Android this is the next step".

Google is revealing itself more and more like the other established players as Microsoft, IBM, Oracle etc. They are actually building a stack of software and platforms that lock in any company that is standardising on their software.

Google is more and more becoming a "normal" IT-company that sells software and tries to bind customers to their solutions.
There are 3 aspects to this issue:

1. Google still is a company that gets it's revenues from advertising
2. Google is starting to dominate (monopoly??) some markets (advertising)
3. Google is known for giving away free services

I wonder where this is going... Google is not the David fighting Goliath with small stones anymore. They are a David on steroids trying to get Achilles and Goliath at the same time.

Read more about this topic:

• With the Chrome OS, Google's software stack is revealed; Shrapnel everywhere
• With OS project, is Google over-extending itself?

Google announces the 6.0 version of the GSA, Search experts are not impressed

Today Google announced the release of the 6.0 version of their Search Appliance
The main improvements are

• better scalability
  The Box can now hold up to 30 million documents and can be connected to more appliances to hold billions of documents
  
• Flexible architecture
  It is easier to connect more appliances
• Social Search capabilities
  Add page to result.
  

Some excerpts from other sources:

http://www.itworld.com/software/68712/google-search-appliance-now-can-index-billions-documents

In addition, GSA 6.0 provides end users with social search capabilities, such as the ability for them to add results.

"Google has matured in its thinking about how enterprises implement search," said Rebecca Wettemann, an analyst with Nucleus Research.

Enhancing the Search Appliance in areas like scale, customization and integration will resonate positively in the enterprise market, where Google's main challenge is being perceived primarily as a consumer-focused company, Wettemann said.

While the Search Appliance's functionality isn't still as sophisticated as that of high-end products from Autonomy and Microsoft's Fast Search, it has moved upstream with this latest round of enhancements, said IDC analyst Susan Feldman.

http://www.forbes.com/2009/06/02/google-enterprise-search-technology-cio-network-google.html

Forrester Research analyst Matthew Brown says the product improvements may be as dramatic as Google says they are. But taken in context, Google is still trying to play catch-up to a slew of start-ups and giants like Oracle, he says.

"I'm just kind of unimpressed with what they're announcing," says Brown. "They're coming to market so late, with requirements that were established years and years ago."

Google add a page to a result

I didn't know that this feature existed.
You can add a page that you missed in de result to the result page of a specific query.

Another brilliant idea from the boys of Google. This way you collect "best results" with the direct input from (active) users.
A similar functionality is the "SearchWiki" which let you add comments to pages in the results that are return by a query.

Google Apps is getting more and more users

According to this blog post by Google, they have more than 3 million users at educational organizations.

It is not so surprising that Google gets very much attention in this user base. Students nowadays grew up with one company they now best. It is the company that delivers the only internet searchengine they know. The company that delivers the E-mail application they use. The company that lets them navigate on their mobile phone. Read the post on Classroom collaboration to illustrate what I mean.

Computing in the clouds is so easy and Google delivers.

Just as a complete generation grew up with the products of Microsoft, A complete generation is now growing up with Google. I see no big difference. It is just a matter of time when Google gets the same negative public attention that Microsoft got. The reason is simple...

When a company gets to "big" or has to much influence the natural "big brother" reaction of the public takes control....