Recently I took part in a panel discussion at SemTech 2009 on the timely topic of “semanticity” of search engines. In the last few years, many "semantic” search engines have been launched, and the term “semantic” has become open to broad interpretation and use due to the lack of defined industry metrics. It’s a question I’m often asked at conferences: “What qualifies a search engine as semantic?” Is the use of some Natural Language Processing (NLP) technologies sufficient to award such a definition? When can a user say that the search box really understands his or her queries in the same way another human would?
In a new paper entitled, “If You Ask Nicely I Will Answer – Semantic Search and Today’s Search,” which I’ve co-authored with Alessio Signorini, we propose a family of metrics to evaluate the semantical invariance of search engines, and we report experimental results for well-known search engines. I will present and discuss this paper at the IEEE Conference on Semantic Computing (ICSC 2009) this fall in Berkeley.
Here is an abstract of our conclusions, and let me start with an example from our IEEE paper:
Imagine a four-year-old as a human search engine…
Imagine a four-year-old as a human search engine…
Let’s suppose that search engines had the intelligence of a four-year-old child. If that is the case, one could just imagine the following dialogue taking place:
User: How is the weather in Hawaii at this time?
Engine: I do not know.
User: What is the weather in Hawaii islands right now?
Engine: I don’t know!
User: OK, “current weather in Hawaii”
Engine: How many times will you ask the same question? I told you already, I have no idea!!
User: How is the weather in Hawaii at this time?
Engine: I do not know.
User: What is the weather in Hawaii islands right now?
Engine: I don’t know!
User: OK, “current weather in Hawaii”
Engine: How many times will you ask the same question? I told you already, I have no idea!!
Even though that human search engine is entirely clueless about the status of the weather in Hawaii, it is nevertheless semantic: it knows that it does not know. But it does understand that the user keeps asking the same query, although differently phrased.
Humans recognize quickly that two questions can really be simply a different phrasing of the same one. Yet search engines most often don’t understand this. And until they do, they cannot properly be called semantic. So, “Top 10 songs”, and then, “Top ten songs” bring different – albeit still relevant – results. But they should not…
It’s not the technology you use, but the effect which you achieve…
How invariant search engine results are under rephrasing (paraphrasing, extra hints, etc.) is, to us, a reflection of how semantic the search engine is.
It is largely irrelevant how semantic invariance is achieved (i.e. which search technology is used: NLP, statistical analysis of query sessions, etc). What matters is the final effect. If semantic invariance is poor, users have to work harder… or – in human terms – they must “ask nicely”. But what that tells us is that search engines are not doing the work they should be. They are not bearing the burden for the searcher – which they should be. I believe this can be measured – as we propose in our IEEE paper – by massively testing semantic invariance.
To this end, we propose simple metrics based on the entropy of the results that a search engine returns for clusters of semantically equivalent search queries. We can measure the overlapping of results (are they stable? does a different URL move to the top if you just ‘ask differently’?). Does an “ORA” (one right answer), such as “who won the Super Bowl in 2006,” become impacted by how you actually phrase the question?
Here are our conclusions, which we provide in more detail in our IEEE paper:
1. First, the invariance of results for general search queries is still poor. Today’s search engines are very sensitive to the way queries are actually phrased. They are all mostly keyword-based, and far away from simulating human query understanding.
2. Second, search queries with a “One-Right-Answer” for popular subjects (such as the Super Bowl query) appear to be well served by today’s search engines, which manage to return the correct answers in their result pages with surprising invariance to the form or manner of the query. Unfortunately, this is most likely to be a Pyrrhic victory to today’s engines – because this success is most likely a consequence of the massive redundancy of information on the web. There are multiple pages talking about the same “facts” in different ways. For topics of massive interest (e.g. the World Cup) many people create pages with the same or similar content. The subtle differences on the language and structure used to present this information help search engines to deliver at least one copy of the information through simple keyword matching.
2. Second, search queries with a “One-Right-Answer” for popular subjects (such as the Super Bowl query) appear to be well served by today’s search engines, which manage to return the correct answers in their result pages with surprising invariance to the form or manner of the query. Unfortunately, this is most likely to be a Pyrrhic victory to today’s engines – because this success is most likely a consequence of the massive redundancy of information on the web. There are multiple pages talking about the same “facts” in different ways. For topics of massive interest (e.g. the World Cup) many people create pages with the same or similar content. The subtle differences on the language and structure used to present this information help search engines to deliver at least one copy of the information through simple keyword matching.
The recent widespread adoption of Search Engine Optimization (SEO) techniques also plays an important role in this challenge. While those techniques are unfortunately often associated with spam, their original intent was legitimate: help search engines to do a better job while indexing and ranking page contents and URLs.
A truly semantic search engine would take care of invariance at the query level, clustering together all its possible rephrases into one unique concept to each answer, and would deal equally well with popular and unpopular topics (from the World Cup to, say, a small local soccer league in Quito, Ecuador).
The data we collected also confirms that the stability of results under rephrasing for general queries is still poor in all the major search engines. Our experiments with simple numeric synonym replacements (e.g. “10” with “ten”), as well as the ones involving the addition of redundant category terms, indicate the heavy reliance on text matching.
The keywords used in the queries, and their position, strongly influence the distribution and the order of the results returned. This is unacceptable in a semantic world of advanced search engines, where the common goal must be to lift from users’ shoulders the burden of “asking in the right way” to get the right answer.
There’s much more on semantic search you’ll be hearing from me and from Ask in the weeks and months to come. You can pick up a copy of our paper at IEEE to see what else we found… and look for it here too on the Ask Blog.
Comments:
1) tm lewin voucher discount code : Making search engines semantic is a best way for promotion.
2) wiggles bikes discount code : I was seeking search engine article from Dr. Tomasz Imielinski. thank you.
Comments:
1) tm lewin voucher discount code : Making search engines semantic is a best way for promotion.
2) wiggles bikes discount code : I was seeking search engine article from Dr. Tomasz Imielinski. thank you.
