Last week I attended the October edition of the London Enterprise Search meetup, which gave us (among other things) our usual monthly fix of great talks and follow up discussions. This time, one of the topics that particularly caught my attention was the question of how to measure the effectiveness of enterprise search. Several possible approaches were suggested, including measuring how frequently users can “find what they are looking for” within a fixed period of time (e.g. two minutes).
Now I’m not saying findability isn’t important, but in my opinion metrics like this really seem to miss the point. Leaving aside the methodological issues in defining exactly what is meant by “find what they are looking for”, they seem predicated on the notion that search is all about finding known items, as if to suggest that once they’re found, everyone can go home. In my experience, nothing could be further from the truth.
Most ‘finding’ tasks are but a small part of a much larger overall task, and are at best the beginning of an information interaction episode, rarely ever the end. Much of the value we can add in delivering enterprise search solutions should be in understanding the complete task lifecycle and helping the user complete their overall information goals, which invariably extend far beyond simple known-item search. To me, findability is but one element of the overall search experience, which (particularly in enterprise environments) often involves significant elements of higher-level problem-solving behaviour such as analysis and sensemaking:
So why the fixation with findability? Part of the reason may be because it is both easy to understand (intuitively and quantitatively) and relatively easy to measure, with readily available metrics such as precision, recall, etc. But like the drunk searching for his car keys under the lamp post, just because it is more convenient, doesn’t mean it is the right place to look.
So I took the liberty of testing my own hypothesis against the data we used in the recent EuroHCIR paper, to see whether these intuitions have any basis in reality. I reviewed the scenarios we used in that study and counted how many of them actually were bona fide ‘findability’ tasks.
The answer? Two. Out of 104 enterprise search scenarios, less than 2% were categorised as findability tasks (i.e. locating a known item). The rest were focused on much broader goals, such as comparing, comprehending, exploring, evaluating, analysing, synthesising, and so on. Moreover, when findability was an influence, it was invariably part of a larger, composite activity, embedded in a longer sequence of analysis & sensemaking activity. So in that context, measuring the time it takes to “find what you are looking for” is at best a crude instrument; at worst, it simply measures the wrong thing.
Now of course, I’ve used a reasonably modest data sample here, and if you gather your own data, I’m sure your mileage will vary. So I plan to extend the analysis and dig a little deeper to look for further evidence to support (or contradict) the hypothesis above.
In the meantime, if you have some data or your own & you’d like to share (or even better, collaborate), I’d love to hear about it, either here or by email.
BTW, if you want to learn more about the ideas I’ve talked about above, the following are all good resources for further reading:
- Bates, Marcia J. 1979. “Information Search Tactics.” Journal of the American Society for Information Science 30: 205-214
- Cool, C. & Belkin, N. 2002. A classification of interactions with information. In H. Bruce (Ed.), Emerging Frameworks and Methods: CoLIS4: proceedings of the Fourth International Conference on Conceptions of Library and Information Science, Seattle, WA, USA, July 21-25, 2002, (pp. 1-15).
- Jarvelin, K. and Ingwersen, P. 2004. “Information seeking research needs extension towards tasks and technology”, Information Research, Vol. 10, No. 1. (October 2004)
- Kuhlthau, C. C. 1991. Inside the information search process: Information seeking from the user’s perspective. Journal of the American Society for Information Science, 42, 361-371.
- Marchionini, G. 2006. Exploratory search: from finding to understanding. Commun. ACM 49(4): 41-46
- Peter Pirolli and Stuart Card (2005). ‘The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis’, Proceedings of the 2005 International Conference on Intelligence Analysis, McClean, VA, May 2005
- O’Day, V. and Jeffries, R. 1993. Orienteering in an information landscape: how information seekers get from here to there. INTERCHI 1993: 438-445
- Rose, D. and Levinson, D. 2004. Understanding user goals in web search, Proceedings of the 13th international conference on World Wide Web, New York, NY, USA
Tony,
I couldn’t agree with you more. But…
People tend to think different about what an enterprise search solution is. In many occasions the search solution being analyzed is simply an intranet search solution. In that case findability or know-item-search is simply what it is for.
This confusion is created because of the propositon of for instance, the Google Search Appliance. That is sold with the concept of enterprise search, while most of the time it is implemented as intranet or document search.
In many cases that’s what the business and users want and expect.
So, I agree, but it is important to be clear about the definitions of search solutions and purposes.
Good post Tony. I get your point, however wouldn’t a search use case where the main task is to explore, learn, synthetise etc could also be considered a findability case scenario? At the end of the day is all about finding information (not just a single piece of content as you point out) that helps the user complete a task with success. Anyway, it’s probably a matter our semantic interpretation of “finding stuff”. Nevertheless, we use this concept mainly cos people just get it right away.
Edwin: yes, I think that’s a good point. There are ES systems out there that are at the ‘simple’ end of the spectrum. But that said, I think their influence is waning as user expectations and aspirations grow. And if we take this definition as our starting point:
Then it’s known-item search that becomes the edge case, rather than exploratory search.
Borja: yes, I think the issue here is one of terminology: I tend to draw an explicit distinction between findability and analysis / sensemaking to give a more fine grained lens for understanding user behaviours (and indirectly for identifying systems that *don’t* support the latter two behaviours). But I agree it’ll be a while before they become mainstream concepts in the same way as ‘finding stuff’.
A very good interesting topic. I agree that Search is just starting point. It is just the initial first few seconds experience. Later one gets into analysis.
Also we have to remember that Enterprise search is quite different from general web search.In web search probably 70-80 % times, we try to search something new or different every other day whereas in an Enterprise everyone have their roles defined and know that what he/she has to do. It does not change every now and then.So once I search and get the content what I want ,make them favorite or create a dashboard.And life goes on.
Long story short, I want to understand what does an Enterprise Search add value to a end users rather content consumers specially in a Business intelligence world.
I agree. I also heard Gary Marchionini saying that we should focus more on understanding than just finding in HCIR’11 Keynote. However, knowing that known-item finding is the most frequent search type in desktop environment (I’m writing a thesis on that topic in UMass), I’m curious how you got to the conclusion that the portion of finding task is only 2%. For example, was it based on the time user spent or the action taken?
Thanks.
Jin: It was based on the number of scenarios captured during professional services engagements that featured that activity. So, it’s biased in the sense that the data set was sampled from a population that is skewed towards BI-type applications (and for other reasons which we discuss in the paper). But it’s quite revealing nonetheless. We hope to acquire some new data to further validate these patterns (e.g. from non-BI search scenarios).
[…] Integration”. Both of these suggest a wider re-framing of the search problem, in which findability is just one (small) part of the overall search experience. In this context, the focus is no longer on low-level activities such as selecting relevant […]