Last week I attended the October edition of the London Enterprise Search meetup, which gave us (among other things) our usual monthly fix of great talks and follow up discussions. This time, one of the topics that particularly caught my attention was the question of how to measure the effectiveness of enterprise search. Several possible approaches were suggested, including measuring how frequently users can “find what they are looking for” within a fixed period of time (e.g. two minutes).
Now I’m not saying findability isn’t important, but in my opinion metrics like this really seem to miss the point. Leaving aside the methodological issues in defining exactly what is meant by “find what they are looking for”, they seem predicated on the notion that search is all about finding known items, as if to suggest that once they’re found, everyone can go home. In my experience, nothing could be further from the truth.
Most ‘finding’ tasks are but a small part of a much larger overall task, and are at best the beginning of an information interaction episode, rarely ever the end. Much of the value we can add in delivering enterprise search solutions should be in understanding the complete task lifecycle and helping the user complete their overall information goals, which invariably extend far beyond simple known-item search. To me, findability is but one element of the overall search experience, which (particularly in enterprise environments) often involves significant elements of higher-level problem-solving behaviour such as analysis and sensemaking:
So why the fixation with findability? Part of the reason may be because it is both easy to understand (intuitively and quantitatively) and relatively easy to measure, with readily available metrics such as precision, recall, etc. But like the drunk searching for his car keys under the lamp post, just because it is more convenient, doesn’t mean it is the right place to look.
So I took the liberty of testing my own hypothesis against the data we used in the recent EuroHCIR paper, to see whether these intuitions have any basis in reality. I reviewed the scenarios we used in that study and counted how many of them actually were bona fide ‘findability’ tasks.
The answer? Two. Out of 104 enterprise search scenarios, less than 2% were categorised as findability tasks (i.e. locating a known item). The rest were focused on much broader goals, such as comparing, comprehending, exploring, evaluating, analysing, synthesising, and so on. Moreover, when findability was an influence, it was invariably part of a larger, composite activity, embedded in a longer sequence of analysis & sensemaking activity. So in that context, measuring the time it takes to “find what you are looking for” is at best a crude instrument; at worst, it simply measures the wrong thing.
Now of course, I’ve used a reasonably modest data sample here, and if you gather your own data, I’m sure your mileage will vary. So I plan to extend the analysis and dig a little deeper to look for further evidence to support (or contradict) the hypothesis above.
In the meantime, if you have some data or your own & you’d like to share (or even better, collaborate), I’d love to hear about it, either here or by email.
BTW, if you want to learn more about the ideas I’ve talked about above, the following are all good resources for further reading:
- Bates, Marcia J. 1979. “Information Search Tactics.” Journal of the American Society for Information Science 30: 205-214
- Cool, C. & Belkin, N. 2002. A classification of interactions with information. In H. Bruce (Ed.), Emerging Frameworks and Methods: CoLIS4: proceedings of the Fourth International Conference on Conceptions of Library and Information Science, Seattle, WA, USA, July 21-25, 2002, (pp. 1-15).
- Jarvelin, K. and Ingwersen, P. 2004. “Information seeking research needs extension towards tasks and technology”, Information Research, Vol. 10, No. 1. (October 2004)
- Kuhlthau, C. C. 1991. Inside the information search process: Information seeking from the user’s perspective. Journal of the American Society for Information Science, 42, 361-371.
- Marchionini, G. 2006. Exploratory search: from finding to understanding. Commun. ACM 49(4): 41-46
- Peter Pirolli and Stuart Card (2005). ‘The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis’, Proceedings of the 2005 International Conference on Intelligence Analysis, McClean, VA, May 2005
- O’Day, V. and Jeffries, R. 1993. Orienteering in an information landscape: how information seekers get from here to there. INTERCHI 1993: 438-445
- Rose, D. and Levinson, D. 2004. Understanding user goals in web search, Proceedings of the 13th international conference on World Wide Web, New York, NY, USA