Introduction to Natural Language Processing (slideshow)

Earlier this week I gave a talk called “Introduction to NLP” as part of a class I am currently teaching at the University of Notre Dame. This is an update of a talk I originally gave in 2010, whilst working for Endeca. I had intended to make a wholesale update to all the slides, but noticed that one of them was worth keeping verbatim: a snapshot of the state of the art back then (see slide 38). Less than a decade has passed since then (that’s a short time to me 🙂 but there are some interesting and noticeable changes. For example, there is no word2vec, GloVe or fastText, or any of the neurally-inspired distributed representations and frameworks that are now so popular (let alone BERT, ELMo & the latest wave). Also no mention of sentiment analysis: maybe that was an oversight on my part, but I rather think that what we perceive as a commodity technology now was just not sufficiently mainstream back then.

Advertisements

Read Full Post »

Google Scholar: not so smart?

Most of us are familiar Google Scholar: a freely available subset of Google that indexes the world’s scholarly literature across a range of disciplines. With its database of over 389 million documents including articles, citations and patents, it has become an indispensable resource for scholars and researchers across the globe. Which is why we recently added Google Scholar integration to 2dSearch, thereby offering a tool of immediate utility to anyone wishing to search the world’s scientific literature in a systematic manner.

Now, we’d always known that despite its extensive index, searching GS is subject to the ‘secret sauce’ of Google’s search algorithms, and that this can compromise the ability of users to formulate sophisticated, reproducible searches. But what we perhaps didn’t realise was just how limited that support is. In particular, GS seems to support only 1 set of synonyms (i.e. a substring with terms separated by OR). For example, the following search string would seem to parse correctly:

Read Full Post »

The snakes and ladders of search query formulation

If you were tasked with designing a universal framework for search query formulation, where would you start? Well, you could start from a command-line paradigm. After all, that’s the approach adopted by most query builders and ‘advanced search’ forms. But convention aside, is that really the best place to start? Personally, I am not convinced – IMHO command line approaches reflect the days when searches were conducted via remote terminals to subscription databases, and in that respect, they represent the past, not the future. Moreover, using Boolean strings to articulate complex information needs suffers from a number of fundamental shortcomings, in particular regarding scalability, efficiency and transparency. So what’s the alternative?

Read Full Post »

Towards transparency in systematic searching

A couple of days ago I published a piece on Medium called “Searching PubMed just got smarter“, in which I talked about how we’ve added PubMed integration to 2dSearch, thereby offering a tool of immediate utility to anyone wishing to search MEDLINE in a systematic manner. The service has only been live a week or two, but the post did seem to strike a chord with a number of people. And some of that commentary got me thinking about various issues around reproducibility (with regard both to scientific research and to searching methodologies). But it also raised an issue around transparency, and the degree to which the formalisms we use to express information needs can scale to accommodate new use cases and requirements.

Read Full Post »

A universal framework for information needs

A couple of days ago I published a piece on Medium called “Towards a universal language for search“, in which I talked about how we’ve added Google integration to 2dSearch, thereby allowing a user to apply the same search strategy (or Boolean string) to search multiple databases. Of course, this may not matter much to your ‘average web searcher’, but for some professions, this is a big thing. Now the point isn’t so much about Google (or Bing for the matter), that just happens to be the instance we have used to illustrate the concept. And crucially, the point isn’t about query languages either (in the programmatic sense) – important though they are, converting a user’s information need into a API call is a different problem.

Instead, what we’re contemplating here is the prospect of a universal framework for information needs. 

Read Full Post »

BS in, wisdom out

Over the last few weeks I’ve been sharing various ideas about ways in which techniques from the field of data visualisation can be applied to help solve complex search problems, with particular focus on the process of query formulation. In those posts, we’ve discussed the scientific (and, one might argue, commercial) rationale for adopting such techniques in the development of future search strategies.

But what we haven’t really considered thus far is legacy content – in particular, the many archive collections of solutions to common search problems that are stored as curated collections of Boolean strings and search filters. These repositories offer a vital source of inspiration and guidance and play a key role in the propagation of knowledge and best practice for a variety of professions.

Read Full Post »

Field tags, search, and universal formalisms

One small step for an app, one giant leap for advanced search… well, maybe not. But introducing field tags on 2dSearch is potentially more significant than it sounds, so I hope you’ll indulge my reflection for a moment here.

Superficially, we’ve added a way for user to articulate commonly used search operators such as:

• ‘site:linkedin.com’ (i.e. search for webpages from the site ‘linkedin.com’)
• ‘intitle:profile’ (i.e. return pages that have ‘profile’ in the title).

This would be rendered in the UI as follows:

So far, so simple. But what we’re doing is actually much more than that: we are providing a generic mechanism to differentiate between content and metadata, and have the system interpret that construction according to the semantics of the underlying database. This has far broader implications.

Read Full Post »