An interesting podcast from ITO America on Maximizing the Effectiveness of Enterprise Search – in interview with Matthew Glotzbach, Director of Product Management at Google Enterprise.
One of the points the interviewee raises is how user’s can be given the capability to promote specific search results – normally it is the search / site adminstrator’s role to identify the “best bets”, but Google feel that this is something that users within an enterprise can participate in. The challenge is making such a feature quick and easy to use.
However, there are obvious benefits – it is often the case with Inranets that the administrator is not knowledgable on the subject matter, so allowing users to do this may lead to better results.
A useful case study from IDC. (You have to register with Coveo to get a copy.) The company, Hayley and Aldrich, have estimated benefits from an improved search platform. To quote the report “… even with no change in individual search usage, H&A has saved a total of 39,931 hours in the first year. Based on an average work week of 45 hours, this translates to savings of $4.1 million a year, or the equivalent of 15–17 full-time employees that H&A did not have to hire and train to complete the number of projects that it finished.”
I think the approach that H & A took is the way that benefits should really be calculated – look at some real processes and measure how much time is saved by applying new technology (or other changes), and then take this information and extrapolate across the organisation. However, it is rarely attempted.
As an example, “find electronic files” was reduced from 2 minutes to 1.5 minutes. Another task “Reconstruct cases from closed files” was estimated as taking 270 minutes before implementing a new search platform and 3 minutes after implementation.
Unfortunately, the brief summary report does not provide sufficient detail to understand how many users were involved in the benefits evaluation process. Savings of $4 million per annum sound impressive, so it would have been useful to know the number of staff that are represented by this figure.
While the search facility on web sites is usually prominently displayed – some web sites have gone further and promoted search as the default method of navigating a site.
- Lancashire County Council – the home page consists of a search box and a link to “switch to classic view” and little else on the page.
- Westminster City Council - the search box pretty much fills the screen, although there are a number of links to popular tasks immediately underneath the search box, and a long way further down the page there are links to services by category.
Clearly this approach raises the stakes on search quality – if a site is expecting a higher proportion of users to navigate via search, then the content owners need to keep a close eye on search results on a fairly regular basis – assuming content is changing.
It will be interesting to see if this approach spreads further – and it would be interesting to see the impact of this radical change on search usage and overall website satisfaction for these sites.
Some of the main points to emerge from the search discussions were:
- attendees felt that search was an important and well-used capability on the web site. Estimates of the proportion of users made use of search during a session ranged between 39% and 65%.
- the importance of training content editors to “write for the web” was raised, and in particular, providing an understanding on how content and metadata can be tailored to ensure that content can be found using a search engines.
- the Local Government Service List (LGSL) – a standard list of UK local authority citizen facing services, and used by many councils to control metadata – is not effective for indexing content on citizen-facing websites, as it does not use terminology used by the citizen.
- initiatives to improve search continue to be hampered by the lack of effective search analytics.
- the lack of integration between the web channel and call centres was raised. For instance, many councils do not provide access to the web site for their call centre staff (so there is limited opportunity to divert callers to the web site).
- a council has seasonal search topics that need to be planned for – for instance – “wasps” becomes a popular search term in summer months, only.
- some of the material (e.g. council minutes) is provided and therefore published on web sites in PDF format (without HTML “landing pages”). This can cause problems for both internal search engines and web search engines. In the case of web search engines, old PDF content is often cached when a replacement PDF file is provided on the website.
Time Savings For Internet-Facing Sites
In the previous blog postings, I outlined a method of calculating the time wasted through poor quality search results – the examples assumed that the time wasted was for employees – rather than users of public-facing websites.
We can extend the benefits model for information-based external websites (which are not directly selling something) using similar principles.
For instance, a government information site has the following objectives:
- avoid wasting the citizen, business person or tourist’s time when finding the correct information (and by doing this, persuading the user that taxpayers money is being well spend)
- provide the information required in order that the citizen does not use more expensive channels (e.g. in person calls, phone, email, post).
For a commercial web site, the above two objectives still apply, although there are additional benefits:
- by providing the information to potential customers, there is a greater chance that they become or continue to be customers.
Many e-commerce sites make extensive use of search to find the right products for the user. An e-commerce site has its own measures, based on sales, so it is not necessary to develop an business benefits model for this scenario – i.e. the better search is, the more products are sold via the eCommerce site.
However, it is useful to develop a model that addresses the first two objectives, particularly for sites that are not directly selling products online.
Time Wasted for Citizens and Business Users
Assume that the usage of the site is a combination of:
- citizens (or potential customers/tourists) acting in their own capacity, and
- employees of businesses
We can estimate the time wasted, as we have for employee time wasted. Unfortunately, time wasted for citizens has a fairly low value and is a “non-cashable” benefit as far as the website provider is concerned.
What figure should be used for time wasted for citizens or employees searching in work time? One approach that can be used is to take a similar approach to that used for calculating time savings for transport projects. For transport projects , time savings are given a value based on either:
- Journey to / from work. If a project save a citizen’s time, it will not be used for productive activity. The savings provide more leisure time, which has a relatively low value. Transport projects in the UK use a value of approximately £4 per hour for time time savings in this category.
- Work time – If the traveller is an employee of a company or organisation and travelling during work time, then the salary and other costs are used to calculate the value of time savings. By reducing the time wasted for employees searching for information, we are contributing to the efficiency of the business, and across the economy as a whole, we are improving overall productivity.
We could take a similar approach to estimating time wasted on, for instance, a government information web site. To calculate the benefits this, it is necessary to understand the split between citizen and business use to a web site. In the UK, we could assume an hourly rate for citizens of £4 per hour and an hourly rate for business users similar to that used for employee time savings. (The estimates that I have used are £25 per hourin earler examples.) Given this split, and using the formulae shown in the earler posts, we could calculate the benefits from improved search.
As mentioned earlier in this post, these benefits are non-cashable to the organisation. However, indirectly they could lead to increased revenue or reduced operational costs. Part 6 introduces ways in which we could estimate how search contributes to the reduction in the use of higher cost channels.
I am chairing a discussion group on web site search at the Building Perfect Council Websites Conference ’10 – at Olympia, London on the 14th July. Details of the conference programme are here. It looks like a great event for anyone involved in Local Authority websites.
I’d be interested in any thoughts on the important issues relating to search on Local Authority web sites, which I can raise in the discussion group.
The Cost of Repeat Searches
Parts 3 of the benefit model estimated the cost resulting from “no hit” searches i.e. those searches that do not find any pages or documents.
There is another type of wasted search – where a search does produce one or more results – but the results are such that the user carries out another search almost immediately – without viewing any of the results on the SERP (Search Engine Results Page). (If the user does select result to view, then the costs or time wasted associated with this aspect of search are covered in Parts 1 and 2.)
With more advanced logging (than that required for Part 3), it should be possible to identify the quantity of repeat searches. For instance, it would be necessary to identify where a user has carried out a search, not viewed any pages, and then carried out a subsequent search.
The model for resources wasted should include the additional cost of:
- Constructing and running a Repeat search (where the results will not be used)
- Reviewing the results of the Repeat Search (before constructing a successful search)
t = time for a user to construct and run a search (in seconds)
ts = time to scan each result (secs)
nr = number of results on each page
c = equipment cost of running a search (an article estimated that each search on Google costs £)
n0 = number of Repeat Searches
Time cost of a search = (( ts * nr) + t) x n0
Resources cost of search = c x n0
A site is often seen as a success if the number of searches being carried out increases (for instance, after the implementation of new technology). However, if an increase in searches means an increase in repeat searching, then that should clearly not be seen as a desirable change.
I recently contributed an article to the Industry Voices section of the excellent content management news site FierceContentManagement on “Debunking Common Enterprise Search Myths”.
Rather than reproduce it – the link is here.
Honda cars use Google to provide search results on their web site. However, the Google outsourcing option that Honda has selected means that the first results are sponsored links from other car manufacturers. In Bowen Craggs words..
A search for ‘hybrid cars’ produces a set of 239 results from the Honda site. However, these are topped by a Sponsored Links panel consisting of three options, the first and third of which are for Honda rivals (Lexus and BMW respectively). A second Sponsored Links panel runs down the right of the page and features eight matches, all for other car companies and headed by Japanese competitor Toyota.
And Honda are not the only car manufacturer providing advertising for their competitors. I guess it is worth car manufacturers paying a bit extra and removing the sponsored links!
The Cost of “No-Hit” Searches
The model in parts 1 and 2 is based on the “popular searches” and does not take account of searches that user attempted that resulted in “no hits”. In some cases these would have been attempted before arriving at a popular search term. In other cases, a no hit result may lead to the user abandoning the site and / or using another channel to find the information required. (The cost of abandonment and repeat searches will be covered in later post).
The “no-hit” results page could be caused by one of a number of problems:
- a spelling mistake in the search term
- using the incorrect search syntax e.g. use of AND when it is not supported
- using incorrect punctuation (e.g. spaces, hyphens, full stops) – for instance “i Pod” rather than “iPod”
- selecting a combination of words for the search query that is too narrow and leads to zero results (a problem if the search defaults to AND
- there may genuinely be no information for the search provided
- poor operation of a search engine.
If we look at the total number of “no-hit” searches, which is easily done using search logs, then it is possible to estimate the cost to a business or organisation of these.
t = time for a user to construct and run a search in seconds
n0 = number of no hit searches (for a search term)
c = equipment cost of running a search (A widely-reported article in 2009 estimated that each search on Google produced 7grams of CO2.)
Time cost of a search = t x n0 x salary costs
Resources cost of search = c x n0
There are a variety of techniques that can be adopted to reduce the number of “no-hits” results. For instance:
- implementing a synonym capability (or “did you mean?” suggestion) for common misspellings – some search engines can be set up to do this automatically, given a list of synomyns
- add synonyms for common misspellings as content (in metadata)
- providing auto filled drop-down boxes that suggest correct spellings for popular searches
- faceted search to allow the user to narrow down a search – if the list of facets includes the number of results for each facet, then the user will know which criteria will produce a number of results
- configure the search engine to deliver both “all words” and “any words”, with priority given to the “all words” results.
Using the above model, it is possible to measure the number of “no-hits” before and after changes and estimate the benefits.