Ted Carroll's Corporate Search Blog

This post is a response from “guest blogger”  Oli Parker to my earlier post on Search & Collaboration Networks.  Oli Parker runs DeepGreen Consulting; an information management consultancy with a particular interest in ECM systems and collaboration. He can be contacted at ogp@deep-green.co.uk.

You’ve put your finger on a number of commonly-recurring issues with EDRM systems; issues which the EDRM (or ECM as it’s now called) community would do well to address.

However I think that we need to lay out some territory for the discussion before doing the addressing.  While it is easy to list the problems with certain approaches, it is also worth looking at what they offer.  In fact, before doing that, perhaps we should look at what is trying to be achieved – to find some requirements and investigate the way in which they can best be met.

Differing Requirements

The main distinction we need to make is between documents and records.  Or putting it another way, are we setting up a system in which in to store records (bearing in mind the various definitions and particularly the importance of records) or a system in which information more generally will be stored?  These are two significantly different requirements which will lead to quite different systems if addressed comprehensively.  (Yes, there may be a requirement for a system that does both of these, more later.)

If records are to be stored, a more formal approach needs to be taken.  Records bring with them retention schemes and access control issues.  Applying these is something to be taken seriously; often these are backed up by legislation which it is wise to stay on the correct side of, and sensitivities which need to be respected.  Records ‘belong’ to a corporate body, and the corporate body has a duty of care for them.  The internal structure of the corporate body may (and often does) change, and the storage system needs to be robust in such a circumstance.  Equally, the corporate body may split, with functional parts moving to another organisation, or parts of other organisations joining. (Machinery of Government changes are the public-sector equivalent of changes of ownership of companies or brands in the private sector).  The change in ownership of records in these circumstances needs to be handled efficiently, with as little administrative overhead as possible.

CFP’s are Good! (Honestly)

And these are the reasons for a Corporate File Plan (CFP).  By keeping records relating to a certain function, activity or transaction together, you can readily apply both access permissions and retention rules to them all, as access and retention are usually boundaried by the limits of a given activity or function.  By dividing records by function, you can re-organise the teams within your corporate body as much as you like without needing to change your record storage.  You can also very readily pass a particular function (perhaps that of regulating financial markets or of issuing licences for exhuming dead bodies) to another organisation, and by ‘snipping off’ the CFP at the relevant point, you can pass on all the relevant records pertaining to this function at the same time.

These are all good reasons for applying the rigour of a CFP to your corporate record storage.  Without structured organisation of records such as that offered by a CFP you may gain apparent short-term benefits (such as it being easier to find a ‘suitable’ place to save items), but in the longer-term you will find managing records to be much more difficult because – simply put – they aren’t organised.  Trying to find all the HR records relating to an employee who left 7 years ago when no effort has been made to keep them in the same place will be an exercise akin to trying to find a complete pack of 52 playing cards for a game of poker when they are not all kept together.  You can’t play the game with an incomplete pack, but while the appeal of both exercises wears off after only a few minutes of searching, the game of poker can be readily abandoned; the other can’t.

The benefits of a CFP are therefore clear.  But let’s not confuse a CFP with an ECM (EDRM) system.  An ECM system is simply a set of tools which automate a number of the more laborious aspects of record-keeping;  tools which work on a collection of records, in whatever structure they inhabit.  These tools are primarily concerned with security, retention and access, although a mature ECM tool will offer a raft of other functionality as well.

It is important that the storage structure is seen as a separate entity to the collection of ECM tools.  All too often, the design of a CFP and the implementation of an ECM system are seen as being two parts of the same project;  a view which often gets them dangerously confused in the eyes of the user.  They should be introduced and presented to the user separately, in order that their separate purposes are clear.  A CFP can be used to store information regardless of medium or system; in an ideal world, the CFP would be used to structure the storage of both electronic and hard-copy media.  It should be used to manage eMails as well as ‘Office’ items (why would you want to use a different structure for eMails?), for microfiched as well as paper-based items.  A CFP could (and should) out-last an ECM system by several generations; if an ECM is correctly seen as merely the tools that are available to manage items within the CFP, changing the tools for a new – updated – set should be readily possible, without any need to change the underlying structure of the information being managed.

CFPs therefore offer useful benefits which are hard to ignore.  Users need to understand the purpose of a CFP and the reason for using one, in much the same way that they need to understand the purposes of and reasons for keeping records (all of which are commonly dismissed as ‘just filing’; an attitude that is as unhelpful as it is short-sighted and too often badly overlooked).  However CFPs are also difficult to get used to and to use; people don’t think in terms of functional structure when either saving or looking for an information item. (They think in terms of areas of work that they are responsible for, or that their team are responsible for, and tend to file accordingly.)  For precisely this reason, users should not be presented with a CFP without also being shown a mechanism for easily navigating to the parts of the structure which they will most frequently use.  In the same way that a sales employee has no need to understand the structure of the HR department,  so such an employee need not know the detail of the HR function of the CFP;  that both exist is all they need to be aware of.

Nonetheless, the aforementioned sales employee may need ready and quick access to a number of apparently disparate parts of the CFP;  the sales and marketing area, the product development area and their own personnel file, perhaps.  For this reason, it is important for them to be able to (and to understand how to) set up shortcuts to these parts of the CFP relevant to them, such that in their regular work they need only access these shortcuts and not concern themselves with the rest of the fileplan; a fileplan which is not ‘theirs’  (it belongs to the corporate Records Management department) and over which they have no control. They must then be able to arrange these shortcuts in whatever way they please and thus can create their own personal fileplan (of shortcuts), for their own use.  This personal fileplan needs to be available to them whenever they access the record store – either for retrieving or saving items.  Once created, these shortcuts should be persistent – if created correctly there is no reason why they should ever lose that which they point at.

When a CFP is less useful

However for the majority of information held by most organisations this is still a bit tedious, and focuses heavily on the storage of items.  It hasn’t addressed the central concern of information retrieval and collaboration.

CFPs are useful for keeping records, but such rigour isn’t necessary for the vast majority of the information held by organisations.  Notes, papers, drafts, discussion documents, articles and blogs are all very useful information artefacts, but do not need the degree of management offered by a CFP.  They are not usually subject to a retention schedule or access restrictions as Records are.  Such items need a gentler management touch; they can be held in such a way as to be readily searched and retrieved, such that the intellectual value contained within them can be readily capitalised upon.

An information environment that maximises the utility of such items will, by it’s nature, be a lighter-weight and faster-moving affair.  Storing items needs to be quick and easy, and yet they need to be readily retrieved when necessary. This presents an interesting challenge in terms of tagging and indexing, but such an environment can be designed around a looser structure, using a different set of tools (most notably search) to retrieve items.  Given that items within such an environment are more likely to be used for research (as opposed to reference), the use of search as a tool to navigate them becomes more appropriate.

Search is, by its nature, a processor-intensive task, and as Moore’s Law tells us that processing will get ever cheaper with time, we can assume that search tools will become more useful. Your article outlines a number of areas where search has a large amount to offer, and we can expect to see good progress on a number of these fronts in the near future.

Then again …

However, you also talk about a number of pre-requisites for the operation of a search tool, as follows:

“For search to work across a collaboration network,  then there is a requirement to:

• apply appropriate access controls to collaboration areas within the network
• search across all collaboration areas (subject to access controls) in the network
• provide consistent metadata across all collaboration areas in the network (to allow search to work accurately)
• actively manage the long term retention of content i.e. close down collaboration areas that are not being used and migrate content to archive areas
• provide governance in the setting up, management and disposal of collaboration areas (to avoid the situation where the number of collaboration areas grow in an uncontrolled manner or individual collaboration areas to not conform to the standards of structure or metadata) “

I wholeheartedly agree with these points, but would suggest that many of them are simply re-stating the same needs outlined earlier, namely those of access and retention; needs which were met by a CFP.  While I would agree that the rigour of a CFP is not necessary for collaborative systems, it would appear that there is a need for some form of structure to underlie the information holdings. Indeed, you mention such a need in an earlier stage of your article – “It makes sense that an EDRM system needs a folder and file structure at a basic level.”
So, what are we saying?  That there is a need for some form of structure in which to hold information artefacts; a structure that allows management of these artefacts.  Different artefacts need different degrees of rigour in management, with those being held as records demanding more rigour than those that are not.

Steps to Improvement

Given these differing needs of governance for different forms of information, what can be done corporately? A modern organisation will need both records management as well as information management, so how is this to be achieved? One-size-fits-all doesn’t work, as you have pointed out; you either find yourself applying too much rigour to information, or lack the management capabilities for records.  What is the solution?

I’d suggest a two-pronged approach. Recognise the differences, and use systems accordingly.

1. Prong one would be a CFP for corporate records, with an ECM system to manage the contents.
2. You could then implement this alongside a less-structured system (Prong two). Perhaps use some (relatively) unstructured nodes within Microsoft SharePoint (much loved by users for its support for collaborative working), and encourage users to do their collaboration and thinking using this system.

Aforementioned users should need to know enough about the importance of record management to understand the differences between the two, and to save discussions from Sharepoint to the ECM as and when they become records.  That sounds hopeful.  And you could also then run an enterprise-wide search tool to retrieve from either (or both) system.

Conclusions

You’re right.  Trying to persuade users to use a CFP when it’s not necessary will cause users problems, and introduce a barrier to the adoption of an ECM system.  And this is perhaps where many an ECM implementation has foundered; trying to use the wrong tool for the job is always a bad idea.

However, the converse is also true.  Trying to manage important, lasting records without the correct degree of governance and control is also going to cause problems – of a different form.  Again, the correct tools are needed for the job in hand, and the CFP is a very useful tool to use in the management of records.

The right way forward is to understand what is trying to be achieved (management of records or management of information), and building the system accordingly.  Such an understanding needs to be agreed by everyone up-front – ‘everyone’ including system owners, project sponsors, records management departments, systems integrators and users alike.
Only when such understanding and agreement is reached can an ECM implementation be successful. To try and proceed with any project without this agreement is to embark on a project which has – at best – a limited future usefulness.

Introduction

There is a  belief within the document and records management community that a comprehensive and detailed Corporate File Plan (CFP) is essential for the successful deployment of an EDRM (Electronic Document and Records Management) or ECM (Enterprise Content Management) project. The question posed in this post is “does a combination of search and team-based collaboration solutions provide a more efficient and effective way of managing files or documents  in large organisations?”

This article outlines how an over-complex Corporate File Plan may actually prove to be a barrier to EDRM / ECM deployment and, more importantly, a barrier to information and knowledge management within large organisations.   A new approach, utilising search and other technologies, in a “collaboration network”, is proposed.

The Corporate File Plan Approach

Folder and sub folder functionality appeared within the first generation of document management systems (e.g. Saros, Documentum) and quickly became a prerequisite for any product in the EDRM space.   Interestingly, PC Docs, (now Hummingbird / Open Text), probably the most successfully deployed early EDRM system, did not use a folder and sub folder approach.  Users of PC Docs entered properties identifying the author, subject, title, project number and various other values, and the document was filed away and could be retrieved with any combination of the appropriate properties.  The location of a file (and its name) was not required. ( It could be argued that one of the key reasons for the success of PC Docs implementations was the simplicity of filing and retrieving documents.)

The folder and sub folder structure had some successful uses in department-based document management applications in the private sector.  However, the setting of electronic records targets by governments across the world led to the adoption of the “Corporate File Plan” as the de-facto approach for managing large document collections, based on a functional “Business Classification Scheme”.   As a result, the starting point for many EDRM programmes has been the development of the Corporate File Plan -  an all- encompassing hierarchical set of folders and sub folders into which any user could store material and find it again.

In many cases, the process of creating the Corporate File Plan has grown into a large project in its own right,  lasting years in some cases, delaying the implementation of EDRM, and eroding the benefits of new technology and new ways of working.  (Projects have often suffered from changes in organisation responsibility midway, resulting in extensive reworking.)  It makes sense that an EDRM system needs a folder and file structure at a basic level, but I would argue that many projects have gone way beyond this,and have developed products that are too complex to understand by the people who need to understand them – the users.

Experience shows that file plans work best where there is already a logical folder structure for information (which is understood by users), with consistent naming convention for sub-folders, and the logical structure is related to repeatable processes.

Examples include:

• information to take to a law court within a “Case File”
• insurance claim file
• a patient health record file.

Most users experience has been that file plans are not as effective for information where the processes of managing, creating and reviewing information are less repeatable than the examples shown above.

Corporate File Plan Issues

Frequent complaints from users regarding EDRM systems are:

• “It takes too long to save a document”
• “I can’t remember where I put  the document -  it in the new file plan!”
• “The file plan is too complicated”
• “I have to enter too much metadata!”

One of the challenges of the folder / sub folder model is that there is too much choice  – do you set up a new sub folder for this document?  Where do you put the document in the current structure?  Should you add descriptive information to the folder, file name or a property in a document? What keywords  and other metadata should be used?

File Plan issues can be summed up as:

• File plans slow down the process of saving and accessing documents, through having to navigate complex folder hierarchies (each time the user opens or saves files).  The “Save As” and “Open” dialogs are particularly unwieldy when navigating up and down multiple areas of file plans.
• For anything but repeatable processes with very rigid folder structure, users find it difficult to understand and remember the file plan structure and where to file information.  The Function / Activity / Transactions approach (often recommended to organisations)  is obscure as the basis for file plan structure, particular for strategic rather than operational information.
• It is easy to misfile a document, either through misunderstanding the file plan structure, or simply by making a mistake when using the Save dialog.
• File plans lead to accidental duplication of content – it is too time consuming to check if a document has already been saved elsewhere (due to the lack of functionality within the “Save As” dialog), so users create a new copy just in case – for instance, when a number of recipients have been sent the same attachment.
• File plans are not “email-friendly” – we often work off a different set of filing structures within email systems.
• Metadata is spread across a number of containers e.g. disk name, folder & sub folder names, folder and sub folder metadata, document metadata.  This makes search ineffective, as we don’t know which metadata fields to search (and whether to search folders or documents).
• Lack of easy to use facilities to reference a file in more than one location  at once (i.e. shortcuts are hard to set up and don’t get automatically updated when the original is moved).
• It is time consuming to agree the Corporate File Plan and challenging to keep up to date as the organisation and its information evolves.

It is remarkable, given the accepted wisdom of the use of file plans, how little research has been carried out on the usefulness of the file plan for information and knowledge management, and the lack of information on the benefits achieved in major programmes.  Anecdotal evidence shows that the file plan is often a major barrier to adoption of EDRM / ECM solutions, rather than a help.   Where are the case studies or evidence that this is a suitable approach for managing corporate information?

The Role of Search

It is also worth considering the changes in technology over the last five years that point towards search as an effective method for accessing corporate information.

• Search is increasingly popular with users (and understood by users), due to the growing acceptance of Google and other internet search engines.  Evidence from web usage shows that the majority of navigational access of web sites is via search, rather than menus. (Little information is available on Intranet or EDRM use, but it may show similar user support for search.)
• Increased computing power and low cost search products makes search-based solutions affordable, and effective  e.g,. Google Search Appliance, Lucene
• Increasing use of mobile devices (with small screens) make it less practical to display a complex multi-level file plan.
• Increasing amount of information is not held within discrete files e.g. “Web 2.0” content, such as wikis, blogs, and email content, where file plan structures are not applied, so search is the main route to find this information.

However,  while search is an attractive option, poor quality descriptive information within document properties (i.e. the metadata) would mean poor quality search results also.

Search and The Collaboration Network

If we downgrade the significance of the Corporate File Plan, what do you replace it with? One option is to simply base document collections around team working and collaboration, in the way that collaboration tools such as Lotus Notes and Microsoft SharePoint allow separate collections to be set up.  I am calling this a “Collaboration Network”.

This could comprise collaboration sites, (including “Web 2.0” content) and also legacy EDRM stores, even from multiple vendors. Search inevitably forms a key capability of the “Collaboration Network”.  Search allows additional “document context” views to be developed.  For instance:

• What’s new
• Recently accessed documents (for a user/team/organisation)
• Documents created or accessed by a user
• Team documents
• Saved searches (to retrieve commonly used documents against more complex criteria)

Search has a number of advantages over the Corporate File Plan approach. Search can:

• be faster to find documents – assuming the metadata is present – particularly for very large collections
• find specific documents or collections in many different combinations i.e. via name or combinations of other metadata.  The Corporate File Plan is not as effective for presenting corporate information where there are many different views of information
• be tuned / customised for common search requests – i.e. if many people want a particular form, then a “best bets” approach can be adopted
• deliver a “serendipity effect”  -  users are more likely to find information they did not know existed through search, compared to browsing folders.

For search to work across a collaboration network,  then there is a requirement to:

• apply appropriate access controls to collaboration areas within the network
• search across all collaboration areas (subject to access controls) in the network
• provide consistent metadata across all collaboration areas in the network (to allow search to work accurately)
• actively manage the long term retention of content i.e. close down collaboration areas that are not being used and migrate content to archive areas
• provide governance in the setting up, management and disposal of collaboration areas (to avoid the situation where the number of collaboration areas grow in an uncontrolled manner or individual collaboration areas to not conform to the standards of structure or metadata)

Enhancing The Collaboration Network

Search is an essential component  but what techniques can improve the effectiveness of the Collaboration Network, by ensuring high quality search results?   There is a wide range of potentially useful technology out there that could be used to improve the quality of content in a  collaboration network, using an “active content management” approach – compared to the “fire and forget” approach of current document and records management solutions.

Some of these technologies will include:

• Metadata Analytics – it is possible to use techniques to look for missing information (including metadata, summaries used in search snippets, titles etc.) and either add automatically or flag the omission to content owners.  Additionally, it is possible to add automatically based on other documents with similar properties (or earlier versions of the same document).
• De-duplication – we can replace duplicate copies of content with links to a single master copy.  (Some hardware and software solutions do this already).  A system could prevent two files being created with the same name – the Corporate File Plan allows two files to have the same name but in different folders.
• Semantic Analysis – many tools exist to automatically generate metadata – including those that look for proper names, scientific terms or terms from a supplied  business thesaurus.  It is possible to use this analysis to auto-populate missing data.
• Linguistic Support – we can use glossaries, thesauri, taxonomies, ontologies etc. to to help steer the user to the most appropriate content, and help with the process of indexing information.
• User Learning – we can utilise web analytics to identify popular content.  This approach is well established for ecommerce sites such as Amazon i.e. recommendations, drop down suggestion boxes based on users past activity.
• Search Analytics – we can analyse search queries and search results to provide insights into content and feedback these results to content owners – do common searches deliver the best content to users?

And finally, I think the world is ready for a much improved  Open / Save / Save As dialog.  It does not work well with a Corporate File Plan –  the main problem is that when you are in the middle of a Save or Save As, you can’t check if a document does or does not exist, and inevitable duplication takes place.  A redesign of this dialog is needed for the Collaboration Network.

The Collaboration Network approach would involve using a range of tools as shown  above to continuously asses and improve content.  Much of the time consuming work for this can be automated (using technology that has been around for many years), although clearly it would be beneficial to involve the user in the  process i.e. provide the ability to veto or adjust any automated changes to content.  For instance, a user could be prompted to add metadata to a recently created document – using semantic analysis to suggest the most appropriate values from a thesaurus.

Conclusions

Many professionals would concede that the majority of corporate EDRM / ECM deployments have delivered patchy success, if at all.  The deployment of related technologies including collaboration software, email archiving, network drives and “Web 2.0” tools continues in parallel, with significant consequences for knowledge management. Information management programmes should build on team collaboration initiatives to form “collaboration networks”, rather than expend too much effort on developing and deploying a Corporate File Plan and corporate EDRM solutions.

Collaboration networks with search will enable information to be found and reused more efficiently and effectively than a single EDRM solution based on a Corporate File Plan, should it even be possible to achieve. This approach requires a much greater emphasis on search , which, in turn, is reliant on improving the quality of descriptive information associated with content – whether metadata or within the content itself.  We need to use “active content management” and develop processes and techniques to continually assess and improve content quality.. either using fully automated methods or automated assistants to feedback to content owners.