Managing Information

Information and records management

Intelligence operations rely on high quality record keeping and information management. 

The intelligence we gather can be too fragmentary or imprecise to be of immediate value on its own. But small details can be important when brought together, and historic information can play a role in current investigations. 

This is why high quality information and records management is at the core of MI5’s work and supports sustained, integrated research and analysis that underpins our investigations. 

Under the Security Service Act 1989, we may only obtain information where it is needed to carry out our functions. We may only disclose information to fulfil our statutory purposes, to prevent or detect serious crime or for the purpose of any criminal proceedings. 

Even within MI5 there are controls which limit our and others’ accesses to particularly sensitive information. 

Opening investigations and new records

MI5 has strict criteria for investigating a person or organisation in line with our statutory functions as defined in the Security Service Act 1989. Our records are kept under continual review and are formally checked at least every year to ensure they remain up-to-date and relevant. 

A small proportion are open for active investigation and the remainder are categorised as closed and scheduled for disposal. Staff are not allowed to access records unless it is necessary to do so in the course of their work and they are specifically authorised to do so. 

Retention and disposal of files 

MI5 has to take account of a number of potentially conflicting factors when considering whether to keep files that are no longer of current interest. 

There are some specific legal requirements. We have a responsibility to provide the Investigatory Powers Tribunal with any details it requires when considering complaints related to our conduct. Relevant records must therefore be retrievable. 

File releases to The National Archives 

We also comply with the requirements of the Public Records Act 1958 to identify records of historical interest for permanent retention and eventual transfer to The National Archives (previously the Public Record Office). In practice, this means selecting files for retention that would otherwise have been destroyed as obsolete. 

In 2001, we agreed with The National Archives criteria for deciding which files to select on historical grounds. These criteria incorporate recommendations made in 1998 by the Lord Chancellor's Advisory Council on Public Records and are published in the Operational Selection Policy on The National Archives website. National Archives staff regularly sample files we have earmarked for destruction, to ensure that the selection policy is being appropriately applied. 

To date we have transferred thousands of files to The National Archives. 

When considering the release of historical papers, we have to take into account the need to protect former staff and agents. It remains a fundamental principle that we provide a commitment of confidentiality for the identities of such individuals indefinitely. We must also consider whether the release of intelligence records into the public domain will cause substantial distress to an individual and their family. 

In order to fulfil our commitment to both confidentiality and privacy, we consult other government departments about sensitive information that might be contained in their releases. 

File retention policy 

Aside from these specific requirements, the general principle underlying our file retention policy is that we will retain only those records that are necessary for our legal responsibilities. We therefore regularly review the need to retain older records to help us fulfil our functions in the future, such as for re-opening unresolved investigations should new intelligence become available. 

We have to strike a balance between the need to keep files that might later have potential intelligence value against the need to ensure that files are not kept unnecessarily, particularly files on those individuals who are no longer a threat to national security. 

Between our formation in 1909 and the early 1970s, over 175,000 files were destroyed as obsolete or because of major contractions in the size of MI5, most notably after the two World Wars. This has sometimes caused problems. For example, in the late 1960s we faced difficulties investigating some spy cases because relevant records had been destroyed. We therefore decided at that point to retain records indefinitely. 

However, in the early 1990s, following the collapse of Soviet communism and the associated decline in the threat from subversion, the review and disposal process was reinstated. Since then, about 200,000 files have been destroyed. Most of the files destroyed had been opened for counter-subversion reasons and retained because Soviet and Warsaw Pact intelligence services had previously sought to recruit spies from within certain subversive groups.