Managing Information

Intelligence operations rely on high quality record keeping and information management systems.

Some intelligence leads are too fragmentary or imprecise to be of immediate value. Even so, small details are important. Collectively they provide the raw material on which assessments are based, against which new intelligence is judged, and from which further investigations can be developed.

Information and records management is a core business activity in MI5. It supports sustained, integrated research and analysis that underpins our work.

Our records include both paper and electronic files. Paper files remain important to MI5. However, we make extensive use of computer systems to index and retrieve records. Electronic documents and data are of key importance to us.

Under the Security Service Act 1989, we may only obtain information where it is needed to carry out our functions. We may only disclose information to fulfil our statutory purposes, to prevent or detect serious crime or for the purpose of any criminal proceedings.

There are special limitations placed on the product of intercepted communications to ensure the safeguards set out in the Investigatory Powers Act 2016 are fully complied with.

A number of further controls limit our and others’ access to particularly sensitive information.

Opening files

MI5 has strict criteria for opening files on a person or organisation. These specify the circumstances in which opening a file and initiating enquiries are justified within the terms of our statutory functions (as defined in the Security Service Act 1989). Files are kept under continual review and are formally checked every year to ensure that they remain up-to-date and relevant.

A small proportion are open for active investigation. The rest are categorised as closed and are scheduled for disposal. Staff are not allowed to access files unless it is necessary to do so in the course of their work and they are specifically authorised to do so.

Retention and disposal of files

MI5 has to take account of a number of potentially conflicting factors when considering whether to keep files that are no longer of current interest.

There are some specific legal requirements. We have a responsibility to provide the Investigatory Powers Commissioner's Office with any details it requires when considering complaints related to our conduct. Relevant records must therefore be retrievable.

File releases to The National Archives

We also comply with the requirements of the Public Records Act 1958 to identify records of historical interest for permanent retention and eventual transfer to The National Archives (previously the Public Record Office). In practice, this means selecting files for retention that would otherwise have been destroyed as obsolete.

In 2001, we agreed with The National Archives criteria for deciding which files to select on historical grounds. These criteria incorporate recommendations made in 1998 by the Lord Chancellor's Advisory Council on Public Records and are published in the Operational Selection Policy on The National Archives website. National Archives staff regularly sample files we have earmarked for destruction, to ensure that the selection policy is being appropriately applied.

To date we have transferred more than 5,000 files to The National Archives. We usually release two tranches of historical records a year.

When considering the release of historical papers, we have to take into account the need to protect former staff and agents. It remains a fundamental principle that we provide a commitment of confidentiality for the identities of such individuals indefinitely. We must also consider whether the release of intelligence records into the public domain will cause substantial distress to an individual and their family.

In order to fulfil our commitment to both confidentiality and privacy, we consult other government departments about sensitive information that might be contained in their releases.

File retention policy

Aside from these specific requirements, the general principle underlying our file retention policy is that we will retain only those records that are necessary for our legal responsibilities. We therefore regularly review the need to retain older records to help us fulfil our functions in the future, such as for re-opening unresolved investigations should new intelligence become available.

We have to strike a balance between the need to keep files that might later have potential historical or intelligence value against the need to ensure that files are not kept unnecessarily, particularly files on those individuals who are no longer a threat to national security.

Between our formation in 1909 and the early 1970s, over 175,000 files were destroyed as obsolete or because of major contractions in the size of MI5, most notably after the two World Wars. This has sometimes caused problems. For example, in the late 1960s we faced difficulties investigating some spy cases because relevant records had been destroyed. We therefore decided to retain records indefinitely.

However, in the early 1990s, following the collapse of Soviet communism and the associated decline in the threat from subversion, the review and disposal process was reinstated. Since then about 200,000 files have been destroyed. Most of the files destroyed had been opened for counter-subversion reasons, and retained because Soviet and Warsaw Pact intelligence services had in the past sought to recruit spies from within certain subversive groups.