ABOUT US
The Security Service has to take account of a number of potentially conflicting factors when considering whether to keep files which are no longer of current interest.
There are some specific legal requirements. First, we have a responsibility to provide the Investigatory Powers Tribunal with any details it requires when considering conduct of the Service in relation to a complaint to the Tribunal; relevant records must therefore be retrievable.
We also comply with the requirements of the Public Records Act 1958 in identifying records of historical interest for permanent retention and eventual transfer to The National Archives (new window) (previously the Public Record Office). In practice, this means selecting files for retention that would otherwise have been destroyed as obsolete.
In 2001, we agreed with The National Archives criteria for deciding which files to select on historical grounds. These criteria incorporate recommendations made in 1998 by the Lord Chancellor's Advisory Council on Public Records (new window). The "Operational Selection Policy (PDF 70Kb - new window)" is available on The National Archives website. National Archives staff sample regularly files we have earmarked for destruction, to ensure that the selection policy is being appropriately applied.
To date we have transferred nearly 4,000 "pieces" to The National Archives (a "piece" is a term used by The National Archives and may represent a whole file or a discrete portion of it). Included in the releases so far are files dating up to 1958. Further tranches of historical records will continue to be released, usually twice a year.
When we consider the release of historical papers, we have to take into account the need to protect former staff and agents. It remains a fundamental principle that we provide a commitment of confidentiality for the identities of such individuals indefinitely. We must also consider whether the release of intelligence records into the public domain will cause substantial distress to an individual and their family.
In order to fulfil our commitment to both confidentiality and privacy, we consult other departments about sensitive information that might be contained in their releases.
Aside from these specific requirements, the general principle underlying our file retention policy is that we will retain only those records that are necessary for our legal responsibilities. We therefore regularly review the need to retain older records to help us fulfil our functions in the future, such as for re-opening unresolved investigations should new intelligence become available.
We have to strike a balance between the need to keep files that might later have potential historical or intelligence value against the need to ensure that files are not kept unnecessarily, particularly files on those individuals who are no longer a threat to national security.
Between our formation in 1909 and the early 1970s, over 175,000 files were destroyed as obsolete or because of major contractions in the Service, most notably after the two World Wars. This has sometimes caused problems, for example in the late 1960s when we faced difficulties investigating some spy cases because relevant records had been destroyed. We therefore decided to retain records indefinitely.
However, in the early 1990s, following the collapse of Soviet communism and the associated decline in the threat from subversion, the review and destruction process was reinstated. Since then about 200,000 files have been destroyed. Most of the files destroyed had been opened for counter-subversion reasons, and retained because Soviet and Warsaw Pact intelligence services had in the past sought to recruit spies from within certain subversive groups.
The Cabinet Office established the Advisory Group on Security and Intelligence Records (new window) in 2004. The Group includes representatives from the three intelligence agencies, the Foreign and Commonwealth Office, Home Office, Ministry of Defence, The National Archives and members of the academic community. Its remit is to:
"facilitate scholarly development and use, by historians and other researchers, of security and intelligence records that are already available, and to facilitate and identify other security and intelligence records which might be made publicly available".
Further details about the Group are available on the Cabinet Office website (new window).