1 Few countries — if any — have amassed as much experience in counter-insurgency as Great Britain. From the American Revolutionary War to the wars in Afghanistan and Iraq, British forces have conducted counter-insurgency campaigns in almost every part of the world.1 Although not every campaign ended in victory — the American war being a case in point — with few exceptions it is generally conceded that no one does counter-insurgency better than the British.2 Even the historically ‘un-doctrinaire’ British Army has developed a sophisticated doctrine for such campaigns, one that combines some fundamental principles with the flexibility to adapt these to specific local conditions.3

2 The centrality of intelligence to counter-insurgency is now widely acknowledged, but this has not always been the case: not even for the British. Recently published sources, in particular Christopher Andrew’s official history of MI5,4 shed new light on how intelligence gradually emerged as a centerpiece of British counter-insurgency theory and practice mainly in the period since 1945. This article will show that the British experienced a steep learning curve: sound theory and practice were no guarantee of success, since overall victory or defeat was determined largely by local conditions and British political considerations, and some intelligence practices that were effective in overseas conflicts proved problematic when applied in the domestic arena of Northern Ireland. Even so, in that conflict British counter-insurgency theory and practice became ‘intelligence-driven.’ While this article does not attempt to assess British intelligence in the Afghan and Iraq wars, it is clear that previous experience — for better or for worse — will be relevant for those historians who come to study intelligence in these more recent campaigns. But one caveat should be noted here: the primary focus of this study is intelligence at the operational and tactical levels, not at the strategic level, which deserves a study of its own.

3 This article consists of five parts: the first sets the development of British intelligence in historical context; the second examines pre-1945 thinking on intelligence in counter-insurgency; part three discusses how post-war operational experience modified pre-war doctrine through the process of ‘trial and error;’ part four looks at the intersection — and limitations — of theory and practice in the unique conditions of Northern Ireland; and finally, key themes are identified and discussed in the conclusion.

4 The development of British counter-insurgency intelligence practice did not take place in a vacuum. It was a product of its times, experiences, and changes within the institutions of which intelligence was a part.

5 At the end of the nineteenth century, military intelligence was still in its infancy as a formal military function. A strong bias against intelligence prevailed in the British Army: it was regarded as a professional backwater, suitable only for those officers deemed unfit for command. Consequently, intelligence staffs and units were created on an ad hoc basis, often late in a conflict, and then disbanded or run down when the fighting stopped; thus, intelligence support often was insufficient, as it was in the Boer War.5

6 The same thing happened during and after the First World War: rapid expansion was followed by decline. The inter-war era saw the intelligence staffs of all three services starved for funds and people. Even though military intelligence mobilized and expanded rapidly and provided vital information support to decision-makers and commanders during the Second World War, there was strong opposition to proposals to place the army’s Intelligence Corps on a permanent post-war footing. It did not achieve that status until 1957, long after many of Britain’s counter-insurgency campaigns had begun. Driven by necessity, however, training expanded considerably thereafter. By the time the army deployed to Northern Ireland, it had an Intelligence Corps that was still small but was professional and experienced in counter-insurgency. However, Northern Ireland was not its primary focus: army intelligence resources were committed mostly to supporting Britain’s NATO forces in Germany and to command and staff support at headquarters, bases, and the Ministry of Defence (MOD) in Britain.6

7 The secret services had an even less auspicious beginning and an equally tenuous existence prior to the Second World War. Established simultaneously in 1909, the Security Service and the Secret Intelligence Service (MI5 and MI6 respectively) owed their creation to a pervasive sense of imperial frailty and rising German menace. Both trends found popular expression in a spy hysteria manifested most visibly in a wave of spy novels. The actual German spy threat was almost non-existent, but the alarmist mood forced the government to act: MI5 and MI6 were the results. Their early efforts were amateurish in the extreme and professionalism took decades to develop.7 Signals intelligence and cryptanalysis started from almost nothing at the outset of the First World War but, by the end, had become quite effective at decrypting enciphered messages.8 However, like military intelligence, all of the secret services struggled to survive during the inter-war years. The Second World War brought expansion, greater professionalism, and success. The Government Code and Cipher School cracked the German Enigma cipher machine, allowing British and allied commanders to read some German messages. MI5 solidified its reputation with the successful Double-Cross System, which turned and ran German agents against the Abwehr. These two achievements were vital components in allied strategic deception efforts against the Nazi regime and its military leaders.9 As shown later, experience in recruiting and running double agents paid off in Northern Ireland.

8 But just as in the case of military intelligence, counter-insurgency in the colonies was not always the highest priority of the secret services in the post-war era. Rather, it was the Cold War with the Soviet Union.10 Thus, when considering the counter-insurgency intelligence efforts described below and the problems they encountered, it is important to recall the context in which they occurred.

9 First published in 1896, Colonel Charles E. Callwell’s treatise, Small Wars: Their Principles and Practise, is widely regarded by historians as the first British counter-insurgency manual.11 Drawing on the nineteenth-century imperial experience (and on the Boer War in later editions), Small Wars was a practical ‘how to’ guide for fighting rural guerrillas. As such, it emphasized the British Army’s strengths: mobility and firepower. Since Callwell did not regard guerrilla warfare as a legitimate form of resistance, he did not discuss the political dimension of such conflicts, but he was prescient in one key respect: the role of intelligence. "In no class of warfare," he wrote, "is a well organized and well served intelligence department more essential than in that against guerrillas."12

10 Historians concur: as Martin Thomas points out, the survival of Britain’s empire depended on intelligence, though its tiny security service often failed to foresee revolt.13 Callwell’s firepower and mobility approach lingered on well into the inter-war era, long after changing attitudes at home and political conditions within the Empire had rendered it anachronistic. His intellectual successor was Major General Sir Charles Gwynn, commandant of the Staff College from 1926 to 1931. In that capacity he wrote a doctrine paper, "Notes on Imperial Policing," which he later turned into a book, Imperial Policing (1934). Like Callwell’s work, Gwynn’s book was reprinted many times over and was standard fare for any staff college graduate, and like Callwell, Gwynn demonstrated no understanding of the political nature of insurgency and favored firepower and mobility.14 To his credit, however, Gwynn acknowledged the need for a unified intelligence effort with military intelligence supporting the police,15 yet he declined to discuss the one case study that would have emphasized counter-insurgency intelligence: Ireland.

11 That Gwynn chose not to examine the campaign in Ireland (1919-21) is hardly surprising: it had been an embarrassing and highly controversial defeat. Noted counter-insurgency historian Thomas Mockaitis claims that in Ireland the British "made every mistake a government beset by insurgency can make."16 The British campaign not only failed but also highlighted the ‘ugly side’ of counter-insurgency — reprisals, in particular.17 It was not an experience that many wished to revisit.

12 Yet, there was much that could have been learned from the Irish experience, had the British Army been inclined to do so. The Irish Republican Army’s (IRA) rural guerrilla warfare campaign resembled some earlier colonial uprisings, but its use of urban terrorism and propaganda added an entirely new dimension that turned an insurrection into an insurgency.18 The British effort to suppress the insurgency had involved a daunting intelligence task. Until 1919, intelligence had been left almost entirely to the Royal Irish Constabulary (RIC). However, the IRA made a concerted effort to isolate and drive the RIC from the countryside and to ‘neutralize’ the British intelligence and policing apparatus in Dublin. In this it was largely successful, perhaps even decisively so.19 Worse still, the RIC tended to work without written records, so when experienced police officers were killed systematically in 1919, its institutional memory died with them. The army had to build a new intelligence system from scratch and it was ill-equipped to do so. Owing to manpower shortages its intelligence resources were spread thin: army headquarters in Ireland had an intelligence staff of only three officers. Even when the army got new intelligence officers, they were untrained for the task and had to juggle intelligence work with other duties. The army was inclined to leave the job to the secret services and was slow to appreciate that intelligence collection was the duty of every soldier.20

13 By mid-1920, the army had come to acknowledge the importance of the task and was reinforcing its intelligence staffs, but coordination problems persisted. There was little effective cooperation between the army and RIC or even within the army itself. In May 1920, at the suggestion of the General Officer Commanding (GOC) in Ireland, General Sir Nevil Macready, the cabinet appointed a Joint Intelligence Chief to supervise the work of all intelligence agencies in Ireland. But the man chosen — Brigadier Ormonde Winter — was not up to the task, and Macready himself undercut and limited Winter’s remit. Still, historian Peter Hart, who has used previously unseen records, argues that before the end of the conflict British intelligence was becoming more effective against the IRA. He concludes that the IRA’s intelligence effort was not an unvarnished success, nor was its British counterpart such an unmitigated failure as the standard narrative has portrayed.21

14 One of the few who recognized the change represented by the Irish war — and the value in studying it — was retired Colonel H. J. Simson, whose 1937 book, British Rule and Rebellion, drew explicitly on that conflict. Writing with the intention of improving British methods against the ongoing Arab revolt in Palestine, Simson accurately described this type of conflict as a fusion of terrorism and propaganda. This strategy, he argued with notable insight, comprised a psychological and political war against the government and a campaign to isolate the police so as to ensure the survival of the subversive insurgent organization. It would also disperse the security forces on defensive duties, denying them the initiative. In stark contrast to Gwynn, Simson asserted that existing army tactics were not designed to fight this kind of war. If the use of martial law to defeat this "sub-war" was not feasible, he recommended appointing a Director of Operations supported by a joint civil/police/military staff to run the campaign. Equally important, he stated that security forces had to give priority to destroying the subversive organization that fostered the insurgency. To do this, they needed more effective intelligence services.22 Simson’s work thus displayed a remarkably prescient comprehension of insurgency and of the role of intelligence in responding to it.

15 Simson’s was a voice in the wilderness: even if studying the Irish campaign could have been instructive, there was no reason at the time to suppose that it had been anything but an aberration. Moreover, the anti-intellectual atmosphere prevailing in the inter-war army discouraged thoughtful inquiry into the critical political, military, and psychological aspects of insurgency and counter-insurgency.23 Thus, to the extent that there was reflection on the campaign in Ireland, the tendency — quite understandably — was to look for answers along familiar lines.24 Hence, officers assigned to internal security duties in Palestine, as the Zionist insurgency was gathering momentum in 1945, were urged to read not Simson, whose work was virtually unknown, but Gwynn’s Imperial Policing: after all, repression had succeeded against the recent Arab revolt.25

16 Likewise, the army’s long-standing discomfort with the intelligence task precluded serious examination of the subject. There was a tendency to define and assign intelligence roles in discrete terms: as a military function in wartime only and a civilian task in peacetime. Not surprisingly, therefore, troops deploying to post-war Palestine were discouraged from undertaking "duties of a detective or secret service nature."26 The problem posed by insurgency and counter-insurgency, however, was that they comprised a grey area that was neither peace nor war: they blurred the lines of responsibility between the military and the civilian authorities. The police were usually seen as the lead agency for intelligence,27 but in many post-war conflicts, they were not up to the task.

17 The British Army entered the post-war era intellectually unprepared for the counter-insurgency intelligence task. Consequently, it was often outclassed by its enemies in this crucial aspect of warfare. The Zionist insurgency in Palestine was a case in point. Its combination of terrorism and propaganda consciously emulated the IRA model, and its political complexity confounded British administrators, policy-makers, military commanders, and the security forces themselves.28

18 To combat the insurgency the British government had at its disposal in Palestine several intelligence organizations, including the Palestine Police Force (PPF) and its Criminal Investigation Department (CID), GSI (military intelligence), Defence Security Officer (MI5 station), MI6 (Secret Intelligence Service), MI9, and the signals intelligence service.29 The army and the PPF had access to the Egypt-based Combined Services Detailed Interrogation Centre (CSDIC).30 The CID’s political branch was meant to be the lead agency for intelligence, but it was under-staffed, overworked, isolated from the Jewish community, and penetrated, compromised, and intimidated by the insurgents;31 thus, it was not, as the PPF’s ‘official history’ claims, "devastatingly effective against terrorism."32 That said, Steven Wagner credits the CID with greater effectiveness than the standard narrative relates. Still, he writes, it was "not enough to deliver a killing blow."33

19 Lieutenant-Colonel The Hon. Martin Charteris, the head of GSI, felt that one of his main tasks was "to make sense for the soldiers out of the tangle of the Palestine problem, so that they may see things in their true perspective."34 Thus, GSI’s "Fortnightly Intelligence Newsletters" contained a high proportion of political intelligence that was often quite perceptive.35 In short, it provided what we now call "White Situational Awareness."

20 Charteris’s intention may have been noble, though it is not clear whether officers or other ranks had either the time or inclination to ponder the complex nuances of Zionist politics as presented in GSI newsletters. Moreover, in devoting time and effort to this task, GSI seemed to lose sight of its primary mission: providing operational intelligence to army formations and units. One thing is clear, however, its intelligence assessments had a rather mixed record of accuracy. For example, newsletter no. 16, issued 9 June 1946, discounted reports that predicted an imminent resumption of terrorism; in fact, it stated there was a "good chance" this would not occur.36 A major insurgent offensive began the next day.37 In fairness to GSI, it depended greatly on the CID for its intelligence and thus it cannot be held wholly responsible for its limitations. Yet, it appears that there was an abdication of professional responsibility, albeit one encouraged by existing doctrine.

21 Army operations, such as patrols, road blocks, observation posts, and snap searches designed to provide security by deterring and disrupting insurgent operations, also fulfilled an intelligence function. They familiarized the troops with their Areas of Operational Responsibility (AORs) and generated a routine flow of background information to the unit, formation, and headquarters intelligence staffs. Divisional, brigade, and district staffs compiled the information into detailed reports for distribution to all units. Divisional headquarters were responsible for issuing ‘warning intelligence’ messages.38 The Army’s Field Security Sections (FSS), of which five were operating in Palestine at any one time, gathered security intelligence to protect army units and bases, but they also were the exception to the army’s ban on "detective or secret service" work. The FSS carried out plain-clothes and disguised intelligence collection patrols, conducted surveillance, and provided personnel to the CSDIC to interrogate captured insurgents.39

22 MI5’s primary concern was the serious Zionist terrorist threat to the UK itself, but its official historian, Christopher Andrew, reports that MI5 had no dedicated counter-terrorism section in 1945. Instead, Zionist terrorism and related Middle East affairs "fell to the interdepartmental SIME (Security Intelligence Middle East)," which had been a mainly military organization in wartime but was transferred to MI5 in December 1946.40 It was SIME, based in Cairo, Egypt, that maintained the Defence Security Officer (DSO) network and liaised with the Intelligence Division of the Security Service in London. The DSO in Palestine, Sir Gyles Isham, had a staff of eight to ten intelligence officers in Jerusalem and four to six Area Security Officers (ASOs) reporting to him from the major urban areas of Palestine, where the most extreme Zionist insurgent groups were based.41

23 The DSO’s monthly intelligence reports on developments in Palestine were comprehensive and insightful, indicating both better collection and higher quality analysis. For example, by contrast with GSI, the DSO accurately predicted the revival of terrorism in June 1946. This record could be explained by the DSO’s access to high-level sources, including leaders within the Jewish Agency, the semi-official ‘shadow government’ of the Jewish community. Many of the DSO staff also had lengthy service in the country and could speak the languages, and thus they were better able to assess the information they collected. The assassination of Major Desmond Doran, the ASO for Tel Aviv, in 1946 lends weight to this view. According to a former Stern Gang member, Doran was killed precisely because he was a threat to them: fluent in Hebrew, a skilled interrogator, and an expert on the group. Steven Wagner, however, offers a cautionary assessment of the DSO’s efforts, noting that high-level sources like Teddy Kolleck sometimes deliberately provided misleading information in order to protect certain insurgent activities.42

24 While there was no single body either directing the intelligence collection effort or collating and analyzing centrally the intelligence gathered from these organizations, there were structures in place to facilitate cooperation between them. Army, police, and DSO staff sat on the central and district security committees that met regularly and served as joint operational planning groups. GSI and the DSO held joint intelligence meetings once or twice per week. Combined army-police intelligence courses were conducted to share knowledge and experience, and joint army-police operations were part of the normal routine. The army, police, and SIME cooperated in the use of the CSDIC,43 but this cooperative atmosphere apparently did not prevail in all operations. Wary of lax PPF security, the army frequently resorted to deceptive measures against the police in order to ensure operational security.44 Consequently, army-police relations were not always harmonious. Security-force operations often were launched with insufficient information and, consequently, only rarely produced significant arrests, prosecutions, or reductions in the level of insurgent activity.45

25 The British defeat in Palestine resulted primarily from political failure. Britain was unable to craft a policy that could accommodate the wishes of all parties contending over Palestine: without that, it had no strategy.46 Its decision to withdraw was also driven by the loss of control over the territory. That was an operational failure, which in turn can be attributed in large measure to defeat in the intelligence battle, which former Irgun leader Menachem Begin called "the decisive battle in the struggle for liberation."47

26 The British Army did learn from its experience in Palestine. The 1949 internal security manual placed considerable emphasis on the need for accurate and timely intelligence. It also stressed the need for an ‘all-source’ approach. Army commanders were urged to consider scrutiny of the intelligence system as their first priority upon taking command. The manual forewarned them to expect deficiencies in the allotment of intelligence staffs, which meant that the army should not try to do the job of the police Special Branch, still regarded as the principal source of operational intelligence. Rather, the manual placed more weight on the need for close army-police cooperation and mutual confidence, to be achieved by frequent intelligence conferences.48 Although this document still saw the army and police intelligence tasks as separate and distinct, it had raised the profile of intelligence in counter-insurgency, giving it greater salience, and had gone some way to prepare the ground intellectually for the more integrated approach that eventually characterized the Malayan and subsequent campaigns.

27 Indeed, by the time that manual appeared, the Malayan Emergency had been underway for a year. From the perspective of British intelligence, it had got off to an inauspicious start. The Malayan Security Service (MSS), then engaged in a turf battle with Security Intelligence Far East (SIFE), had been completely taken by surprise by the premature outbreak of the insurgency in June 1948. Just two days before, John Dalley, head of the MSS, had stated that while the situation was changeable and potentially dangerous, "At the time of writing there is no immediate threat to internal security in Malaya."49 Although aided by the weakness of the insurgent movement and the revival of the Malayan economy following the outbreak of the Korean War,50 the British still took 12 years to defeat the insurgency. Intelligence was central to their victory.

28 After the initial shock and recognition that they knew little about the enemy, the Malayan administration reacted and started reforming the intelligence structure with astonishing speed.51 At the urging of Malcolm Macdonald, the Commissioner General for Southeast Asia, a Combined Intelligence Staff (CIS) was created in July 1948. This integrated MSS, police, and military intelligence at the highest level and required each to report to the Local Security Committee. At the same time, joint intelligence centres (JIC) were co-located with police headquarters at the state level where they received information passed along by army and police units. Then, in November, on the recommendation of heads of MI5 and SIFE, police Special Branch replaced the floundering MSS.52 Still, the intelligence picture improved slowly because of weaknesses in the army and police53 but also because of institutional rivalry: Nicol Gray, the Commissioner of Police, and Sir William Jenkin, the Director of Intelligence, "were not on speaking terms."54 Appointed in August 1950, Jenkin resigned in September 1951. Consequently, as late as 1950 Lieutenant-General Sir Harold Briggs, the Director of Operations, was forced to admit:

29 But it was the ‘Briggs Plan’ that reorganized the structure and direction of the counter-insurgency campaign as a whole and intelligence along with it. He created a joint intelligence advisory committee and the position of Director of Intelligence (DI) to assist the new Federal War Council. The CIS, which now consisted of members from the civil service, police, army, and RAF, worked under the DI, coordinated intelligence products, and made recommendations to the Director of Operations Committee.56 By the time Briggs’s successor, General Sir Gerald Templer (a former Director of Military Intelligence at the War Office), took over as both High Commissioner and Director of Operations in 1952, the flow and quality of intelligence had improved considerably.

30 Templer claimed that, "The Emergency will be won by our intelligence system — our Special Branch."57 But the Branch needed reform. Jack Morton, previously head of SIFE, became Templer’s DI, and on his advice, it was separated from the CID and focused solely on the counter-insurgency campaign.58 The army seconded 30 special military intelligence officers to work inside the Branch on insurgent order of battle and tactical intelligence, and to serve as a conduit for passing contact intelligence to the army. This freed the Branch, now the main intelligence-producing body, to focus on its primary role: penetration, deception, and manipulation of the insurgent organization. The Branch decided whether surrendered guerrillas, considered the most valuable intelligence source, could be exploited for their immediate operational intelligence value. If so, the local army unit had to be ready to act on that information on short notice by mounting patrols, ambushes, or raids on insurgent camps. This was also facilitated by integrating brigade-and battalion-level intelligence staffs with the police in joint operations rooms.59

31 Reflecting on the campaign more than a decade later, General Richard Clutterbuck concluded that the best commanders in Malaya had been,

31 Although population control may have generated more intelligence,61 from Malaya onward the need for a single director of intelligence and an integrated intelligence effort based on Special Branch became articles of faith in army counter-insurgency doctrine.62

32 Inevitably, given that no subsequent campaign replicated the social, political, or insurgency conditions in Malaya, the application of these ‘articles of faith’ varied considerably and did not yield the same results.63 The campaign in Kenya against the Mau Mau insurgents was notable for its brutality, and in this respect, it was exceptional among Britain’s post-war conflicts. While the British Army generally performed with restraint, it may have used excessive force in the early stages, and locally raised security forces — the police reserve and the home guard — were notorious for their abusive behavior. This, the internment camps, and over 1,000 executions turned Kenya into a police state.64

33 From an intelligence perspective, the initial British response was similar to the early stages of the Malayan Emergency. The Special Branch had been starved for funds for many years65 and was therefore completely unprepared to confront an insurgency. At the outset, the Branch consisted of only four officers and a handful of other ranks, all based in two cities — Nairobi and Mombassa — with none in the rural areas where the insurgency was brewing. In November 1952, a delegation from MI5, headed by the Director General Sir Percy Sillitoe, visited to assess the intelligence situation and make recommendations. It found the Branch "grossly overworked, bogged down in paper, housed in offices . . . impossible from the standpoint of security or normal working conditions. The officers were largely untrained, equipment was lacking and intelligence funds were meagre."66 The head of the Branch had no access to the governor, no mandate to advise him, and no role in assessing the intelligence he received; thus, neither the Internal Security Working Committee nor the governor paid much attention to reports of impending trouble.67 It is hardly surprising, therefore, that the outbreak of the Kikuyu tribe-based Mau Mau revolt caught the British unprepared, just as they had been in Malaya four years earlier.

34 And just as it did in Malaya, the intelligence effort took some time to improve. Following Sillitoe’s visit, the government implemented his recommendations, which were based on the ‘Malaya model.’ It established the Kenya Intelligence Committee (KIC) to oversee collection and assessment. The KIC was chaired by the new Intelligence Advisor to the Governor and included the head of Special Branch, the GOC East Africa Command, and the Secretary for Law and Order:

34 District Intelligence Committees, which coordinated collection in their areas, sent their reports to the provincial committees, which in turn compiled and forwarded regional reports to the KIC.69

35 This system did not slavishly emulate the Malaya example. Although the emergency was declared in 1952, a Director of Intelligence and Security (D/IS) was not appointed until 1955, because officials in Kenya did not expect the conflict to last and their counterparts in London refused to draw parallels between the two wars. Up to that point, police and military intelligence efforts remained largely separate. The army GSO1 (Intelligence) at GHQ reported to the Director of Operations (D/Ops). A military intelligence officer worked with the head of Special Branch, but the latter did not focus on operational issues and did not report to the D/Ops. This was not an effective use of resources, and in July 1955 Lieutenant. General Gerald Lathbury, the new D/Ops, moved the GSO1 and his staff into Special Branch HQ under the new D/IS, who reported to him. This apparently resulted in a marked improvement in intelligence coordination.70

36 In the meantime, other steps had been taken to improve the flow of intelligence. The government had quickly carried out another of Sillitoe’s recommendations: to push Special Branch out to the provincial and district headquarters. Then, in March 1953, it announced the creation of Joint Army Police Operational Intelligence Teams (JAPOIT). Initially, each team of policemen was headed by a Branch officer and included a small number of Kenya Regiment (KR) NCOs, but the JAPOITs’ contribution was inadequate. The head of Special Branch insisted that they focus on political rather than operational intelligence, and while the KR NCOs, drawn from the white settler population, had local knowledge and language skills, they lacked the rank to exert influence and had limited intelligence training. Worse still, they were implicated in some brutal interrogations, so in July 1953 Britain replaced the JAPOITs with Field Intelligence Officers (FIOs). It assigned experienced regular army officers to the province and district Special Branch to re-orient the Branch to focus on collection and assessment operational intelligence.71

37 The FIOs were responsible for collecting information on the Mau Mau in their area and served as liaisons between civilian authorities, police, and the military. Since their "primary task was to create a network of informers and agents," FIOs had to become "intimately acquainted" with the Mau Mau gangs and their local support infrastructure.72 However, as the FIOs were not professional intelligence officers and did not know the languages, they needed local help. Former KR JAPOIT personnel were assigned to them as Field Intelligence Assistants (FIAs), and loyal Kikuyu tribesmen were used as handlers to recruit and gather information from the informers. Even so, at the outset this system provided only a marginal improvement over earlier efforts.73

38 The capture and interrogation of Mau Mau deputy commander ‘General China’ and the search of Nairobi in Operation Anvil (both in 1954) generated a great deal of intelligence. ‘China’ provided considerable detail about the forces under his command and forced Special Branch to dramatically revise upwards their estimates of the size of the gangs in his area. Although that came as a surprise, he also revealed that his forces were very poorly armed. Likewise, the Nairobi search led to the breakup of the Mau Mau support infrastructure, dealing "the movement a blow from which it never recovered."74 Together, these actions probably constituted major "turning points" in the campaign.75

39 But the Kenya Emergency was also notable for an innovation in intelligence collection and exploitation: the ‘pseudo-gang’ or ‘counter-gang.’ While not invented in Kenya — the concept was tried with unintended, disastrous results in Palestine76 — it was refined to an art form in the African campaign. Then Captain (later General) Frank Kitson, an FIO, first used the collection of extensive information on a Mau Mau gang member to turn him into a double agent, who then returned to his gang and passed along information about them. Later, Kitson used turned insurgents to help pseudo-gangs (made up of loyal Kikuyu and Europeans in Mau Mau disguise) locate and gather intelligence on real Mau Mau gangs and their support networks. Then, guided by an FIA trained in the use of pseudos, the security forces would move in and kill or capture the gangs and their supporters. Finally, once the practice had been officially sanctioned for wider use, the KR began using pseudos for offensive operations. This also reflected an improving intelligence posture. Special Branch was becoming more proficient, and the flow of information was increasing from a population, now concentrated in protected villages, that no longer feared retribution from the gangs.77 The war was effectively over by 1956.

40 Even as the Kenya campaign was winding down, another insurgency was brewing, this time in Cyprus, where the Greek movement EOKA was striving to force Britain to relinquish control so that the island could be reunited with Greece. Britain was opposed to this not only for its own strategic interests but because such a unification would have led to a war between two NATO countries, Greece and Turkey, and because the large Turkish minority on the island would have refused to accept Cypriot union (Enosis) with Greece. EOKA’s campaign, which began in April 1955 and combined rural guerrilla warfare and urban terrorism, resembled the Jewish insurgency in Palestine: its aim was to provoke the British into using excessive violence that could be exploited by EOKA propaganda in order to embarrass Britain and convince the British public that Cyprus was a liability rather than an asset.78 In this, EOKA very nearly succeeded, only to be betrayed by Greek Cypriot political leaders who opted to accept independence instead of Enosis.

41 Just as in Kenya and Malaya, the security forces were caught flat-footed by the outbreak of the insurgency. In the absence of impending trouble, the government had let the police run down, and thus neglected, they were unable to warn the government. In fact, the head of SIME described Cyprus’s Special Branch (which had only been established in September 1954) as a "right royal muddle."79 EOKA aggravated this situation further by deliberately targeting the police and Special Branch in a blatant attempt to blind the British intelligence system. George Grivas, the leader of EOKA, claimed that his group’s attacks "struck a fatal blow against police morale [and] shattered opposition to EOKA among the Greek police," and a British report from November 1955 seems to confirm the poor state of police morale.80 Worse still, the intelligence files were woefully out of date, and British intelligence lacked personnel with Greek language skills.81

42 But in stark contrast to Kenya, the British acted quickly in Cyprus to rectify the deficiencies in intelligence and in their counter-insurgency methods generally. In May 1955, an MI5 officer was seconded to the government of Cyprus to fill the new post of Director of Intelligence (D/I). When Field Marshal Sir John Harding became governor of Cyprus and D/Ops in October, he immediately reorganized the counter-insurgency campaign, creating a Malaya-style committee system. This strengthened the D/I’s ability to coordinate an integrated intelligence effort and produced immediate improvements. The British deployed the full panoply of intelligence skills and resources against EOKA: informers; ‘Q-patrols’ (a virtual carbon-copy of the pseudo-gangs); interrogation — and in some cases torture — of captured insurgents; surveillance; special-forces operations; and document exploitation (including Grivas’s diary, which gave the British an in-depth, albeit ex post facto, understanding of EOKA’s structure and methods). By mid-1956, even Grivas had to concede that the British had gained valuable intelligence about EOKA. They disrupted its courier system, located and raided many of its bases, and captured or killed many senior insurgents, including its second-in-command.82

43 What British intelligence lacked, however, was real-time intelligence and deep-penetration moles within EOKA that would have allowed them to capture Grivas (although they came close several times). They also knew that EOKA had penetrated the police and Special Branch: its agents there allowed EOKA to identify and kill informers and other ‘traitors’ and to tip off the group to impending security-force operations.83 In the end, just as they had done to the insurgents in Malaya, the British defeated EOKA by outmaneuvering them politically. But even if stymied, the counter-insurgency campaign was not wholly in vain and had demonstrated both the benefits and limits of intelligence.

44 Subsequent campaigns tended to reinforce established principles rather than to display great innovation. The ‘Confrontation’ with Indonesia over the fate of Borneo and Malaysia (1963-66) involved British, Malaysian, and Commonwealth efforts to prevent the infiltration of Indonesian guerrillas (and later regular Indonesian troops) into Borneo; thus, it blended conventional defensive operations with now standard counter-insurgency practices. The nature of the campaign, and the deep-jungle terrain in which it took place, dictated a war of patrols that relied on a multi-layered ‘early warning system’ designed to detect infiltrators even before they left Indonesian territory. Cross-border covert surveillance missions by small patrols from the Gurkhas and the Special Air Service (SAS) Regiment located Indonesian bases and observed and reported troop activity and movements. The SAS also provided medical and other forms of help that cultivated a bond of trust with the indigenous people along the border, which allowed the SAS to create a network of local ‘eyes and ears’ essential for tracking Indonesian raiders and demonstrated once again the vital link between winning (or retaining) ‘hearts and minds’ and gaining intelligence.84

45 That last point was driven home in a very negative way in South Arabia just as the Borneo campaign was winding down. In 1963, local Arab nationalists supported by Egypt had launched an insurgent campaign to drive out the British, who faced first a guerrilla war in the rural Radfan and then a terrorist offensive against the British base in the port city of Aden. Unlike the Cyprus and Borneo cases, the British reacted slowly to the insurgency. They did not implement a security-committee system until early 1965 and, owing to local political sensitivities, did not appoint a Director of Operations. Several mitigating factors hindered the development of accurate and timely intelligence on the insurgents. First, the local intelligence structure was too complex and was not reorganized (with the appointment of a Director of Intelligence) until well into the campaign. Second, the British government’s announcement in February 1966 that it would withdraw from Aden by 1968 completely undermined its ability to encourage local allies to cooperate with them. Third, those who might have been tempted to cooperate faced a grim fate once the British left. The insurgents made this abundantly clear by conducting a campaign of assassinations against the Aden Special Branch as well as the population in general. As a result, historian Thomas Mockaitis writes, "Intelligence virtually dried up."85 The insurgents escalated their attacks on the British Army, which had little operational intelligence to counter them. If there were intelligence lessons learned from the campaign, they may have been the wrong ones: it was during the Aden campaign that the army first tried the ‘in-depth interrogation’ techniques that later led to accusations of torture in Northern Ireland.86

46 The British fared much better in the Dhofar campaign (1970-75) in nearby Oman. There, the campaign plan reaffirmed the critical link between gaining the support of the population and receiving intelligence from them; so intelligence was given high priority — the British ran the Omani intelligence service — and a sustained development campaign, supported by skillful Information Operations, created over time a cooperative atmosphere among the Dhofari population. Just as in Malaya, surrendered guerrillas were seen as vital intelligence sources, but unlike in the Cyprus and Aden campaigns, the British used less intense interrogation procedures that nonetheless produced valuable information. Intelligence, thus, was a centerpiece of a carefully orchestrated campaign that by 1975 had yielded Britain’s most clear-cut counter-insurgency victory since Malaya.87

47 Unlike many conflicts that arose during the withdrawal from empire, the outbreak of the ‘Troubles’ in Northern Ireland did not come as a complete surprise. Although it had devoted no more than "one part-time desk officer" to Irish issues, MI5 understood — and had warned the government about — the sources of and prospects for violence in the province in late 1968, a full eight months before the conflict spilled onto the streets. The government began to hold regular meetings on the problem in spring 1969, and the Joint Intelligence Committee (JIC), which until now had given scant attention to the province, established a "Current Intelligence Group on Northern Ireland." The JIC’s June 1969 assessment was as gloomy as MI5’s had been months earlier. In the meantime, the province’s police force, The Royal Ulster Constabulary (RUC), told MI5 that its "Special Branch was overwhelmed" and requested that the Security Service post an officer to the Branch. The officer was in place at RUC headquarters by the end of April. At the same time the Service established a full-time, but small, desk to cover Irish security intelligence. It quickly became very busy, almost buried in information,88 but even being forewarned did not allow the government to head off the seemingly inevitable clashes between the Protestant and Catholic communities in August 1969.

48 In light of what the conflict later became, it is important to remember that, when sectarian violence first brought the British Army onto the streets, there was no insurgency and, thus, no campaign plan to defeat one. Instead, the army’s role initially was akin to peacekeeping — preventing clashes by keeping the two communities apart.89 Within six months, however, two events changed the army’s mission, pushed it into a policing intelligence role, and stood the proven counter-insurgency intelligence model on its head.

49 The first was the marginalization of the RUC. Dominated by Protestants with a deeply ingrained bias against the Catholic minority, it had used excessive force during the year of civil-rights disturbances that culminated in the clashes of August 1969. Its actions had thoroughly discredited the force. The RUC was disarmed in autumn 1969, and the army largely took over the policing function, especially in the Catholic areas where it was welcomed by the besieged population.90 That, in turn, provided the catalyst for the second event: the split in the Irish Republican Army (IRA). Already divided over political strategy, its inability to protect the Catholics during the riots and their positive attitude toward the British Army had shamed the movement. Some were determined to restore the IRA’s pride of place as protectors of Northern Ireland’s Catholics by reasserting its ‘physical force tradition.’ At an ‘extra-ordinary’ meeting of the IRA’s Army Convention in December, the militants broke away to form the Provisional IRA with a commitment to eventually confront the British forces with violence. In the short term, it intended to build its strength and support in the Catholic areas by exploiting clashes with the British forces; its long-term goal was to use enough violence to force Britain to negotiate withdrawal on the PIRA’s terms.91

50 That confrontation took about a year to emerge and another to mature. By winter 1972, Britain was faced with a full-blown insurgency aggravated by political paralysis in London and intransigence in the province, PIRA terrorism, and heavy-handed security-force tactics. Together, they pulled the province into an escalating spiral of violence.

51 When the British Army took the lead in policing, it learned very quickly that it was operating in an intelligence vacuum.92 Policing of the Catholic neighborhoods had collapsed long before the riots of 1969. One army officer observed in 1973 that,

51 Operation Demetrius — the internment of IRA suspects in August 1971 — demonstrated that the police files on the IRA were out of date and inaccurate. Many of those rounded up had nothing to do with the PIRA and had to be released.94 Clearly, if the army was going to contain the growing insurgency, it needed better intelligence, but the army and other agencies would have to build the intelligence picture from scratch by applying the tools that had proven effective in past campaigns. Perhaps more than any previous British counter-insurgency campaign, Operation Banner (the Northern Ireland campaign) eventually became ‘intelligence-led’ and ‘intelligence-driven.’

52 But that did not happen immediately or easily. First, the British government had to resolve a constitutional conundrum: the Ulster legislature could call on and deploy the army in the province but was not answerable to London for how the army was used. That issue was solved in March 1972 by proroguing the legislature and imposing direct rule from London. Second, the GOC was given responsibility for all security operations and for advising the Secretary of State for Northern Ireland (SSNI) on military issues.95 Finally, the newly established Northern Ireland Office (NIO), now responsible for running the province, needed more intelligence. To remedy this, the SSNI (William Whitelaw) appointed a Director and Coordinator of Intelligence (DCI), and an Irish Joint Staff (IJS) was established by MI5 and MI6.96

53 The ‘director’ part of the DCI’s title was, Christopher Andrew says, "a partial misnomer." The DCI did not direct intelligence operations but rather liaised and coordinated tasks that Andrew describes as "difficult and sometimes thankless." Whitelaw saw the DCI as his security advisor and his link to the army and police in the province. The DCI’s staff produced daily intelligence summaries, and the DCI was the conduit for passing Northern Ireland intelligence to the JIC; however, because MI5 was not ready to handle a domestic insurgency, and no one of sufficient seniority in the Security Service was willing to fill the job, it did not take over the DCI position until 1973.97

54 Whether the effort to improve coordination was wholly successful is an open question. Unofficial sources suggest that the various intelligence bodies operated independently of each other until the apparatus was reorganized in the late 1970s. But MI5’s official history claims that by mid-decade the British had "a well-established single machine for processing intelligence and covert work" in the province with the DCI coordinating intelligence operations. With the insurgency in decline at that time, the British had adopted "The Way Ahead" strategy of ‘Police Primacy.’ This was intended — wherever feasible — to put the police back into the forefront of the visible security campaign and to allow the army to lower its profile and concentrate on the ‘hard’ areas, such as the ‘bandit country’ along the border in South Armagh. Behind the scenes, the security forces were applying more sophisticated intelligence-gathering techniques. They were also working more closely together: the RUC Special Branch was rebuilt, they pooled intelligence at the Castlereagh RUC station, and joint army/police/MI5/SAS Tasking and Coordination Groups (TCG) decided on how to exploit it.98 If the police were to retake the lead in the counter-insurgency campaign, they would need to be ‘Intelligence-led.’

55 But at the ‘sharp end’ it was the army, drawing upon decades of its counter-insurgency experience, which had provided many of the intelligence assets and, thus, dominated the process. In the early 1970s, this included many more soldiers trained in intelligence, the secondment of Intelligence Corps personnel to RUC Special Branch, joint Army/RUC operations centers, foot and mobile patrols to familiarize troops with their AORs and to pick up background information, observation posts (OPs), ‘snap’ vehicle check points, an army/police murder investigation squad to probe sectarian killings, camera surveillance, linked intelligence computers, house searches, and the creation of a street-by-street register of the urban population. The latter involved building a card index of known and suspected insurgents, their families, friends, and habits, cross-referenced to reports and intelligence summaries that allowed the army to put together a dossier, including photographs, of any suspected insurgent or supporter.99 This primitive form of Social Network Analysis improved ‘situational awareness’ that could help the troops anticipate trouble and pre-empt it, as well as allowing the security forces to identify, locate, and dismantle the insurgent organizations, such as the PIRA.100

56 Then-Brigadier Frank Kitson, writing during the early stages of the Troubles and drawing upon his experiences in Kenya and elsewhere, observed that "The problem of defeating the enemy consists very largely of finding him."101 Given the rapid growth of the PIRA insurgency, the security forces struggled to identify and find insurgents, and at an early stage, they began to employ some of the techniques that had been used in campaigns overseas, Aden in particular. One of these was ‘in-depth interrogation.’ It was tried on a limited scale in conjunction with internment with mixed results. While some sources allege that it yielded valuable information, the consensus seems to be that whatever gains were achieved were more than offset by the controversy and negative publicity arising from the moral and legal implications of the harsh methods employed. Even army insiders argued that the purpose of interrogation should not be to extract information but to persuade captured insurgents to become informers who would keep the security forces informed from inside the organization.102 As discussed below, this advice was heeded with some success but was not without problems.

57 The army also turned to covert operations. An early example of these was the Mobile [or Military] Reconnaissance Force (MRF), attributed to Kitson’s initiative. Using turned IRA members, the MRF ran a series of ‘front’ companies that collected intelligence while operating in the guise of regular businesses. The best-known operation was the ‘Four Square Laundry,’ which analyzed collected laundry for forensic or other evidence that indicated connections to violent activities. Eventually, its cover was blown by a double agent, the van was ambushed, and the driver killed. This was a serious setback to British intelligence at a time (1972) when terrorism was at its worst, but the exposure of the double agent was also a blow to the PIRA. It showed that the movement could be penetrated and as such was a harbinger of worse to come.103 As the conflict evolved, clandestine intelligence operations became more common and more sophisticated. The SAS was deployed in 1976 into South Armagh, where its teams manned hidden OPs watching the border. In fact, it may have been running plainclothes operations in the province as early as 1974. Since the SAS was small and multi-tasked, it could not do all that was needed over a prolonged period, so it trained selected troops from line battalions to form initially a 120-man Special Reconnaissance Unit (SRU), which deployed in November 1972, and later Close Observation Platoons that could perform covert surveillance tasks.104

58 The story gets murkier (and harder to verify) as one probes the roles of the Intelligence and Security Group (ISG). Tony Geraghty places the ISG within the SAS itself, with 14 Intelligence Company, and the Field [or Force] Reconnaissance Unit and an SAS troop under its command. Mark Urban, however, describes it as an army intelligence ‘executive arm,’ with the SAS and 14IC working under its direction. Operating together or not, and parallel to efforts by RUC Special Branch, they conducted deep cover, long-term surveillance operations and ran their own networks of informers and double agents. Here the British had considerable success recruiting a number of moles within the PIRA and loyalist groups, including the deputy head of the PIRA’s internal security section. He was able to compromise operations and finger members of the group, who might then be arrested or killed, but he could do this only to a limited degree. Had he exploited the full range of his knowledge, he would have exposed himself at the cost of his life, so the very nature of the process was fraught with risk not only to the mole, but also to his or her handler. They had to permit their agents to engage in terrorist actions to maintain their credibility. Several moles run by British intelligence were implicated in murders, having passed intelligence to Protestant paramilitary gangs that allowed them to find and kill suspected PIRA members or sympathizers. TCG operations using informant intelligence yielded impressive results; but probably much greater than their value as intelligence sources was the fact that moles and ‘super grasses’ (former insurgents who testified against others on trial) had a devastating impact on morale and trust within the PIRA and similar groups.105

59 By the 1980s, informers had become the primary source of intelligence for all of the security forces. After a long period of reorganization, training, and capacity-building, RUC Special Branch had taken the lead in recruiting and running agents in the province. In spite of problems, such as running the same agents without telling each other and different views on the purpose of their intelligence operations, the army and the Branch were working more closely together and with MI5 (through both the DCI and IJS Belfast station). The latter was now "wholly funded and mainly staffed" by the Security Service and deployed a range of (still) highly sensitive technical and human sources. The IJS station, however, was stood down in 1984 as MI5 took full control of the operations it had shared with MI6 since the early 1970s.106 The PIRA were very concerned about British technical surveillance efforts and probably with good reason. The security forces had extensive electronic surveillance and communications interception capabilities at their disposal.107 Former RUC Special Branch officers assert that informers were more important sources than the technical systems, but insufficient information is available to either challenge or validate this claim.108 What can be said with certainty is that it would have been inconsistent with historical experience had British intelligence not used its technical systems in concert with human and other means of intelligence collection to identify, locate, and track known or suspected insurgents.

60 As the violence declined in Northern Ireland after 1972, the PIRA stepped up its attacks on the British mainland and against British targets on the continent. Accordingly, the counter-terrorism intelligence task had to expand to encompass those areas and to include tracking arms purchases and shipments from abroad. Just as in Northern Ireland itself, this was a multi-agency (and multi-national) task, with the Metropolitan Police Special Branch taking the lead within mainland Britain until supplanted by MI5 in 1992.109 Once the PIRA decided to end its campaign, MI5 facilitated the initial ‘backchannel’ for ceasefire negotiations.110

61 Counter-insurgency is a multi-faceted, multi-agency mission that entails the integration of security, political, social-economic, and information activities toward the common goal of protecting and winning support of the ‘center of gravity’ (the population). Intelligence influences all of these activities but is probably nowhere more important than in the security domain, which is addressed in this article. If collected, analyzed, and used effectively, intelligence can allow security forces to anticipate trouble and pre-empt it, to discern friend from foe, and, thus, to apply coercive and persuasive measures (kinetic and non-kinetic) with precision and minimal collateral damage — human or material. In this way, intelligence contributes to the effective application of other counter-insurgency actions.

62 That said, as important as intelligence is, the campaigns discussed here showed that by itself it is not a war-winning weapon: it cannot completely offset heavy-handed security operations and/or ineffective information and political programs. In such cases, intelligence can only allow the security forces to impose some degree of order that will buy time for political leaders to make decisions. Nor is there a single, perfect model applicable in all cases.

63 The British have learned this the hard way through trial and error. They applied intelligence resources to every post-war counter-insurgency campaign but not always to good effect. Police forces usually were weak at the outset, and the army was initially unprepared for and resistant to the counter-insurgency intelligence task; consequently, security forces were often surprised and then outperformed — at least at the outset — by the insurgents in this aspect of these campaigns. The fact that army doctrine and then practice in Malaya shifted so quickly to emphasize intelligence was an indicator of significant institutional learning, and further experience generated several key lessons.

64 First is police primacy. In theory, as the security force supposedly closest to the community, the police ought to be the primary intelligence collector. In practice, their role was not so clear cut. The fact that an insurgency or communal violence had developed indicated that policing had failed to some extent. In that case, the military and secret services would have to reinforce the police with personnel and resources until such time as they could retake the lead. In Northern Ireland the army temporarily had to take on the intelligence task itself. By that time, it had gained valuable operational experience in counter-insurgency intelligence and the Intelligence Corps had been professionalized. Earlier campaigns, however, were marked by a greater degree of ad hocracy in this field.

65 Second is direction and coordination. At the outset of these campaigns, the various security forces tended to work independently — collecting, analyzing, and exploiting intelligence in piecemeal fashion and not always toward a common goal. The Malayan experience demonstrated the value of an integrated approach and suggested that a single director of intelligence should be appointed to ensure coordination. For a variety of reasons — political, structural, and personal — this proved harder to achieve in subsequent campaigns, so collaborative efforts often fell short of the ideal, even in Northern Ireland. The 1979 appointment of Sir Maurice Oldfield, former director of MI6, as Security Coordinator for Northern Ireland with a mandate to ‘fix’ army-police cooperation111 suggests that the intelligence machinery was not functioning as smoothly as required. That he, like others in previous campaigns, was able to turn this around points to the significance of personalities — having the right person in the right place at the right time. This places intelligence coordination to some extent at the mercy of fate since no amount of training or planning can ensure that just such a person is available when needed most.

66 Third are sources and methods. To be effective, counter-insurgency intelligence must be an ‘all source’ effort encompassing human sources, signals and sensors, imagery, forensics, money, material, observation, surveillance, and contact. At the ‘sharp end,’ every soldier (and police officer) ought to think of him/herself as a collector and see every contact as a potential source. The post-war campaigns demonstrated the high value of human sources, including captured insurgents, defectors, informers, and moles. The latter, in particular, offer not only the possibility of access to high-grade intelligence but also the potential to manipulate and disrupt the insurgency from within. But human sources are also fraught with problems, from reliability through duplication of source handlers to the methods used to extract information from prisoners. With regard to the latter, the efficacy of torture can be debated, but for democratic states, propriety must take precedence. In wars where the contested ground is the allegiance of the population (both in the war zone and on the home front), torture is morally and politically unacceptable.

67 Developing situational awareness and generating intelligence from a multiplicity of sources is time-consuming and manpower-intensive. By September 1972, some 500 civilian and military personnel (excluding MI5 and MI6) were assigned to intelligence work in Northern Ireland, and, according to the British Army’s after-action report, by the end of that decade one-eighth of the regular force soldiers deployed in the province were directly involved in intelligence operations.112 The ‘all source’ approach, the manpower commitment it entails, and the tendency of intelligence organizations to work in separate ‘stovepipes’ points to two requirements: ‘surge capacity’ in personnel and resources and the need for ‘fusion centers’ to ensure that information gathered by various services is collated and analyzed in a holistic fashion that connects the dots.

68 While new information technologies place techniques, such as data-mining and Social Network Analysis, at the fingertips of modern counter-insurgency intelligence collectors and analysts, these are not panaceas; indeed, they carry the risk of information overload. Nor is there a "one size fits all" solution to counter-insurgency intelligence. Each conflict is unique, and the first task of intelligence must be to clarify the context and nature of the conflict so that appropriate means can be applied; nevertheless, the British experience offers some timeless lessons that transcend specific conflicts. It suggests that time and resources invested in strengthening the police, inter-agency coordination, and developing and exploiting human sources will yield disproportionate dividends to the counter-insurgent with the patience to see these efforts through to fruition.

David A. Charters is a Professor in the Department of History at the University of New Brunswick.