Why spies never discover anything useful

I’ve long maintained the view that spies never discover anything useful about a country’s foreign enemies, though they are very useful in suppressing domestic opponents. This is a straightforward implication of game theory, but my attempts to explain it haven’t worked in the past, and I don’t know how to do much better. So, I’m going to restate my arguments from 10 years ago, against the massive expansion of spying that was already under way, and make the observation that the evidence since then strongly supports my case.

Despite an espionage and surveillance effort unparalleled in history, the US NSA has been unable to produce any convincing evidence of stopping even one domestic terror plot. Its best case was someone alleged to have sent a few thousand dollars to Al Shabab in Somalia. The NSA not only missed actual terror plotters like those in Boston, but also performed poorly relative to ordinary police methods which have produced numerous convictions (many of them admittedly, by methods that verge on entrapment).

But if anti-terrorist espionage has proved ineffectual, spying on friendly governments is just plain stupid. This isn’t a zero-sum game, like espionage in warfare, it’s a negative sum game. Australia is now finding this out, but the reflex reactions of “everyone does it”, “we don’t comment on intelligence matters” and so on, remain as firmly embedded as ever.

Of course, while this is stupidity as regards the public interest, or even that of Australian political and business elites as a whole, it is massively beneficial to the security apparatus, and the complex of interests it supports. It’s striking that the only Indonesians who’ve given Abbott any support have been their own spies and secret police, who can expect more funding and greater powers. Doubtless our own spooks will return the favor in due course, if their Indonesian counterparts are caught doing something we don’t like.

The spy myth (2003)

If there is an emblematic figure for the 20th century, it is surely the spy. The first decades of that haunted century saw the rise to prominence of the spy novel, with such exemplars as John Buchan’s The Thirty-Nine Steps and Erskine Childers’ The Riddle of the Sands. The final decade saw retired spymasters reach the pinnacle of power in the United States (George Bush Sr.) and Russia (Vladimir Putin).

In espionage, as in so many other cases, life imitated art. The crucial motifs that made up the 20th-century concept of espionage, including the secret plan, the conspiracy and the femme fatale, were all present by the late 19th Century, in Sherlock Holmes stories like The Bruce-Partington plans and A Scandal in Bohemia. Yet actual espionage played an insignificant role in the real international relations of the day.

As late as 1930, Herbert Hoover’s Secretary of State, Henry Stimson repudiated the whole dirty business with the observation that ‘gentlemen do not read other gentlemen’s mail’. This statement would, by then, have been regarded as hopelessly naive in most of the European capitals, where gentlemen had cease to play any role in international politics. As we shall see, however, it reflects a more sophisticated view of the world than one based on the idea that the employer of spies can gain access to the secrets of his or enemies and therefore defeat them.

The first test of the literary concept of the spy came with the outbreak of the Great War in 1914. Popular hysteria about secret weapons bore little relation to the grinding attrition of the Western Front. Although the Germans and Allies achieved surprises with chlorine gas and the tank respectively, the resulting gain of a few miles of mud did nothing to shift the balance of the war.

The dominance of literary concepts over reality was even more evident in the trial and execution of alleged female spies. The Allies tried and shot the notorious exotic dancer Mata Hari. In retrospect, she appears to have been guilty of little more than a taste for self-dramatisation and indiscreet gossip, but the persona she constructed for herself fit perfectly the stereotype of the femme fatale spy. The Germans, with the disastrous sense of PR they displayed throughout the Great War, chose to shoot a British nurse, Edith Cavell, easily represented as a Madonna figure to contrast with the symbolic Whore, Mata Hari.

The gap between myth and reality was similarly great in other theatres. The Russian activities of Sidney Reilly, the famous ‘Ace of Spies’ have formed the basis of a string of books and a TV series starring Sam Neill. A recent study in Intelligence and National Security gave the following more prosaic judgement.

Thus, with the evidence now at hand, the famous ‘Lockhart Plot’ can at last be seen for what it was: on the one hand, a real, if pitiful, anti-Soviet conspiracy concocted (or perhaps deliberately provoked) by the megalomaniacal Sidney Reilly in likely collusion with the eager but inexperienced Bruce Lockhart, and, on the other, a superb example of police provocation brilliantly conceived and expertly executed by the crafty agents of the Cheka.”

Spy hysteria continued to mount after the end of active hostilities in 1918. The totalitarian regimes that rose to power in the aftermath of the Great War made liberal use of spies, and gave rise to the first organized espionage machines, the Gestapo and the various incarnations of the Russian Cheka. These organisations proved ruthlessly effective in suppressing internal opposition, and use both financial and ideological appeals to recruit foreign agents in large numbers. The spy myth, it seemed, had become a reality.

Yet the actual achievements of these shadowy regiments were unimpressive. In most cases, espionage agencies can cloak their failures in secrecy, but the defeat of the Nazis paved the way for a look at the record of one of the most-feared espionage networks in history. The Hitler regime made numerous attempts to infiltrate spies into Britain and to recruit British agents. As far as can be determined from the German records, all were captured and many were ‘turned’, being induced or forced to transmit disinformation to Berlin.

As usual, art makes the spy look better. In the thriller, The Eye of the Needle, the German agent, played by Donald Sutherland, gets within seconds of exposing the subterfuges by which the British simulated preparations for an invasion in the Pas de Calais, thereby diverting German defences from Normandy. In reality, none of Hitler’s spies got anywhere near this.

An even more telling example is that of Pearl Harbour. A variety of intelligence sources gave the US government warnings of an impending attack. These warnings have formed the basis of subsequent conspiracy theories in which Roosevelt deliberately allowed the Japanese to succeed in order to force a reluctant US population into the war. The reality is more prosaic. Most of the warnings were vague inferences from intercepted communications, indicating that the Japanese were up to something, but not when and where. Reports from agents claiming to have inside information on Japanese plans were discounted, on the sensible basis that such claims usually turned out to be either attempts to extract financial rewards, or Japanese misinformation. In the latter case, preparations against a supposed attack would serve to expose security flaws in the Japanese military and perhaps also serve as a casus belli.

Similarly, despite ample warning, Stalin failed to prepare against Hitler’s invasion of Russia. As with Roosevelt he faced the problem that preparation on a serious scale would amount to an overt declaration of hostility against a supposedly friendly power.

It was only in 1944 that the basis was laid for a theoretical understanding of the game of spy and counterspy. In that year, John von Neumann and Oskar Morgenstern published The Theory of Games and Economic Behavior, in which they showed how a wide range of phenomena, from wars to markets, could be analysed in terms of games between players with conflicting interests.

The basic lesson of game theory for a game of bluff like that of espionage is that, as long as it is possible for counterspies to generate misleading information most of the time, spies are useless even when their information happens to be correct. If the defence plays optimally, the spymaster can never have any reason to believe one piece of information produced by spies and disbelieve another.

Spying may be worthwhile in cases where it is very hard or very costly to produce misleading information. Two potential cases are those of code-breaking in wartime, where the number of messages an enemy needs to send is so large that their validity can be checked fairly easily, and that of a secret weapon, where the information produced by spies can be checked by actually making the weapon.

In general, code-breaking relies only marginally on traditional spying methods. The most famous success, the British cracking of the German ‘Enigma’ code in World War II, was helped by the Poles who had stolen a machine before the outbreak of war, then smuggled it to England. However, the effort primarily relied on the mathematical analysis of German messages, which was undertaken by a team led by the enigmatic, and ultimately tragic, genius Alan Turing. (He committed suicide after the war, following persecution by the security forces on the basis of his homosexuality).

The secret weapon of all time was, of course, the atomic bomb, and the period after it was dropped on Hiroshima and Nagasaki represented the high point of the spy panic, particularly in the English-speaking world. When the Russians exploded their own bomb, it seemed quite likely that the end of world was approaching. The discovery that British scientists such as Fuchs and Nunn, and American Communists like the Rosenbergs, had passed atomic secrets to the Russians, created a panic.

On the face of it, the reaction to the atom spies seemed justified. The atom bomb was a weapon that could destroy the world (and perhaps still will) and the science on which it was based was popularly associated with the genius of Albert Einstein. Surely, the only way the Russians could create such a weapon was to steal the secrets of the West.

It is now clear, however, that the only real secret regarding the atomic bomb was that it could be made to work. This secret was successfully concealed from the Nazis, who focused instead on the other great secret weapon of the century, the guided missile represented by the V2 rocket. But once the existence of the bomb was known, any competent team of physicists, with access to the right resources, could duplicate it. The Russians had competent physicists of their own, and captured some of leading German researchers. The secrets passed by Western spies probably saved them a year or so in their research program but did not fundamentally change anything. The Chinese, French, Israelis and others made their bombs without significant assistance from spies.

But this is the wisdom of hindsight. The exposure of the atom spies set the scene for the 20th-century apogee of spy hysteria, including such sensational episodes as the defection of Burgess and McLean, the exposure and flight of the ‘third man’, Kim Philby, the McCarthy hearings in the United States, and even our own Petrov crisis. Suddenly, it seemed, spies were everywhere, and an all-out response seemed called for.

In literature, the response was represented by the glamorous professional, James Bond, who transformed the spy novel from an ambiguous cross between detective story and thriller into a fully fledged sub-literary genre. Bond himself was a transitional figure between the gentlemanly amateurism of earlier heroes like Buchan’s Richard Hannay and the grubby professionalism celebrated by, for example, Len Deighton.

Belief in spies declined after 1960. The pivotal moment was probably the first significant step towards detente, the ban on atmospheric nuclear testing agreed between Britain, the US and the USSR in 1963. From this point onwards, it became steadily more evident that keeping nuclear secrets was a bad idea and that trying to steal secrets only encouraged those who wanted to keep them.

The exposure of a wide range of misconduct by the CIA and other agencies produced widespread hostility towards, and even more widespread cynicism about, the whole enterprise of espionage and counterespionage. The decline was charted by the novels of John Le Carre. The title and theme of The Looking Glass War, in which bureaucratic infighting in London results in a doomed attempt at infiltrating East Germany, captured, even more than his more famous works, the pointlessness of the entire enterprise.

The spy myth clearly served the interests of intelligence agencies, which prospered during the 20th century more than any set of spies before them. The real beneficiaries, however, were the counterintelligence agencies or, to dispense with euphemisms, the secret police, of both Western and Communist countries. The powers granted to them for their struggle against armies of spies were used primarily against domestic dissidents. Terms such as ‘agent of influence’ were used to stigmatise anyone whose activities, however open and above-board, could be represented as helpful to the other side.

The supposed role of the secret police, to keep secrets from opposing governments, was, as we have seen, futile. Secret police, and the associated panoply of security laws, Official Secrets Acts and so forth, were much more successful in protecting their governments’ secrets from potentially embarrassing public scrutiny in their own countries.

As spies and the associated fears have faded in their public mind, their place has been taken by terrorists. In many ways, this is a reversion to the 19th century, when the bomb-throwing anarchist was a focus of popular fears and the subject of novels by such writers as Chesterton and Conrad.

As the attacks of September 11 showed us, the threat posed by terrorists is real. Nevertheless, even if terrorists were to mount attacks ten times as deadly in the future, they would still present the citizens of the Western World with less danger than we accept from our fellow-citizens every time we step into our cars.

If the century of the spy has taught us anything, it is that we need to assess the dangers posed by terrorists coolly and calmly rather than giving way to panic.

61 thoughts on “Why spies never discover anything useful

  1. @Peter T

    In the criminal cases, the primary feature of wiretaps and similar is that they provide evidence that’s admissible in court, proving something the police already knew or suspected. And as regards Iraq, the fact that Bush was lying was evident the moment Saddam readmitted weapons inspectors.


    It’s true, though, that the logic of the argument I present isn’t universal. As presented, it applies to the case when one side is trying to predict which of several actions the other will take, and where the two sides are more or less evenly matched in terms of resources for espionage and counter-espionage. The terrorist case doesn’t fit this symmetric model; rather the problem is that there are great many actions the terrorists can take, and the preparations don’t require vast amounts of communication.

    By contrast, domestic dissidents and illegal businesses are more vulnerable, since they need to communicate to keep operating, and they don’t have nearly as much in the way of countermeasures. So, even if surveillance doesn’t discover much of value, it imposes costs that the state can easily afford but the opponents typically can’t.

  2. John

    The intercept evidence produced in court is usually a small fraction of what is collected. Much, of course, is irrelevant, but of what remains much that is not evidence (the Evidence Act sets quite stringent limits on what is admissable), is useful intelligence. And intercept is directed by collection of metadata and by informants.

    Even more than criminals, states need to communicate to keep operating. And they need to communicate at many different levels, to many different sectors of the population. Much intelligence is checking how far the different communications are consistent with public information. And for this you need to build up a picture over time, as different states have quite different modes of operation. Spy novels and news focuses on the one item that makes all the difference, but this does not actually happen in reality – it’s the day to day monitoring and day to day presentation to policy makers that sets the pattern. And most of the time the effect is to reassure, which is surely a good thing.

  3. @Peter T
    But the evidence is that, under conditions of hot or cold war, states can fake this stuff quite effectively.

    Spying on your friends is easier, as long as they don’t suspect you, but it’s a good way to lose friends, as we are now seeing in both Indonesia and Europe.

  4. Yes you can fake it – but it’s not actually easy, and sometimes quite expensive. Britain and US faked the capacity to make several landings in northern France, for instance. But this involved a carefully orchestrated flow of information, mostly small items but all together making up a false picture (troop numbers, HQs, radio nets, landing ship production and so on), coupled with effective suppression of surveillance (other than what they permitted). But my point is that a lot of “spying” is actually fairly passive (eg communications network pattern monitoring, or satellite surveillance), mostly it goes into continual small corrections to the daily news slowly building a picture for policymakers, mostly it involves not sudden surprises but small adjustments.

    The surprise in Snowden was that the NSA had overstepped the bounds by hoovering up evrything they could on their “friends” – in defiance of their commitments. As you say, no way to win friends.

  5. @derrida derider politics is a family business.

    knowing who is supporting whom and who is paying who is useful in understanding the goings on of corrupt countries.

    I am sure the phone calls of imelda marcos would have been good listening on the state of filipino politics and who is going to be the winning side, so you can back the winning side.

    The british were far more effective in their use of Ultra intelligence than the germans and americans were through their code breaking.

  6. “breaking” news – canada – yesterday

    Top secret documents retrieved by U.S. whistleblower Edward Snowden show that Prime Minister Stephen Harper’s government allowed the largest American spy agency to conduct widespread surveillance in Canada during the 2010 G8 and G20 summits.


    the “largest american spy agency” is the nsa, and the “canadian partner” gowingly mentined inthe documents is the csec – communications security establishment canada – which is the outfit exposed earlier this month as having conducted surveillance on brazil’s mines & energy ministry.
    due to an australian, in the form of wayne swann, being there when it happened, it may well make the local news, but you heard it first from me.

    twenty five is a lot of countries to piss off all at once i reckon.

    i wonder if the australian govt will register being pissed off. or if abbott will strike a sanguine pose. i wonder if shorten will strike a sanguine pose in support of “team australia”, or be pissed off, like, for australia. what about swann? new zealand? not to mention china.

    i don’t know about others, but, to the extent this story gets any traction in australia, i intend to severely judge present politicians by what stand they take in light of this revelation. for now i expect milne will denounce it. i expect abbott and shorten to grovel and slavishly mouth loathsome platitudes about the value of the alliance versus the right to privacy and the sovereignty of nations. i’d be pleased to be wrong. interesting times.

    and all this while the sophomoric tosser bishop rattles china.

  7. But if anti-terrorist espionage has proved ineffectual, spying on friendly governments is just plain stupid. This isn’t a zero-sum game, like espionage in warfare, it’s a negative sum game.

    Not really. Imperialist countries, that is countries that meddle with other smaller countries, take advantage of them, place puppet governments and such, so as to secure trade and arm agreements in their favor, cheap natural resources, diplomatic stronholds, etc etc — have a lot to win by spying on friendly governments.

    That’s because they want those countries and their governments not to be “friendly but equal” (like actual peers), but “friendly as subordinates” (like lackeys).

    So they use all they find, be it dirty laundry by politicians, secrets and information against people that are against their meddling in their countries (e.g people against some unfavorable trade agreement or investment, people that want a genuinely popularly loved person for their PM instead of who the larger country promotes, etc) to get the upper hand in lots of situations.

  8. Moin,

    > This is a straightforward implication of game theory, but my attempts to explain it haven’t worked in the past, and I don’t know how to do much better.

    This sounds really interesting. Can you point me to the URL explain it, or write an other blog about that spy game theory?


  9. This post prompted me to buy Peter Fitzsimmon’s biography of Nancy Wake, aka the White Mouse. What an amazing and uplifting story of a very courageous person, who played a key role in the French Resistance during WWII.

    Anyway, Nancy Wake did little in the way of spying (according to the book), her main job was organising the incredibly successful drop off of supplies to the Resistance from planes leaving Britain.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s