I’ve long maintained the view that spies never discover anything useful about a country’s foreign enemies, though they are very useful in suppressing domestic opponents. This is a straightforward implication of game theory, but my attempts to explain it haven’t worked in the past, and I don’t know how to do much better. So, I’m going to restate my arguments from 10 years ago, against the massive expansion of spying that was already under way, and make the observation that the evidence since then strongly supports my case.

Despite an espionage and surveillance effort unparalleled in history, the US NSA has been unable to produce any convincing evidence of stopping even one domestic terror plot. Its best case was someone alleged to have sent a few thousand dollars to Al Shabab in Somalia. The NSA not only missed actual terror plotters like those in Boston, but also performed poorly relative to ordinary police methods which have produced numerous convictions (many of them admittedly, by methods that verge on entrapment).

But if anti-terrorist espionage has proved ineffectual, spying on friendly governments is just plain stupid. This isn’t a zero-sum game, like espionage in warfare, it’s a negative sum game. Australia is now finding this out, but the reflex reactions of “everyone does it”, “we don’t comment on intelligence matters” and so on, remain as firmly embedded as ever.

Of course, while this is stupidity as regards the public interest, or even that of Australian political and business elites as a whole, it is massively beneficial to the security apparatus, and the complex of interests it supports. It’s striking that the only Indonesians who’ve given Abbott any support have been their own spies and secret police, who can expect more funding and greater powers. Doubtless our own spooks will return the favor in due course, if their Indonesian counterparts are caught doing something we don’t like.

The spy myth (2003)

If there is an emblematic figure for the 20th century, it is surely the spy. The first decades of that haunted century saw the rise to prominence of the spy novel, with such exemplars as John Buchan’s The Thirty-Nine Steps and Erskine Childers’ The Riddle of the Sands. The final decade saw retired spymasters reach the pinnacle of power in the United States (George Bush Sr.) and Russia (Vladimir Putin).

In espionage, as in so many other cases, life imitated art. The crucial motifs that made up the 20th-century concept of espionage, including the secret plan, the conspiracy and the femme fatale, were all present by the late 19th Century, in Sherlock Holmes stories like The Bruce-Partington plans and A Scandal in Bohemia. Yet actual espionage played an insignificant role in the real international relations of the day.

As late as 1930, Herbert Hoover’s Secretary of State, Henry Stimson repudiated the whole dirty business with the observation that ‘gentlemen do not read other gentlemen’s mail’. This statement would, by then, have been regarded as hopelessly naive in most of the European capitals, where gentlemen had cease to play any role in international politics. As we shall see, however, it reflects a more sophisticated view of the world than one based on the idea that the employer of spies can gain access to the secrets of his or enemies and therefore defeat them.

The first test of the literary concept of the spy came with the outbreak of the Great War in 1914. Popular hysteria about secret weapons bore little relation to the grinding attrition of the Western Front. Although the Germans and Allies achieved surprises with chlorine gas and the tank respectively, the resulting gain of a few miles of mud did nothing to shift the balance of the war.

The dominance of literary concepts over reality was even more evident in the trial and execution of alleged female spies. The Allies tried and shot the notorious exotic dancer Mata Hari. In retrospect, she appears to have been guilty of little more than a taste for self-dramatisation and indiscreet gossip, but the persona she constructed for herself fit perfectly the stereotype of the femme fatale spy. The Germans, with the disastrous sense of PR they displayed throughout the Great War, chose to shoot a British nurse, Edith Cavell, easily represented as a Madonna figure to contrast with the symbolic Whore, Mata Hari.

The gap between myth and reality was similarly great in other theatres. The Russian activities of Sidney Reilly, the famous ‘Ace of Spies’ have formed the basis of a string of books and a TV series starring Sam Neill. A recent study in Intelligence and National Security gave the following more prosaic judgement.

Thus, with the evidence now at hand, the famous ‘Lockhart Plot’ can at last be seen for what it was: on the one hand, a real, if pitiful, anti-Soviet conspiracy concocted (or perhaps deliberately provoked) by the megalomaniacal Sidney Reilly in likely collusion with the eager but inexperienced Bruce Lockhart, and, on the other, a superb example of police provocation brilliantly conceived and expertly executed by the crafty agents of the Cheka.”

Spy hysteria continued to mount after the end of active hostilities in 1918. The totalitarian regimes that rose to power in the aftermath of the Great War made liberal use of spies, and gave rise to the first organized espionage machines, the Gestapo and the various incarnations of the Russian Cheka. These organisations proved ruthlessly effective in suppressing internal opposition, and use both financial and ideological appeals to recruit foreign agents in large numbers. The spy myth, it seemed, had become a reality.

Yet the actual achievements of these shadowy regiments were unimpressive. In most cases, espionage agencies can cloak their failures in secrecy, but the defeat of the Nazis paved the way for a look at the record of one of the most-feared espionage networks in history. The Hitler regime made numerous attempts to infiltrate spies into Britain and to recruit British agents. As far as can be determined from the German records, all were captured and many were ‘turned’, being induced or forced to transmit disinformation to Berlin.

As usual, art makes the spy look better. In the thriller, The Eye of the Needle, the German agent, played by Donald Sutherland, gets within seconds of exposing the subterfuges by which the British simulated preparations for an invasion in the Pas de Calais, thereby diverting German defences from Normandy. In reality, none of Hitler’s spies got anywhere near this.

An even more telling example is that of Pearl Harbour. A variety of intelligence sources gave the US government warnings of an impending attack. These warnings have formed the basis of subsequent conspiracy theories in which Roosevelt deliberately allowed the Japanese to succeed in order to force a reluctant US population into the war. The reality is more prosaic. Most of the warnings were vague inferences from intercepted communications, indicating that the Japanese were up to something, but not when and where. Reports from agents claiming to have inside information on Japanese plans were discounted, on the sensible basis that such claims usually turned out to be either attempts to extract financial rewards, or Japanese misinformation. In the latter case, preparations against a supposed attack would serve to expose security flaws in the Japanese military and perhaps also serve as a casus belli.

Similarly, despite ample warning, Stalin failed to prepare against Hitler’s invasion of Russia. As with Roosevelt he faced the problem that preparation on a serious scale would amount to an overt declaration of hostility against a supposedly friendly power.

It was only in 1944 that the basis was laid for a theoretical understanding of the game of spy and counterspy. In that year, John von Neumann and Oskar Morgenstern published The Theory of Games and Economic Behavior, in which they showed how a wide range of phenomena, from wars to markets, could be analysed in terms of games between players with conflicting interests.

The basic lesson of game theory for a game of bluff like that of espionage is that, as long as it is possible for counterspies to generate misleading information most of the time, spies are useless even when their information happens to be correct. If the defence plays optimally, the spymaster can never have any reason to believe one piece of information produced by spies and disbelieve another.

Spying may be worthwhile in cases where it is very hard or very costly to produce misleading information. Two potential cases are those of code-breaking in wartime, where the number of messages an enemy needs to send is so large that their validity can be checked fairly easily, and that of a secret weapon, where the information produced by spies can be checked by actually making the weapon.

In general, code-breaking relies only marginally on traditional spying methods. The most famous success, the British cracking of the German ‘Enigma’ code in World War II, was helped by the Poles who had stolen a machine before the outbreak of war, then smuggled it to England. However, the effort primarily relied on the mathematical analysis of German messages, which was undertaken by a team led by the enigmatic, and ultimately tragic, genius Alan Turing. (He committed suicide after the war, following persecution by the security forces on the basis of his homosexuality).

The secret weapon of all time was, of course, the atomic bomb, and the period after it was dropped on Hiroshima and Nagasaki represented the high point of the spy panic, particularly in the English-speaking world. When the Russians exploded their own bomb, it seemed quite likely that the end of world was approaching. The discovery that British scientists such as Fuchs and Nunn, and American Communists like the Rosenbergs, had passed atomic secrets to the Russians, created a panic.

On the face of it, the reaction to the atom spies seemed justified. The atom bomb was a weapon that could destroy the world (and perhaps still will) and the science on which it was based was popularly associated with the genius of Albert Einstein. Surely, the only way the Russians could create such a weapon was to steal the secrets of the West.

It is now clear, however, that the only real secret regarding the atomic bomb was that it could be made to work. This secret was successfully concealed from the Nazis, who focused instead on the other great secret weapon of the century, the guided missile represented by the V2 rocket. But once the existence of the bomb was known, any competent team of physicists, with access to the right resources, could duplicate it. The Russians had competent physicists of their own, and captured some of leading German researchers. The secrets passed by Western spies probably saved them a year or so in their research program but did not fundamentally change anything. The Chinese, French, Israelis and others made their bombs without significant assistance from spies.

But this is the wisdom of hindsight. The exposure of the atom spies set the scene for the 20th-century apogee of spy hysteria, including such sensational episodes as the defection of Burgess and McLean, the exposure and flight of the ‘third man’, Kim Philby, the McCarthy hearings in the United States, and even our own Petrov crisis. Suddenly, it seemed, spies were everywhere, and an all-out response seemed called for.

In literature, the response was represented by the glamorous professional, James Bond, who transformed the spy novel from an ambiguous cross between detective story and thriller into a fully fledged sub-literary genre. Bond himself was a transitional figure between the gentlemanly amateurism of earlier heroes like Buchan’s Richard Hannay and the grubby professionalism celebrated by, for example, Len Deighton.

Belief in spies declined after 1960. The pivotal moment was probably the first significant step towards detente, the ban on atmospheric nuclear testing agreed between Britain, the US and the USSR in 1963. From this point onwards, it became steadily more evident that keeping nuclear secrets was a bad idea and that trying to steal secrets only encouraged those who wanted to keep them.

The exposure of a wide range of misconduct by the CIA and other agencies produced widespread hostility towards, and even more widespread cynicism about, the whole enterprise of espionage and counterespionage. The decline was charted by the novels of John Le Carre. The title and theme of The Looking Glass War, in which bureaucratic infighting in London results in a doomed attempt at infiltrating East Germany, captured, even more than his more famous works, the pointlessness of the entire enterprise.

The spy myth clearly served the interests of intelligence agencies, which prospered during the 20th century more than any set of spies before them. The real beneficiaries, however, were the counterintelligence agencies or, to dispense with euphemisms, the secret police, of both Western and Communist countries. The powers granted to them for their struggle against armies of spies were used primarily against domestic dissidents. Terms such as ‘agent of influence’ were used to stigmatise anyone whose activities, however open and above-board, could be represented as helpful to the other side.

The supposed role of the secret police, to keep secrets from opposing governments, was, as we have seen, futile. Secret police, and the associated panoply of security laws, Official Secrets Acts and so forth, were much more successful in protecting their governments’ secrets from potentially embarrassing public scrutiny in their own countries.

As spies and the associated fears have faded in their public mind, their place has been taken by terrorists. In many ways, this is a reversion to the 19th century, when the bomb-throwing anarchist was a focus of popular fears and the subject of novels by such writers as Chesterton and Conrad.

As the attacks of September 11 showed us, the threat posed by terrorists is real. Nevertheless, even if terrorists were to mount attacks ten times as deadly in the future, they would still present the citizens of the Western World with less danger than we accept from our fellow-citizens every time we step into our cars.

If the century of the spy has taught us anything, it is that we need to assess the dangers posed by terrorists coolly and calmly rather than giving way to panic.