Dr. Hill's purpose in this experiment was to show just how difficult it is to fake data convincingly. It just isn't that easy to make up a random sequence. Based on this knowledge, he would astound his students by unerringly picking out the fakers from the tossers!

One of the ways he would do this would be to spot how many times heads or tails would be listed six or more times in a row. In real life, this occurrence is overwhelmingly probable in 200 coin throws. To most of his students this long a sequence is counter-intuitive, an example of what is often termed the Gamblers' Fallacy, i.e. the erroneous perception that independent random sequences will balance out over time, so that for example an extended sequence of heads is more likely to be followed by a tail than a head. The fakers, susceptible to the Fallacy, are thus easily exposed. Ordinary people, even mathematics students, simply can't help introducing patterns into what is random noise.

This is an example of a broader analysis which is usually referred to a Benford's Law, which essentially states that if we randomly select a number from a table of real-life data, the probability that the first digit will be one particular number is significantly different to it being a different number. For example, the probability that the first digit will be a '1' is about 30%, rather than the intuitive 10%, which assumes that all digits are equally likely.

The empirical support for this proportion can be traced to the man after whom the Law is named, physicist Dr. Frank Benford, in a paper he published in 1938, called 'The Law of Anomalous Numbers'. In that paper he examined 20,229 sets of numbers, as diverse as baseball statistics, the areas of rivers, numbers in magazine articles and so forth, confirming the 30% rule for number 1. For information, the chance of throwing up a '2' as first digit is 17.6%, and of a '9' just 4.6%. The same principle applies to trailing (i.e. last) digits. It's a great way, therefore, of checking the veracity of receipts. If, for example, there is an unusual number of trailing digit '7's, there's a decent chance that the figures are cooked. Tax authorities are alert to this.

So randomness is not so random, it seems. Applying the analysis to some recent real-world events, Nate Silver, founder of www.fivethirtyeight.com , polling guru and all-round statistical wizard, examined the results of the recent Iranian election, declaring them "probably forged." More recently still, he's turned his attention to one of the US's high-profile pollsters. The results of his analysis of their findings and his interpretation make fascinating reading.

It's important that survey results are genuine, as they are integral factors in driving the markets, and in particular the election markets. If you are able to differentiate the genuine from the phony, it may thus give you the edge in beating the odds. Let's call it Benford's bonus!

But why is an atom the size that it is? That's determined by the size of the orbits of the electrons around the atom, which in turn is determined by the mass of the electron. Smaller mass means smaller orbits, means smaller everything. But why is the electron the mass that it is? Indeed, why does it have any mass at all? In fact, theory dictates that for all the elementary particles that compose the atom to interact as they do, their masses should actually be zero.

So what's happening? The neat solution proposed by Professor Peter Higgs would seem to have it all figured out. He suggests that there is a field which permeates space, which moving particles interact with, thus acquiring the appearance of mass. Imagine a weightless pea (if you can!) moving through treacle.

So does this field actually exist? Quantum theory tells us that fields are associated with particles (a thing called 'wave-particle duality') so there must be a particle complementary to the Higgs field. For example, the particle associated with the electromagnetic field is the photon. The particle is known as the Higgs boson, and a lot of time, money, energy and intellect is being applied to finding out whether this particle, and therefore whether the Higgs field, actually exists.

This is where the Large Hadron Collider comes in, built by the European Organization for Nuclear Research (CERN). It works by accelerating protons around a kind of 17-mile underground racetrack, in order to smash them together at astronomically high speeds, with the purpose of creating smaller bits of matter, one of which could be a Higgs boson. If created, it would exist for only a tiny fraction of time, but long enough it is hoped to be detected.

The problem is that the closer we seem to get to finding it, the more things go wrong to prevent that very thing happening. From the cutting off of funding for a Super Collider in the US, to the 'meltdown' of the experiment at CERN last year, to the recent arrest of a top scientist associated with the project for alleged terrorist connections, it seems like someone, somewhere doesn't want it to happen. Seems like, but not really! Surely!

Which is where two well respected scientists, Holger Bech Nielsen and Masao Ninomiya come in. In two recent papers they suggest in effect that nature has in fact created a sort of prediction market in reverse. Nature knows the consequence of an apparently successful outcome for the experiment, they suggest, and is trying to stop it.

The creation of the boson, according to this theory, would cause a ripple backwards in time to stop what created it. As Nielsen puts it, "It is based on mathematics, but you could explain it by saying that God rather hates Higgs particles and attempts to avoid them."

As Einstein once put it in a letter to the family of a friend, "For those who believe in physics, this separation between past, present and future is only an illusion."
Now let's pause for thought!

So why the surprise? Simply put, it was because the systems were not in place with which to put the jigsaw together in time. The 9/11 Commission Report stated the problem like this: "The biggest impediment to all-source analysis - to a greater likelihood of connecting the dots - is the human or systemic resistance to sharing information."

James Surowiecki, in his book, 'The Wisdom of Crowds', offers the following perspective on this failure: "What was missing in the intelligence community ... was any real means of aggregating not just information but also judgements. In other words, there was no mechanism to tap into the collective wisdom of National Security nerds, CIA spooks, and FBI agents. There was decentralization but not aggregation ..."

The question is whether the market can help achieve this. Some people within the US Department of Defence already thought so, and had been working on just such an idea for several months when al Qaeda struck. Indeed, in May 2001 the Defense Advanced Research Projects Agency (DARPA) had issued a call for proposals under the heading of 'Electronics Market-Based Decision Support' (later 'Future Markets Applied to Prediction' (FutureMAP).

The remit prescribed for FutureMAP was to create market-based techniques for avoiding surprise and predicting future events. It was not long, however, before the media and key members of the political class, began to train their guns on the idea of such a market. After all, it isn't difficult to portray what was devised as a policy analysis market as no more than a forum for eager traders to profit from death and destruction. The populist arguments won the day and DARPA was forced to cancel the project.
While most of the arguments against the market were specious, there was some genuine intellectual concern as to how effective it would be likely to be.

In particular, Joseph Stiglitz, winner of the 2001 Nobel Prize for economics, argued in an article published in the Los Angeles Times on 31 July 2003 ('Terrorism: There's No Futures in It'), that the market would be too "thin" (i.e. there would be too little money traded in the market) for it to be a useful tool for predicting events meaningfully. His argument was based on work he had previously published showing that markets can never be perfectly efficient when information is costly to obtain. The cost of obtaining and processing this information is, by implication, likely to act as a significant disincentive particularly in the context of a thin market (and hence low rewards).

I am not so sure. Is it obviously the case that a properly constructed market, populated by suitably motivated (and perhaps screened) players can be viewed in this way?
Well, the jury's still out on this, and in the meantime so are the so-called 'terrorism futures'. For how long, I wonder?

Clearly this makes the weather in central Florida of critical importance to the market price of orange juice. One would expect, therefore, that the weather forecast for central Florida should be a key factor influencing the price at which orange juice futures trade. That's one thing. But what now if we phrase the issue in reverse? Can we use these orange price futures to actually forecast the weather? And if so, how well do forecasts so obtained compare with the forecasts issued by the official National Weather Service?

A study carried out by Professor Richard Roll of the Graduate School of Management at the University of California sought to find out. In the resulting research paper, called "Orange Juice and Futures", published in the American Economic Review in December 1984, Roll found what he termed a "statistically significant relation between OJ returns and subsequent errors in temperature forecasts issued by the National Weather Service for the central Florida region."

In particular, if the closing orange juice futures price is higher than its opening price, we would obtain a more accurate prediction of the weather by adjusting downward the National Weather Service's weather forecast. This is because a higher futures price implies a smaller crop and colder weather. Correspondingly, if the orange juice futures price is lower than its opening price, we should adjust upwards.

A front-page article in Florida's St. Petersburg Times, on Monday, October 17, 1983, puts this in some perspective. "The National Weather Service spends $280 million each year predicting the nation's weather," it runs, "and it uses some of the niftiest gadgets ... to do it. But it may be overlooking the price of orange juice."

In an interview with the paper, Roll noted how in 1981, when the temperature in central Florida fell into the 20s (Fahrenheit) for four consecutive nights, the price of orange juice futures rose 40%. "They [the traders] have their own meteorologists", he explained. "They have a bigger incentive than the Weather Bureau itself to predict it correctly." In particular, if traders can anticipate falling temperatures they can buy before the price goes up and sell after it does.

An important point to note here is that nobody is saying that the official weather forecasting service is of no value. Indeed, the market might have been a lot less accurate if these forecasts didn't exist. What Roll's study does suggest, however, is that the commodity markets add some information above and beyond this, much as opinion polls can help forecast election outcomes, without representing the totality of information available to those trading the election betting markets.

Whoever would have thought the economics of orange juice could be so interesting!

When you have the edge in a competition, it would seem that the rational strategy is to minimize the role of luck in determining the outcome. So if Roger Federer agrees to play one point against me at tennis, on his own serve, with a forfeit of £100,000 if he loses the point, I doubt that his optimal strategy would be to serve flat out. This would be to increase the risk that he double faults, which is by some margin my best hope of winning the point. In other words, variance is my friend but Mr. Federer's enemy.

The same would seem to apply in a horse race, where the jockey's optimal strategy when aboard the best horse in the field is unlikely to be that of boxing himself in on the rail behind a wall of horses. Yet this is what Mick Kinane managed to achieve in Sunday's Prix de l'Arc de Triomphe atop perhaps the best racehorse ever. Kinane puts it like this: "I didn't have any worries because I knew I was on the fastest horse in the race. His acceleration was fantastic." If so, he must have nerves of steel, because I doubt there was anyone else watching who shared that sentiment less than three furlongs out.

To my mind, Sea the Stars won quite simply because he was so far superior to his rivals that he was able to overcome those obstacles which beset ordinary flesh and blood. And that includes variance! Kinane is a great jockey and perhaps he knew that. Perhaps! Which brings us to another puzzle arising from the result of the race. Why were the bookmakers afterwards bemoaning a lost fortune on the race? According to a spokesman for Coral, for example, the success of Sea the Stars in the Arc would have cost bookmakers "a sizeable seven-figure sum." It is a lament we hear so often, and more often than usual this year.

The bookmakers take a severe hit whenever they suffer a series of winning hot favourites. But why? If favourites are such losers in the book for the layers, why do the layers not simply shorten up the price of those particular horses or football teams or whatever, and lengthen the price of their rivals.

In the case of the Arc, this would mean cutting the odds offered about Sea the Stars by a couple of notches, or even more, and lengthening the others accordingly. Is it because the betting public would bet the same amount on the favourite regardless of the price? Unlikely! But even if true, the liability would in any case be less.

So who is behaving irrationally? Is it the bookmaker or those who bet with them? And does the same apply on the betting exchanges? There are answers to the puzzle, but for once I'd just like to pose the questions. Any thoughts?

It's a few days since I backed Treaty Flyer to win the 3.40 'rippleeffect.com Handicap Hurdle' at Bangor-on-Dee. Generally available at [8.0] (a little longer on Betfair), the eight-year old chestnut mare was two pounds ahead of the handicapper, had the conditions in her favour, and was coming to the race fresh from a sparkling performance in a competitive Stratford handicap. I was a little surprised, therefore, when she opened on course at [9.0], before drifting steadily to [13.0] at the off, and [16.0] on Betfair.

The instinctive reaction of many, like me, who backed her at the early price may have been to write off their losses before the horses had even got into their stride, or perhaps to lay some of the impending loss off on Betfair, albeit at about twice the odds obtained. For there was no obvious reason to explain this marked drift in the betting, causing some to suspect, I guess, that there were some who knew more than they were telling. In the event, the suspicions were totally unfounded, as the mount of Johnny Farrelly, suited by the ground and the fast pace, got up with to beat her stablemate with something to spare, giving the Alison Thorpe yard an impressive 1-2.

Now move forward a little to Tuesday, September 22, and the 5.10 at Stratford, the Class 6 'Jenkinsons Caterers Standard Open National Hunt Flat race'. This time my money was attracted to the Paul Nicholls-trained gelding, King of the Night. Although fourth on his debut in June, the form of that race had been franked with winning re-appearances by the runner-up and third, and the horse was open to plenty of improvement. Backers could not help feeling some confidence in their selection until a deluge of money came for Gentle Jim, a half-brother to the useful jumper Risk Accessor and hurdle winner Galetollah.

From [12.0] on course at the opening show he went off at [4.5]. Ominously, the Racing Post form expert had that morning written of the horse - "interesting runner on racecourse debut, especially if the market speaks in favour". Well, it just had! In the event, however, and in spite of the hype of the 'At the Races' race commentator, nobody had told the horse. It was simply never in serious contention. King of the Night lived up to its promise and won with more than three lengths to spare.

Money talks, they say. And sometimes it does. But not always!

Some dodgy patterns are a bit less easy to spot, however, one of the best known being the so-called 'Superbowl effect.' This effect, identified and published by Professors Krueger and Kennedy in the 'Journal of Finance' in 1990, ostensibly established a solid relationship between a team from the NFC (National Football Conference) winning America's 'Superbowl' and an improvement the following year in the US stock market. The pattern could be traced back to 1967. Unfortunately for Kennedy and Krueger and fellow pattern spotters, however, it soon came crashing down to earth, along with the Denver Broncos.

Now the 'Superbowl' effect, as it has come to be known, always did have a dodgy feel to it, if only because there didn't seem any rhyme or reason why it should work. Other so-called 'anomalies' are rather more difficult to dismiss, however, one of the most well-known of these being the 'Weather Effect', first highlighted by Professor E.M. Saunders in a famous article published in 1993 in the 'American Economic Review.' In that paper, he argued that there was a link between the weather and the performance of stocks on Wall Street - the better the weather, the better the stocks performed.

The question confronting anyone interested in turning a profit was whether this was a genuine relationship or another freak finding? Well, for once those of a sceptical disposition were chastened, courtesy of a follow-up study by David Hirshleifer and Tyler Shumway, released in 2001. In that study, entitled, 'Good Day Sunshine: Stock Returns and the Weather', they confirmed using a different data set that there was indeed a positive link between stock returns and good weather. The reason, they argued, was a feel-good factor influencing investor sentiment. Interestingly, a Yale University working paper published the following year found that the market-makers played a big part in this, widening the spreads in the financial markets on cloudy days. In other words, the link between good weather and stock returns could, they argued, be traced in significant part to the behaviour of market-makers.

As John Lennon and Paul McCartney once put it, "I need to laugh, and when the sun is out, I've got something I can laugh about. I feel good in a special way: I'm in love and it's a sunny day. Good Day Sunshine, Good Day Sunshine, Good Day Sunshine."

Ah well, if only the duo had applied their idea to the stock market! I guess they could have beaten Professor Saunders to it. And who knows? They might even have become rich and famous.

A seminal study in this regard, published by Professors Lieberson and O'Connor, found little evidence of any link between changes of Chief Executive Officer (CEO) and subsequent movements in company performance indicators such as sales and profits.

Other studies have found, in contrast, that changes of top managers do tend to be followed by sharp improvements in performance, particularly in certain sectors, such as computer equipment manufacture. Some areas of activity do seem impervious to changes at the top, however, as witnessed by a study of the effect of changes of Methodist ministers on church attendance, membership and donations. There was no discernible effect.

There is also a well-established literature which considers the impact of management change on team performance in professional sport, and this can be sub-divided into three distinct theories. According to the "common sense" theory, when a team is under-performing the manager is replaced, and if a better manager takes over, performance should improve. In the "vicious circle" theory, poor performance tends to trigger managerial change, but the disruption caused tends to make things worse. The there is the "ritual scapegoating" theory, in which the appointment of a new manager makes no difference, on average, to team performance.

Rick Audas, Stephen Dobson and John Goddard disentangled the competing theories, for the case of English football, in an article published in the Journal of Economics and Business. A detailed examination of the results of their study reveals that on average it takes up to 16 matches for a team subject to a within-season change of manager to adapt to the usual changes of tactics and playing style which ensue, and even then the team's win rate tends to revert only to where it was prior to the change. The transition period is, on average, simply a sink into which some of the points that would have been earned are emptied.

What these results appear to tell us, then, is that the rate at which managers are replaced in English football is not optimal. The turnover is simply too fast.

So in light of these findings, is it possible to offer a rational explanation for existing attitudes to management change? Well, there may just be such an explanation, and it lies in a thing called "variance", which is the dispersion of results about the average.

The idea here is that a change of manager may not improve performance but it does shake things up a bit. This can only be good news, of course, for a team which is likely to go down anyway. The reason is that while a change of manager may on average mean even fewer points, the change does at least improve the small chance of pulling clear of the relegation zone. Seen like this, it can be likened to an all-or-nothing throw of the dice. That's the rational side of the argument. The problem comes when those in charge of the team's future grow just a little too fond of the dice.

To explain why, we need to travel back to the early nineteenth century and the strange case of the Prussian soldiers and the dangerous horses they rode into battle. Official reports suggested that these horses had been a noticeable cause of death for the cavalry for a period of at least 20 years. When the Czar heard this he was so concerned that he commissioned a study to determine whether this was due to chance or to a punishment inflicted by God.

An answer was provided by Simeon Poisson, who showed that the death rate from kicks followed a particular statistical pattern, which came to be known as the Poisson pattern or more formally the 'Poisson distribution'. So what does this have to do with football forecasting? Well, there is one thing that goals in a football match and lethal kicks by horses have in common and it is this - they both occur relatively infrequently and they both occur at some overall average rate.

In order to estimate the chance of any game ending with a particular scoreline, we first need an estimate of the number of goals a team is likely to score in that match. We can then read off from the Poisson distribution, which can be provided in the form of a table, the percentage chance of that team scoring a given number of goals. The value of the table lies in the fact that the predictions of the Poisson distribution tend to conform to actual football results as surely as they did when the horses were despatching the unlucky Prussians to an early death.

Let's take, as an example, the number of goals we might expect England and Brazil to score in a mythical World Cup final. For the sake of argument, and temporarily suspending disbelief, let's allocate England an expected 1.5 goals and Brazil one goal. By accessing the Poisson distribution we learn that England would have a 22.3% chance of scoring no goals in the final, a 55.8% chance of scoring one goal or less and so on. Similarly, Brazil would have a 36.8% chance of scoring no goals, a 73.6% chance of scoring one goal or less, and so on.

From these statistics, we can provide an accurate estimate of the final itself ending with any particular scoreline. For example, the probability of a goalless final can be derived by multiplying the probability of England failing to score by the probability of Brazil failing to score, i.e. 0.223 multiplied by 0.368, or 8.2%.

That means a 91.8% chance, based on these assumptions, of at least one goal being scored in this World Cup Final between England and Brazil.

If only our assumptions were as accurate as the table!

The play was called 'Vortigern and Rowena', and was proclaimed as a lost work of William Shakespeare. With sceptics in the minority, chief among them being Shakespearean scholar, Edmond Malone, the play opened to a packed, enthusiastic audience on April 2, 1796. The part of Vortigern himself was played by no less a light than the fine Shakespearean actor, John Philip Kemble, brother of the legendary Sarah Siddons. He was also the manager of the Drury Lane Theatre.

The widespread excitement and anticipation among the audience soon turned, however, to bemusement and then literal disbelief, so that by the time Kemble was drawn to hint at his own opinion, repeating with emphasis Vortigern's line "and when this solemn mockery is o'er", the catcalls of the audience told its own story. The play ends with the entrance of the Fool, who admits that the play is not very tragic, as "none save bad do fall, which draws no tear." In fact, there were tears, but these were tears of laughter from those members of the audience charitable enough not to boo it off the stage. The real author, William Henry Ireland, soon admitted to the hoax and promptly left for France.

So here we have a 'lost play' by Shakespeare examined by notables of the day, most of whom were convinced of its authenticity, or at least willing to put their name to that belief. One performance before a crowd of ordinary theatregoers, however, was enough to kill off that notion and indeed kill off the play. Indeed, it was not to see the stage again for over 200 years, when it experienced a so-called 'comedic revival' on November 19, 2008, by the Pembroke Players at the Pembroke College New Cellars in Cambridge.

So what we can learn here about the 'wisdom of crowds'? Was it perhaps the case that Shakespeare is to be played, not read, and the 18th century experts who examined it simply took it on trust that it would appear better when played than read? Could it be that they were not so expert as they were given credit for? Could it be that the real experts were the performers who had played much of the canon of the authentic William Shakespeare, and that their sceptical performances tipped the wink to the theatre-going crowd?

Or could it be that the crowd simply is as wise as many give it credit for, especially when it has paid hard-earned money to get through the doors. More than 200 years on, we can't be sure what the 'Vortigern' fiasco tells us. But of one thing we can be sure. One crowd was enough. 'Vortigern and Rowena' didn't open for a second day.

But are they right?

To find out, I checked the prices on offer at my local branch of a well-known local supermarket chain and found the following pricing structure. A 460g bottle of a leading brand of tomato ketchup was priced at £1.63, while the bigger (by 73.9%) 800g bottle sold at £2.19 (an extra 34.4%). According to the LOOP principle, one might have thought that the 800g bottle would have sold for 73.9% more than the 460g bottle, i.e. for £2.83. So is this a mispricing of 64p. Does this indicate that the market is inefficient? Well, the answer is pretty simple here. There is nothing wrong with the market, since there's no clear way to exploit the mispricing, short of tipping the contents of the bigger bottle into the smaller bottles and selling them yourself. Summers would call this a "deviation due to transactions costs." More fundamentally, the smaller bottle offers advantages that the larger bottle doesn't have. Most obviously, it's easier to store. Perhaps it also looks nicer on the table.

Trading financial assets, on the other hand, is a different issue altogether. Transactions costs are relatively small and assets trading in different markets are often identical, so in these cases one would expect the LOOP principle to more clearly apply. What's the evidence? Well, one well-known apparent violation is the case of Royal Dutch Shell. Royal Dutch and Shell are separate legal entities but merged their interests in 1907 on a 60/40 basis. On this basis, the Royal Dutch shares should automatically have been priced at 50% more than Shell shares. However, they diverged from this by up to 15% until their final merger in 2005.

How about prediction markets? Is it possible to buy low and sell high across different prediction markets? Seems so! For an example, we need only point to the last US Presidential election when it was for several days possible to back John McCain on Betfair at a healthy shade of odds against and to do likewise with Barack Obama elsewhere. A guaranteed profit, even net of commission.

Professor Eugene Fama once defined an efficient market as one in which "deviations from the extreme version of the efficiency hypothesis are within information and transactions costs." On this basis, there would appear to be some evidence that markets (even prediction markets) are not always efficient. Come to think of it, maybe I'll buy that bigger bottle after all!