Woody Allen on his new film Magic in the Moonlight

Written and directed by Allen, this comedy tells the tale of a 1920s stage magician (Colin Firth) whose sober belief in the empirical world is sorely tested by an enticing younger woman (Stone) who claims to be a psychic medium.

Issues of artifice and uncertainty are pervasive in Allen’s work and life. Magic in the Moonlight is the latest of his films to exhibit his fascination with the early 20th century and to offer a philosophical arena where the forces of rationality and spirituality can battle it out, although it is no secret which side the author favours.

“I’m like Blanche DuBois,” says Allen, 78, sitting in a car park outside the cafe. “I hope in life that there’s a certain amount of magic. Unfortunately, there’s not enough. There are little, sporadic things one could think of as magical. But for the most part, it’s grim reality.”

Those remarks may simply reflect the comically crushing scepticism of Allen, who has considerable power to construct his own worlds, fictional and otherwise.

But the comments could also apply to the tumult of recent months, when he was confronted with past accusations he may have thought had vanished; and the reality that Magic in the Moonlight will test whether his audience has disappeared.

Magic has appeared in his movies in various guises, whether overtly, as in films such as Scoop (the 2006 movie in which he played a fumbling would-be Houdini), or obliquely, as in the 1985 comedy The Purple Rose of Cairo (in which a movie character steps off a screen into real life).

As an adolescent, Allen was obsessed with stage magic and taught himself card tricks and sleight-of-hand stunts, even performing them for small audiences. But, given what he calls “my inherent criminal personality,” Allen says, “I was interested in being a gambler, a card hustler, a dice hustler”.

Letty Aronson, Allen’s sister and longtime producer, says that her brother’s early interest in prestidigitation was a sign that he was “very, very observant of small things that happen in everyday life”.

As a film-maker, Allen has been a vocal disbeliever in a world beyond what is perceivable, as certain that there is no foundation for it as he is that such faith could never take root in him, although he holds a certain fascination with those who possess it.

“But if you’re the kind of person that finds it hard to deceive yourself – even though it’s seductive to believe the other thing – then you’re stuck with it,” he says. “The overwhelming amount of logic and evidence is that we’re all victims of a bad deal.”

The trials Firth’s character undergoes in Magic in the Moonlight – he yearns to find that Stone’s psychic is not a fake and even resorts to prayer in one desperate moment – might seem to indicate that Allen is more flexible about his beliefs than he is willing to acknowledge.

“Woody must at least understand that certainty is to be questioned,” Firth says. “Sometimes there are more paradoxes in good writing that are revealed about the person than they might consider.”

As further evidence, Firth cites a favourite line from Allen’s short story The Condemned, a pastiche of Camus existentialism: “Cloquet hated reality but realised it was still the only place to get a good steak.”

Firth says: “You can actually read a lot into that. You can escape and escape, but there are things in the world, hard facts and cause and effect, that you cannot deny.”

A streak of escapism might also account for Allen’s frequent return to pleasant depictions of the pre-Second World War decades, the setting not only of Magic but also of Radio Days, Bullets Over Broadway, Zelig and others.

Allen says this seeming preference is more about pragmatism – the most logical time when these stories would take place – and says he is not a nostalgic person.

“Nostalgia is a trap,” he adds. “It’s a pleasant, sticky substance, like honey, that you fall into.”