“Understanding a particular time, a code of manners, how people were in a different era is always fascinating to me,” says Matthew Bourne. The choreographer is known for creating immersive worlds on stage. Whether it’s 1960s small-town America in The Car Man, a blitz-era Cinderella or the backstage life of an early 20th-century ballet company in The Red Shoes, Bourne’s eye for period detail is astute. But for those who only know his forays into the more fantastical worlds of Swan Lake or The Nutcracker, Bourne’s latest choice of subject may seem a departure.

The Midnight Bell is based on the novels of Patrick Hamilton, author of Hangover Square and 20,000 Streets Under the Sky, as well as the plays Rope and Gas Light (which both famously became films). Hamilton’s milieu is the bleak and seedy Soho of the 1930s, a land of characters who live in single rooms in boarding houses and gather in the pub for company, searching for connection. “It’s not the 1920s and 30s we tend to think of,” says Bourne. “It’s not Noël Coward and glamorous people in silk dresses with cocktails. The Hamilton novels feel so much like the authentic voice of real people. I love that, and I fell in love with the characters.

Hamilton’s own life underpinned his fiction. He was disfigured in a car accident in his 20s and his self-consciousness led him more and more to drink (his father, Bernard, was also a writer and alcoholic). He had an obsessive relationship with a young sex worker, a storyline mirrored in the book that gave Bourne’s show its title, 1929’s The Midnight Bell (the name of a pub). The reason Hamilton writes these worlds so well is because he was living in them. He wasn’t part of literary circles, which Bourne notes may be why he’s not better known today.

Bourne has brought together characters from a number of Hamilton’s stories, alongside invented “Hamiltonian characters”, 10 in all, and follows the journey of six, mostly doomed, relationships. One involves a character from one book, Miss Roach, having an adventure with a man from a different novel. Michela Meazza, a longtime collaborator of Bourne’s, plays fortysomething spinster Miss Roach. “Quite a mysterious character,” says Meazza, “she’s very contained, all the emotions are inside, there’s so much turmoil but she doesn’t explode.”

In past productions, Meazza and the dancers would have begun the creation process with films or images to evoke the period of the work, but this time they started with books full of underlined quotes. Taking dense texts full of internal monologues and turning them into externally visible stories was a real challenge. “At first we were wondering how to put it all into movement,” says Meazza, “but that’s the magic of Matthew. We start from the emotional state and then find the corresponding physical movement,” she says. “It’s not necessarily a dance step, it comes from how you walk, how heavy your feet are, how you sit, how you show internal conflict – is it the hands twisting?”

“There are these rather awkward, fumbling relationships – that was the movement quality I was interested in,” says Bourne. “I felt there was a lot of storytelling to be found in these characters. Regret and relationships that don’t work sear into people more than ones that do.”

In researching Hamilton and his times, Bourne sought out some of the pubs in the novels that still retain a period feel, like the the Fitzroy Tavern in Fitzrovia. The actual pub The Midnight Bell was based on no longer exists. He also unearthed archive song recordings from the 1930s that make up the soundtrack, alongside Terry Davies’s more contemporary score. “Matt was keen that we didn’t have a pastiche period score,” says Davies, “It’s more about the emotional journeys of the characters,” and the sleazy and morose moods of the pub and its denizens. “I’ve got quite a lot of dark sounds in the band,” he says, “I’ve got bassoon, contrabassoon, double bass, cello, low harp – unusual colours, I think. I’ve got some sweeter violins at the top to counterbalance it,” he says. “You have to grasp at the moments of lightness as well.”

Despite all the gloom, this land of lonely souls shouldn’t be alienating. “The relationships don’t turn out well but it’s a world that’s believable,” says Bourne. “It has aspects that probably we’ve all experienced at some point, stories people can relate to,” says Meazza. “Wanting love but finding a person who doesn’t have the same idea of a relationship; not knowing where you belong or how to move through this world.”

“With most pieces that I do there’s a universal truth in there which never really changes,” says Bourne. “People looking for love in a very broad sense, and trying to connect with each other. It’s universal themes, seen through the eyes of another time.”

• Matthew Bourne’s The Midnight Bell is on tour from 9 September to 27 November.