They have been able to hear it in St Thomas, Mount Pleasant and Sketty, and it reverberates across the bay to Mumbles when the wind direction is right - but just what is that banging sound?
People living in various parts of the city have been scratching their heads wondering what the rhythmic thumping noise actually is.
It sounds like something only large machinery could generate, since the noise is so loud and almost feels like it's sending vibrations through the ground.
The answer can be found at a construction site near Swansea Marina, where steel piles are being driven into the bed of the River Tawe where it empties into the sea.
The work is part of a £10 million rebuild of the Swansea West Pier inner wall, which is due to finish next June.
And you had better get used to the banging, as the piling is expected to continue through to mid-May, although by no means continuously through the day.
Get Swansea stories straight to your inbox with our newsletter.
A spokeswoman for Associated British Ports, which has commissioned the work, said operation times at the site were from 7am to 7pm, but that contractors would down tools from December 18 to January 3 to give marina and SA1 residents a break.
Becky Jones, the owner of Coast Cafe, which is a couple of hundred metres from West Pier, said of the piling work: "My parents live in Uplands and they can hear it.
"It's pretty loud. I notice it every now and again. I guess it's just something that has to be done."
She said updates were given to local residents and businesses about the ongoing work.
"It is a massive job," she said.
The 10-month project will shore up Swansea beach, which is just across from the river mouth, and maintain safe navigation for incoming and outgoing vessels.
The main West Pier itself was built in the 1800s. The inner wall has been deteriorating for years, requiring emergency repairs.
In August two people had to be rescued from the pier, which is closed off to the public, after being cut off by a particularly high tide.
