Council tax could be twice the price for second home owners in Swansea .
Proposals unveiled by the Labour administration also recommend a doubling of council tax for properties which have been empty for 12 months or more.
Council chiefs envisage implementing the empty homes proposal in April, 2020, and the second homes one a year later.
But a consultation will take place before any decision is made.
Council leader Rob Stewart said he believed the changes would help increase the housing supply as well as raising around £3 million per year - and bring Swansea into line with some other Welsh councils.
One estate agent said the second home proposal would be "a massive deterrent" to investors looking to buy a second home in Mumbles and Gower , especially following UK-wide increases in second home stamp duty.
There are around 1,400 long-term empty properties in Swansea, whose owners get a 50% council tax discount after an initial six-month exemption.
Under the new proposals, these owners would keep the initial exemption, then pay full council tax after six months, and then double the full council tax after 12 months.
Also in Swansea are some 1,800 second homes - mostly in Swansea West, Gower and Swansea Marina. Owners of these properties would pay double the normal council tax, making the average Band D property £3,200.
How they calculate council tax:
Council chiefs said some exemptions would apply to both types of properties, for example regarding those being actively marketed for sale.
The plans should not affect people who own more than one property if the house is occupied by a tenant, as the tenant would normally be liable to pay council tax. Nor would it affect holiday let owners where the property is subject to business rates.
There are currently around 4,300 people on the housing register in Swansea.
Cllr Stewart said: "One of the ways to address this problem is for the council to introduce a premium on council tax for such properties, the proceeds of which we can use to help bring empty properties back into use and meet local housing needs."
He said steps such as these were being encouraged by the Welsh Government.
Owners of empty homes can apply to the council for a loan to help bring the property back into use.
Julian Morgans, the authority's revenue and benefits manager, urged anyone facing council tax difficulties to seek advice.
"The first thing to do is get in touch with us," he said.
Nicola Rattenbury, branch partner of John Francis estate agent, Mumbles, said the second home proposal would particularly affect second homeowners who only used the property for a few weeks of the year.
"I think it would be a massive deterrent," she said.
Asked if it could make properties more affordable in the area by reducing demand, she said: "It could do, but you just don't know."
Chris Hope, senior partner at Dawsons estate agent, said the proposal could drive away investors who helped tourism by spending money in local restaurants and shops, and penalise people who had invested in an extra property for their retirement.
"Maybe the consultation period will bring this out openly," he said.
Mr Hope said he saw the rationale for the long-term empty homes proposal, as it gave owners an incentive to bring them back into use.
"That does make good sense," he said.
