Repeatedly refusing short taxi trips or requests to turn on the meter will be enough for cabbies to be kicked to the curb, but more inspectors may be needed to enforce the crackdown.
Dodgy taxi and rideshare drivers face losing their licence to operate under Victorian "two strikes and you're out" laws similar to NSW.
The proposed legislative shake-up is set to give industry regulator Safe Transport Victoria greater powers to suspend or cancel the accreditation of drivers who have two fare-related convictions.
A pattern of complaints could also spell trouble for commercial passenger vehicle drivers, Victorian Public Transport Minister Gabrielle Williams signalled.
"There might be complaints made about an individual driver that are very similar but fall below that conviction threshold, it still allows the regulator to do something about it," she told reporters on Friday.
Other changes in the overhaul include compulsory QR codes in taxis and rideshare to link passengers through to a complaints page, harsher penalties for bad behaviour and mandating audio recordings in taxis.
The latter measure will make it easier for taxi operators to investigate complaints, the government said.
Carers will also be able to access live in-vehicle vision if offered by taxi or rideshare companies after a high-profile case of a cabbie abusing and scamming passengers with disabilities.
Ms Williams described the behaviour as "utterly unacceptable and disgusting".
The Transport Workers Union said the two-strikes policy unfairly targeted drivers and failed to address poor passenger behaviour.
"This is thoughtless demonising of transport workers," its national secretary Michael Kaine said.
But Victorian Taxi Association director Peter Valentine hailed the reforms, declaring mandatory audio in cabs would better protect drivers and passengers alike.
"Either one can be the offender," he told AAP.
Mr Valentine wants the video and audio mandate extended to Uber and DiDi operators despite drivers also using them as private cars.
He estimates there are roughly 90,000 commercial passenger vehicle drivers in the state and suggested passengers won't suffer if 10 to 15 per cent are shown the door following the crackdown.
But he concedes the regulator will likely face an uphill battle enforcing the incoming rules due to a lack of inspectors.
"If you've got 90,000 drivers, do you think three people can control them? There's half your problem," Mr Valentine said.
"You stand out there and blow your trumpet but it's no good unless the music comes out."
