Boris Johnson's former standards chief has called for reform to the system, after quitting his post last year following the Priti Patel bullying probe.
Sir Alex Allan said the independent adviser should be given powers to launch probes into breaches of the ministerial code - and warned that allegations that have been "dismissed" could undermine public confidence.
Sir Alex resigned last year after Boris Johnson overruled his advice that Ms Patel - who has always denied allegations of bullying - had breached the ministerial code.
His report found the Home Secretary's conduct "amounted to behaviour that can be described as bullying", including incidents of "shouting and swearing" at staff.
Breach of the code normally triggers a minister's resignation but the Prime Minister, who is the code's arbiter, stood by Ms Patel.
Sir Alex, who was not questioned on specific cases, suggested there was a case for the independent adviser to be able to initiate investigations at a hearing of the Committee on Standards in Public Life on Wednesday.
It is currently up to the Prime Minister whether a potential breach is investigated.
"I think the issue really now is whether the process is actually damaging to the perception of whether ministers do or don't adhere to the code," he said.
"There are allegations of breaches that are essentially trivial or without substance.
"But there are also incidents which prima facie appear to involve a breach of the code but haven't been referred to the independent adviser.
"It's perfectly possible that the investigation would have revealed that the allegations weren't supported by the facts but the way the allegations have been dismissed has raised questions about the operation of the system and about public confidence, or the confidence the public can have in the impact and the effectiveness of the code.
"To that extent, I do see a case for introducing a greater element of independence. It would also clear up the anomaly where there were allegations against the prime minister him or herself which is something we have just seen in Scotland."
He said it was up to the Prime Minister to decide on the sanctions but suggested there should be a range of punishments, rather than the expectation that a minister would automatically resign.
Sir Alex said: "Currently if the PM feels that if the circumstances don't warrant a ministerial resignation, he may nonetheless feel under pressure to say that what happened wasn't a breach of the code because he doesn't think the minister needs to resign.
"I think it would help if there were a clear range of actions that could be taken following a breach."
Sir Alex also recalled how Ms Patel pressed him on his ability to question the PM during a committee hearing in 2012, due to his civil service background.
Most ministers want to abide by the rules, he said, but added that he does "recognise the concerns that have been put forward recently" that the upholding of standards has slipped.
Ministers were also warned that crises like coronavirus are "no excuse for poor behaviour".
Sir Alex said: "I think there is obviously a temptation to feel,'Oh God, the problems we are facing are such that let's just forget about some of the constraints and cut corners,' which is very dangerous."
Sir Philip Mawer, another ex-adviser on ministerial standards, added: "Crises are no excuse for poor behaviour."
