Riot police used rubber bullets and pepper spray to disperse an angry mob protesting against mandatory vaccines for construction workers outside the Melbourne headquarters of the CFMEU.
But the union says the protest was overtaken by "neo-Nazi's and right wing extremists", rather than CFMEU members.
Victoria Police say several people were arrested as the crowd "grew increasingly hostile" during the protest that began before 9am on Monday.
Hundreds of construction workers wearing high-vis, and their supporters, stormed the building, protesting against new mandatory vaccination rules for the building industry.
The protest escalated when two union officials, including Victorian construction branch secretary John Sekta, came outside the Elizabeth Street office to speak to protesters just before midday.
Mr Setka was met with boos and insults from the crowd, while some protesters hurled bottles.
"Please calm down. Can you at least give me the respect to talk? We're not the enemy, I don't know what you have heard," he told protesters.
"I have never, ever said I support mandatory vaccination."
Once Mr Setka went back inside, the protesters smashed a glass door to the building.
Later bottles were thrown at the already smashed glass entrance doors, as those protecting the entry sought refuge indoors.
Protesters also turned against each other, with a number of small fights breaking out within the crowd of bright orange and yellow.
The Victorian branch of the union released a statement just after 4pm, saying it has always supported freedom of choice regarding vaccination.
"We are not going to be intimidated by outside extremists attempting to intimidate the union, by spreading misinformation and lies about the union's position," the statement said.
"The CFMEU will always advocate for safety, jobs, and freedom of choice."
CFMEU national construction secretary released a statement on Monday night accusing the crowd of being "infiltrated by neo-Nazis and other right wing extremist groups".
"It is clear that a minority of those who participated were actual union members," he said.
Chanting "f*** the jab", those protesting are against having to show proof of their vaccination to be allowed back on building sites, with some saying they would rather the entire construction industry be shut down.
There is currently a 25 per cent cap on the industry's workforce in Victoria, with tradies required to have had their first dose of a vaccine by this Thursday in order to be allowed back to work.
Rain and hail did not deter the protesters, many still rallying in the streets until dark.
Some said they would come to the CFMEU office every day until the union bows to their demands.
It is unclear whether all of those protesting are construction workers, after a message on the Melbourne Freedom Rally Telegram group encouraged anti-lockdown protesters to join them.
Premier Daniel Andrews said the protests were "not smart, they are not safe".
"Protests don't work. Getting vaccinated works, following the rules works. That's how you stay open, that's how you get open," Mr Andrews said.
Monday's rally comes after construction workers set up plastic chairs and tables in the middle of streets across Melbourne on Friday, protesting against lockdown restrictions that included shutting down tea rooms for morning breaks.
