Ant Middleton was reportedly axed by Channel 4 after complaints were made by female staff over "inappropriate comments".
Ant was dropped earlier this week because of his "personal conduct".
And the decision came months after he was criticised for branding Black Lives Matter protesters “absolute scum" in June.
But the Mirror reports that his remarks on Black Lives Matter were not the only reason for the decision to axe him from the SAS Who Dares Win series.
A number of staff raised concerns with makers Minnow Films and Channel 4 during an end of year review where they were allowed to speak anonymously.
At the end of 2020, the allegations and complaints from a number of women regarding Ant’s comments towards them on set were then investigated. They felt the way he spoke to them was “inappropriate”.
Ant Middleton categorically denies the allegations.
Following the investigation, Channel 4 held meetings and decided to stop working with Ant, 40, and informed him of this decision on Monday.
The station told the Mirror: “We can’t comment on anonymous allegations, however, we take issues of welfare very seriously. We clearly set out our expectations of behaviour in our Supplier Code of Conduct. We are satisfied appropriate action has been taken.”
Minnow Films said: “The welfare of those working on our productions is of paramount importance. We have robust procedures in place to allow members of staff to come forward and confidentially raise any concerns.”
At the end of the 2019 series consultants investigated the production process.
Improvements were made but some staff were still left upset by the way Ant had treated them during filming last year. The decision to axe Middleton follows him losing several other jobs in the last year.
The ex-Special Forces soldier, who recently appeared with actress Rebel Wilson on his series Straight Talking, was removed from ITV’s Soccer Aid in August amid fears his involvement would offend ethnic minority players.
Keep up to date with local news in your area by adding your postcode below
And in the same month he stood down from the role of Chief Cadet of the Volunteer Cadet Corps, a youth group affiliated to the Royal Navy, after being appointed in November 2019.
Middleton was asked about leaving the show and said: “I think that ultimately what has happened is that the people that make the show and I have very different opinions on where the show started and where it is going in the future.”
A spokesman for Ant Middleton said: “Ant categorically denies your allegations. He is always disciplined and utterly professional.
"This is the first we have been informed of any such complaint which has never been mentioned previously to Ant or any of his team by Minnow or Channel 4.”
