Phillip Schofield told Care Minister Helen Whately that she appeared "cold and heartless" on This Morning today.
Today's show started with the story of retired nurse Ylenia Angeli,73, who had tried to take her 97-year-old mother out of a care home in Yorkshire.
Ylenia, who was visiting her mum with daughter Leandra Ashton, was so desperate to hug her mum that she stormed into the care home to hug her and ended up taking her away in the car.
The care home called the police and Ylenia was arrested for her actions.
Ylenia and Leandra appeared on This Morning to speak about their actions, which Ylenia said showed how "desperate" she was and how "inhumane" the care home visiting rules were.
Care Minister Helen Whately appeared on the show alongside the pair and Phillip asked her what her take on the situation was.
Ms Whately said she sympathised but couldn't speak about individual cases and that she was working on how to facilitate care home visiting.
But Phillip Schofield slammed her and said: "You have a 97-year-old lady who is in her final years.
"The family want to hug her, they want to be with her, damn the consequences.
"They want her, she needs them, you look cold, you look heartless.
"This is affecting people's mental health, there has to be a way around this."
Ms Whately came back and said: "I've sat in tears listening to the stories and have spent a huge amount of time to see how we can make this better.
Enter your postcode below to claim a free scratchcard
"But if there's a care home in an area with high levels of coronavirus, if someone visiting in that area does take covid into a care home its incredibly hard for it to stop spreading.
"I know how restricting visiting has hurt people, that's why we're publishing guidelines that enable visiting, we're doing this against a backdrop that has really really sadly seen 15,000 people in care homes die and many of them have dementia."
*This Morning continues at 10am on ITV tomorrow
