Boris Johnson has said that he doesn't want people bullied into taking the knee - when asked if he would take part in the gesture.
Mr Johnson also admitted he had work to do to get more black people into his cabinet.
Kneeling in protest for black people was popularised as a gesture in 2016 by former San Francisco 49ers quarterback Colin Kaepernick.
He knelt in protest against police brutality during the national anthem, prompting a furore over civil rights in the US.
It has since become a global gesture, with protesters - and even some police officers - joining the gesture.
Football players, politicians, protesters and celebrities have all made the gesture.
But police chiefs have said officers on duty should not make the gesture as it endangers fellow officers.
But Mr Johnson said he wants to see better black representation at the top of the Government.
"Of course we can do more and we will do more. We need to make progress and we will. I think about this a lot. It is something I want to get right. We need to reflect the country we serve," the Prime Minister told LBC radio.
He indicated that he would not be prepared to "take the knee" in support of the Black Lives Matter movement.
"I don't believe in gestures, I believe in substance," he said.
"I don't want people to be bullied into doing things they don't necessarily want to do.
"If you think of what happened with those officers standing at the Cenotaph... They were being insulted in quite aggressive terms by members of the crowd and told to take the knee.
"Some of them did and it was very difficult for the others who didn't. That's my position."
It comes after his Foreign Secretary had to clarify comments after he said he believed it was a sign of "subjugation" from the Game of Thrones.
Labour leader Keir Starmer took the knee as a gesture of solidarity with Black Lives Matter protestors.
