Prince Harry says it is unsafe to travel to the UK with Meghan Markle and their two young children due to insufficient security.
The Duke of Sussex offered to personally pay for UK police protection without taxpayer money - a request the Home Office denied.
He is now seeking a judicial review of the refusal, The Mirror reported.
Prince Harry cited "well-documented neo-Nazi and extremist threats" against his family, revealing his security was "compromised" in a visit to the UK last summer.
A private security team cannot replicate the level of security and access to UK intelligence information, said a statement from Prince Harry.
Without police protection, lawyers said that bringing Meghan and their children over comes at "too great a personal risk".
As a result, the 37-year-old says he is unable to bring three-year-old Archie and seven-month-old Lilibet to the UK.
Megan and Archie have not returned to the UK since the couple walked away from royal life in 2019, while Lilibet has yet to set foot in the country where her dad was born.
Lawyers acting for the Duke of Sussex say they will seek a judicial review if continued security is not provided when he and his family visit.
In a statement his legal team said Harry's security was "compromised" last summer when he unveiled a statue of his mother, Princess Diana, at Kensington Palace.
It said that Harry - who "inherited a security threat at birth" - is "unable to return to his home" with his family, saying they have been subjected to "well-documented neo-Nazi and extremist threats".
The Duke has returned to the UK twice in the last two years - for the statue unveiling in July last year and Prince Philip's funeral three months earlier - but Meghan and the children have stayed in the US both times.
Harry is arguing his private protection team in the US does not have adequate jurisdiction abroad or access to UK intelligence information which is needed keep the Sussex family safe.
It added that he had first offered to pay for police protection for himself and his family in January 2020, but the offer was not accepted.
During last summer's visit to London, Harry's car was chased by photographers - an incident that has made him anxious about his safety.
A source told the Mail on Sunday: 'Harry's argument in a nutshell is: 'You got the law wrong.'
"He feels the decision to remove his security was wrong.
"Pre-action protocol was sent by Harry's lawyers to the Home Office a couple of months ago. This is essentially a precursor to a judicial review."
If the case goes forward it could lead to a legal battle in the High Court between Government ministers and the Duke.
The statement in full
"Prince Harry inherited a security risk at birth, for life. He remains sixth in line to the throne, served two tours of combat duty in Afghanistan, and in recent years his family has been subjected to well-documented neo-Nazi and extremist threats.
"While his role within the institution has changed, his profile as a member of the royal family has not. Nor has the threat to him and his family.
"The Duke and Duchess of Sussex personally fund a private security team for their family, yet that security cannot replicate the necessary police protection needed whilst in the UK. In the absence of such protection, Prince Harry and his family are unable to return to his home.
"The duke first offered to pay personally for UK police protection for himself and his family in January of 2020 at Sandringham. That offer was dismissed. He remains willing to cover the cost of security, as not to impose on the British taxpayer.
"As is widely known, others who have left public office and have an inherent threat risk receive police protection at no cost to them. The goal for Prince Harry has been simple - to ensure the safety of himself and his family while in the UK so his children can know his home country.
"During his last visit to the UK in July 2021 - to unveil a statue in honour of his late mother - his security was compromised due to the absence of police protection, whilst leaving a charity event.
"After another attempt at negotiations was also rejected, he sought a judicial review in September 2021 to challenge the decision-making behind the security procedures, in the hopes that this could be re-evaluated for the obvious and necessary protection required.
"The UK will always be Prince Harry's home and a country he wants his wife and children to be safe in. With the lack of police protection, comes too great a personal risk.
"Prince Harry hopes that his petition - after close to two years of pleas for security in the UK - will resolve this situation. It is due to a leak in a UK tabloid, with surreptitious timing, we feel it necessary to release a statement setting the facts straight."
