Agriculture Minister David Littleproud is afraid it could be too late to plug a farm worker shortage that has the potential to drive up fruit and vegetable prices.
Federal and state governments have been locked in a stalemate over sending overseas workers to pick produce during the coronavirus pandemic.
The federal government has pre-vetted 25,000 workers from 10 Pacific nations but some states have rejected proposals to allow them to quarantine on farms.
Asked whether time was up for harvests around Australia, Mr Littleproud said: "I fear sadly that it could be."
He said states had asked to keep control of quarantine arrangements through national cabinet.
"They've sat on their hands," Mr Littleproud told reporters in Canberra on Wednesday.
"They've been given that responsibility, that ownership, and they've done nothing with it."
Industry estimates put the cost of unpicked crops at more than $40 million nationally.
Shoppers could also feel pain from the impasse.
"Unfortunately what's going to happen for you in metropolitan Australia now is you're going to pay for it at the checkout," Mr Littleproud said.
"That's because you haven't valued what we do out there of putting bread and butter on your table every morning."
The federal minister said the issue was another example of city-focused pandemic responses.
"Unfortunately this has failed farmers. Federation has failed farmers where premiers have come out and talked big but actually haven't delivered."
Labor has called for the government to reconvene an agriculture ministers meeting with state counterparts despite the national cabinet quarantine arrangement.
Mr Littleproud praised the NT government for finding a way for Pacific workers to pick mangoes in September.
Those workers quarantined at the Howard Springs mining camp, the only operational quarantine facility that is a joint federal-state operation.
