A report on HMP Edinburgh called the prison "well-run" and "effective", but said the prison had a "general feeling from both staff and prisoners that it was unsafe".
In a report by HMIPS, the prison was rated as satisfactory, but the inspection had raised a number of issues.
One of the biggest issues highlighted was an ongoing staff shortage due to sickness and absences.
Not only did this affect prisoners through the closing of programmes and work sheds, but left both staff and prisoners feeling that it was "an unsafe environment due to staff shortages, violence and the increased use of psychoactive drugs".
Female prisoners were also less likely to have access to education programmes and work sheds than their male counterparts.
The report noted that senior management were aware and looking for alternative solutions, but that this had been an ongoing problem for over a year.
To receive one WhatsApp message a day with Edinburgh Live's headlines, as well as breaking news alerts, text NEWS to 07899067815. Then add the number to your contacts as 'Edinburgh Live'.
The report also highlighted that there was no evidence Think Twice - a Scottish Police Service (SPS) anti-bullying strategy - was being implemented, with concerns raised that staff were not challenging bullying behaviours nor supporting victims of bullying.
The prison was also recommended to re-introduce the First Night in Custody unit, as well as tightening procedures in the implementation of Talk to Me, the SPS suicide prevention strategy.
Inspectors also recommended a review of guidance, including clearing up ambiguities around use of force, and recommend the introduction of body cameras.
Wendy Sinclair-Gieben wrote in her report: "Staff/ prisoner relationships, which are at the heart of any prison, and were almost universally described by prisoners and staff as positive and respectful.
"Inspectors found good evidence of effective, courteous and humane use of authority and action to promote respect and protect against mistreatment.
"The processes in relation to ensuring lawful detention and liberation were also sound."
However, Ms Sinclair-Gieben added: "Without question staff shortages were adding to the natural logistical challenges in getting prisoners to and from work sheds, education, the library etc.
"Finding ways to avoid having to close work sheds at short notice would have a dramatic impact on the ability of the prison to provide the sort of structured regime needed to support rehabilitative activity.
"Our hope is that the compassionate responsible approach being adopted towards the issue of staff absence will succeed and address some of the challenges of staff shortages, allowing the prison to go from strength to strength and achieve its full potential in all areas."
