The project was approved at a meeting of the Downs Committee yesterday.
Concerns raised by a campaign group that using fences would go against The Clifton and Durdham Downs Act was thrown out by the council's legal representative Frances Horner, who said that "temporary fencing" can be put up so that turf can recover from damage.
But members of Bristol Downs Watch told the Evening Post after the meeting at the Council House that they would be seeking legal advice on the matter because they believed the act referred to damage caused by horses and carriages.
Group member Roger Yates submitted a statement to the Downs Committee meeting on the matter of the goats and said that he did not believe the consultation had been candid.
Mr Yates said: "A period of five years is and was clearly not envisaged by the draftsmen of the act and certainly fencing to enclose the area for livestock was never intended."
The scheme will see about five or six male goats being introduced on the Bristol side of the gorge as part of the restoration of limestone grassland and rare plants.
Members of the Downs Committee, which is made up of the Lord Mayor, councillors and the master and six Merchant Venturers, were broadly supportive of the plan, although Brenda Hugill (Lab, Lawrence Hill) raised concerns about the possibility of poachers.
She said: "Someone I spoke to was quite scathing about the idea of goats on the Downs and said 'they will be eaten by a certain proportion of our population'. What are we doing about the possibility of poaching with these goats?"
Bristol City Council nature conservation officer Helen Hall, who presented the report to the committee, said the Downs Rangers will be responsible for checking on the goats daily.
The meeting also heard that there had not been any problems with poachers causing a problem with the deer at Ashton Court, so there was unlikely to be an issue with the goats.
Sylvia Townsend (Lib Dem, Redland), who was involved in drawing up the draft management plan for the Bristol side of the Avon Gorge, said: "We have taken a lot of time over it and thought very hard.
"The gorge is an amazing place and it is a changing landscape. I really think we have been exhaustive in going through all the possibilities there might be for the gorge. It has been hard work."
Peter Abraham (Con, Stoke Bishop), said: "This is something original we have come up with. It will be new to us here and I think we should go for it."
Anthony Brown, of the Merchant Venturers, said: "I think two further benefits from this will be greater public interest in the area of the Downs and that perhaps schools and so on may be more focused on coming up to look at the goats. They are going to be of huge interest to a lot of people."
After the meeting Mr Yates said: "We are very disappointed. We will be taking legal advice."
Work will now start to clear the Gully area of the Downs this winter, ready to make way for the temporary fencing to go up to mark out the goats' grazing area.