The men, aged 17, 20, 21 and 22, have been told this week that police will be taking no further action against them after prosecutors said officers did not have enough evidence to charge them.
It has been four months since the 40-bed Royal Pier Hotel in Birnbeck Road was torched by arsonists.
Jim Scott, of Esher in Surrey, who bought shares in the hotel, has criticised the police for failing to find the culprit.
He said: "It has been four months since this happened and still the person, or people, responsible have not been caught."
The fire caused major structural damage, after which the council served notice on the hotel's owners, saying it must be secured because the building posed a danger to people in the area.
This action was taken, but since then no steps have been taken to clear up the debris that remains.
Mr Scott says it is not his responsibility.
He added: "It is full of debris and in the winter months with strong winds all the loose parts of the roof could come flying off into the road.
"The hotel is in a terrible state but I have no control.
"I believed I had bought into the hotel two-and-a-half years ago but apparently I'm just an investor now and Wahid (Samady) won't give me my shares back.
"I invested half a million for shares in the company but he refuses to hand them over.
"Until the shares are handed to me the hotel is not my responsibility.
"He plans to submit a planning application this month to convert the site into flats, something I'm against.
"I wanted the site to offer some benefit to Westonians."
The Evening Post was unable to contact Mr Samady.
Wayne Baker, an Avon and Somerset Police spokesman, said: "No charges were brought against the four men and they have now been released.
"The CPS has said there was insufficient evidence to take the case any further."