Hibiscus buds yellowing and dropping

Does anyone know why my hibiscus bushes develop loads of buds only to turn yellow and drop off. I am left with only 6 or so left. Oddly when they were in pots they did very well. Both get plenty of sunshine during the afternoon and evening.

Thanks

 

Could it be lack of water. Or perhaps they need a feed.

I have an Hibiscus in a pot, it's been there four years and although it's healthy has never flowered. I'm frightened to plant it out in my sandy soil, but think I will have to - to see if it will flower.