Did you use both buttercream and ganache? I use only ganache and have never had that problem. If you used buttercream underneath the ganache, perhaps it is not letting the ganache adhere to the cake. If you only used ganache, my guess would be that the ganache sets too hard. Might need to add more cream.
