I know exactly how you feel. I hate saying no to people. But your husband is right. It was not fair of them to only give you a 2 weeks notice. I know you would really love to do it but trying to squeeze it in may not be the best thing. If you rush you may not be able to put your best foot forward and the cake suffers. Plus like you said it would sit for 4 days before it is eaten. Probably not the best thing.
This is just my opinion as a hobby baker who very often has to try to finagle cake and work, but I hope this helps you out and good luck with the guilt :)
