Now, though, it seems pretty understandable that he wouldn't want to hang around a whole lot. Maybe he's feeling very self-conscious around your family, or angry, or hurt.
You said that D can't just pretend like nothing happened. I think that you, your family, and D need to discuss this. The holidays are usually not the best time to bring about family confrontations, so after Thanksgiving get your family and D together and explain that you can't just go on pretending that there isn't tension between the family and D, and that D feels very uncomfortable/hurt/angry about the horrible things said behind his back.
Tell your family what you expect from them; An apology, or at least not making the comments around you (so that you do not have to struggle with whether or not you should repeat it), or bringing to light any real issues they have about D and not being two-faced about it...
Whatever your (or D's) expectations or wants may be, lay them out on the table and hopefully you and your family can talk things out. Be prepared for emotions running high, though. You might not get it worked out right away, but don't give up on finding a way in the long run. I hope my suggestions can help you!