Spend time with your dad doing things he needs to do/wants to get done and help him. You being a fur doesn't have anything to do with your dad "drifting away." You just don't spend enough time together. He doesn't talk about you being a fur a lot because why would he? What's so great and interesting about the fandom that he would know about/care to talk about?
If anything, you should be happy he's not telling you about how weird and outlandish the fandom is (which is probably what he thinks). He doesn't care about what you do with your time or what you like, he just wants his kid around to spend time with. Don't feel so insecure about your dad not accepting you because of a hobby you like. He's your dad, spend time building something together or going out and doing something crazy. Dad's love that stuff.
When the time comes, he'll learn that you're not as strange or untrustworthy as you perceive. He'll become more inclined to spend time with you on the things you like, including the fandom, if you show interest in his life as well.