OMG! Just me? Or Pervert? Or Worse? HELP!?

You said more, or implied more, than what you are saying. If he thought he was a molester, I don't think he'd let his kids stay there. Unfortunately, it seems that the lead you to the conclusion that he's a molester. There is more to this than you are saying. It sounds like you either implied that he isn't protecting his kids from a ual molestation, and he took offense to it. Call him up and talk to him about it. Its all you can do. Ask yourself why he is keeping them away from you?