Sounds cr*p. I think do whatever you think is best wrt your integrity. No more, no less. Understand those terrified feelings..but they go..wear high stiletto heels and sink it into his foot whenever he makes a remark (or maybe not..!!!)
Does the idiot have a boss you can report to? How bad is it?
Anyway..they get what comes to them at some point in time, so worry not.look at Roman Polanski......was that his name?
Your life is more important right and getting over those feeling terrified feelings..(aka give a sh*t)..
That said, give him hell if you want to, but do whatever is right for you.
tx