Don’t want any confrontation with Governor but can’t compromise on rights: T.N. Minister