Removal of Speaker is last option, need restraint: Trinamool