あの人にはとかくのうわさがあった。

There were some unsavory rumors about him.