Rambam, Mishneh Torah, Laws of Opinions 3:1

Let a person not say: envy, lust, honour and similar things are evil ways and remove a person from the world (note-quote from Ethics of the Fathers), and that one should separate oneself from them and go to the opposite extreme, not to eat meat, drink wine, marry, live in a nice house or dress in fine clothes, but dress only in sackcloth and hard wool, etc., like the idolatrous priests do. This too is an evil way and one who follows it is a sinner, as it is written regarding a nazirite, "...and atone for him for that which he sinned by the dead, etc"; the Sages said that if a nazirite, who only abstained from wine requires atonement, then how much more so anyone who abstains from all things requires atonement. Therefore, the Sages commanded that one should not abstain from anything, except from things which the Torah has forbidden, and one should not vow to refrain from using permitted things. The Sages said, `Is it not what the Torah has forbidden enough that you have to forbid for yourself other things?!'. According to this, those who always restrict themselves are not on the path of good. About these and similar things Solomon said, "Don't be excessively righteous, nor make yourself overwise; lest you destroy yourself?".