The dorshe hamurot used to say, You shall surely destroy all the places where the nations whom thou shall dispossess served their gods, upon the high mountains and upon the hills, and under every green tree; you shall tear down their altars, and dash to pieces their pillars (Deut. 12:2-3). How did the wood and stones sin? But on account of them there came upon man confusion, and therefore Scripture said, You shall destroy their altars.
Here follows Yohanan b. Zakkai’s explanation
And behold it is a deduction: If in the case of stone and wood, which have neither merit nor demerit, neither good nor evil, because on their account confusion comes upon man, Scripture said to destroy their altars, a man who causes others to sin, and turns them from the way of life to the way of death, how much more so will he suffer.
Massekhet Semahot 8.7