One of the reason is they are taunted as worshiper of penis in the form of shiva linga. But can someone tell me who said shivalinga means penis?
Shivalinga consist of two sanskrit words- shiva and lingam. Shiva means our very own lord Shiva and lingam means a symbol or representative in Sanskrit. So shivalinga is the representation of Shiva.
If it was penis then it would be called shiva-shishnam instead of shivalinga. Sanskrit word for penis is shishnam.
(P:S I, myself don't know sanskrit and I got meaning of lingam and shishnam from google search.)