Considering it took Van Richten decades to scrape enough of the material up to make one single arrowhead to kill a werebadger whose allergen was the mineral (pretty sure I'm fairly accurate on that), I think some sort of penalty increase seems fitting. You'd be hard pressed to get a piece of armour made of the stuff in even the most high magic of settings in your day to day adventuring life. If they've been moved to the rare drops then that's the ticket but I noticed a hell of a lot of them around.