I fundamentally disagree that consumers are responsible for the practices at any factory or system that produces goods - the factory/business owners are responsible, the blood is on their hands, not mine. For example, I've heard that many factories in China abuse workers - but I doubt anyone is giving up their electronic goods because of it - how would I even be able to know if my product has been produced unethically? No-one is saying I'm a bad person for buying electronic goods, so why am I being told I'm not a good person for buying meat and dairy?
Furthermore, the supply chain is extremely complex, so reducing buying choices down to good/bad is extremely reductive and does not help consumers make ethical choices, if this is even possible in the first place. We're all part of a complex system, it's bigger than any individual's choices.