Depends on your definition of harm. I don't put food in the same category as drugs. But you should read The China Study and reach your own conclusions.
what's your definition of harm
does a meat based diet have a greater chance of causing harm (say heart disease, stroke, cancer,etc..) than smoking pot?
how about eating products made with pot butter or oil?
how about a strict vegan who eats the occasional pot brownie?