Not sure which is the toughest language for English speakers but it sure isn't Spanish or German. Personally, I've found Thai to be ridiculously hard;
I've taken 5 semesters of college German and a year of (nothing but) Russian at the Defense Language Institute in Monterey (I was a Russian voice interceptor, MOS 98G, in the army) so I've always considered myself pretty talented when it comes to languages but trying to learn Thai humbled me.
Holy fuck, tonal languages suuuuuck! Thai has 5 tones: low, medium, high, rising, and falling. So you can have a word like "ma" have 5 completely different meanings depending on your tone. I must say that the grammar is actually pretty simple (especially compared to Russian with its 6 cases) but then, when you try to learn thai's sanskrit-based writing system, it's all bad. I have at least 10 Thai language instruction books but none explain the writing system clearly enough for me to catch on. Oh well, guess I don't really need to communicate with my in-laws anyway, right?