You've got Americans not in the labor force who are retired, who are underage, who are housewives, etc, as well as those normally in the workforce who have given up looking after many months or even years of unemployment.
On the flip side, you've got people taking part time jobs, jobs below their former skill and income levels, and otherwise being "underemployed". These people help bring the "unemployment rate" down, as do those who drop out of the workforce due to longterm unemployment.