It's been known since at least 1914 -
"If there is any fault among colored people that needs an immediate remedy, it is a lack of punctuality, to learn how necessary it is to be on time, to be prompt and punctual in their engagements, to meet on time at social and public gatherings. It is perfectly absurd that so many of our people, sometimes unconsciously and sometimes willfully, wholly disregard the important fact of punctuality... We have in our method of gathering what is commonly called, in the vernacular of the street, 'colored people's time.' "
DW Johnson, Chicago Defender Newspaper 1914.
Willfully is the interesting part, some know white people hate it, so that's part of the joy

.