Yes they did, and a bloody good job they did too.
However compare their approach to how the Spanish built their empire. Most former colonies of the Spanish have nothing but hate for cruel Spanish, whereas many wish the Brits had never left.
The upper-class in England, ever since 1066, have been of predominantly Norman descent. (When they weren't imported from Hanover and other German principalities, that is!) The Anglo-Saxons were, for close to 500 years thereafter, considered an inferior peasant race, and intermarriage was strongly frowned upon.
For the most part, what remains of the Nobility are persons of Norman descent, and it is Normans who built the empire -- on the backs of their Anglo-Saxon sailors, laborers, and servants. WAPS gained a foothold in the Eastern USA, and accumulated enviable wealth and power, but did not build the old Empire.