Elmina Castle
Older than Cape Coast and half an hour down the road. Most people do both in one day; do this one second.
St. George's Castle at Elmina was built by the Portuguese in 1482, which makes it the oldest European building south of the Sahara. It is a UNESCO World Heritage site and, like Cape Coast, it was a holding site for the transatlantic slave trade.
The fishing harbour below the castle is worth the extra twenty minutes on its own.
If you are doing both castles in one day, take Cape Coast first and Elmina second. Doing it the other way round means arriving at the heavier of the two already tired.