Tuesday to Saturday, 10-6, free entry!
This is less obviously 'literary' than the other places on this list, but is worth a visit if you're in Brixton. The Black Cultural Archives aims to collect and preserve the history and culture of Black Britain, and is a pretty remarkable story. It started as a community project, and has remained rooted in that.
The building has a cafe and a small shop, and there's an exhibition room. The current exhibition is an interesting look at the past 100 years of Black British music. The next exhibition, starting in the spring of 2018, is of the Adamah family papers and promises a snapshot of life in the the first half of the 20th century.