"down the pub"

I know the word "down" can have different meanings: go down the street, go down south, etc. In British English it means "move away from the centre". However, the sentence "I bumped into him a couple of weeks ago down the pub and we had a drink" seems a bit unusual to me. What exactly does "down" mean here?