"much difference" or "much of a difference"

Hi,

This may seem a silly question, but does it make any difference whether I say: "Size and taste aside, there isn't that much difference between a cucumber and a gherkin" or "(...) there isn't that much of a difference between a cucumber and a gherkin"? I hear both and tend to use them interchangeably.