Use of "one thing"

Hi

Is the underlined phrase natural English? Any suggestions are welcome.

[The speaker wants to say that the fact mentioned is important but is not the main reason for doing or not doing something.]

A - Are you disappointed because there wasn't much fun and frolic at the party last night?

B- Fun and frolic one thing, but the hosts really rubbed me the wrong way. They didn't know how to welcome their guests. 

A - You don't want to go because of the money?
B - Money one thing, but I've been really tired and I want to rest.

Thanks,

Tom