attracted to or by

I'm attracted to/by his charm/charms.

Which is correct?
Thanks.