Should I tell my boss about this or leave it alone?

In the US, you can't say anything unless he touches you or starts saying things that make you uncomfortable. Just be wary of him whenever he is around though.