You've got too much time on your hands
is really just another way of saying “I don’t approve of how you’re spending your free time.”
We would probably do better to say “I think your hobby sucks” or “I think you’re a boring person” or “Your interests are clearly inferior to mine”
Or we would probably do better to find a better way to spend our own time.