1 ms roughly equals 1 foot of distance from a sound source. I don't play with my head inside the piano, or my ear place up against the Leslie or a monitor speaker. And my bandmates don't sit in my lap, (at least not when we are on stage performing.)
An issue bigger than CPU latency for many VIs is how tightly samples are trimmed at their front end. Playability vs Realism is at issue here. For this reason, I will sometimes perform parts with a tightly edited sample set, then replace with a set that gives us more attack detail.
When recording with mics, round trip latency is only an issue when monitoring thru a DAW. It can be especially bothersome for vocalists. I choose hardware monitoring for this reason.