Yoga means heavy work (deed, kamma), it's related to yoke, carry a yoke. It's not about work or not, but about which work would lead to an end of work.
If body, and what comes along with it, is subject to break apart, worthy to seek union in what's subject to decay?
If feeling isn't lasting...