Should I stop taking Letrozole due to side effects and dating concerns?

I have been taking Letrozole now for alittle over a year.. I keep thinking 4 more years to go! With the body aches, balance,dizziness and other symptoms I am thinking about going off of it..I am afraid to start another drug with new side effects…I feel like my old self when I take a week off of it for family visits or such. I want to get back into the dating world in a few months if my mindset is up for that? Any thoughts or advice or anyone feeling the same way?