tweets starting with @username are replies, and tweets with @username elsewhere are considered mentions. All can see your mentions - BUT replies can be seen by all your followers who also follow @username.
So, based on what you said, if I have this correct, If I reply to @abcxyz, followers of @abcxyz will see my reply. My reply to @abcxyz won't be seen by my followers?
Your reply to @abcxyz won't be seen by your followers, who are not following @abcxyz. It will be seen by all of your followers who also follow @abcxyz. Hope now things are clear for you.
It's similar to tweeting a normal message. Therefore, your twitter followers can view the tweet. ( But, if they follow 1000s of active accounts, the chances of them viewing your tweet is decreased, ) If you don't want your followers to view the message, send the person a direct message.