Suppose I buy a domain name from X, and later on get hosting from Y, do I have to spend any extra money to transfer the domain name to Y? Is that how it works? I am confused about this.
You just need to change your domain's name server which provided by Y in X control panel. You don't need to transfer your domain to Y.
Usually all you need to do is change DNS entries to dns1.y.com and dns2.y.com as mentioned above. But some hosts insist you transfer the domain to them. Usually they wont comprimise due to automated systems. These hosters usually become the official holder of the domain, and they set the info for you. There are various reasons for this but if you decide to transfer you get another 12 months as well as the outstanding days left from x.