Well if you ask me, try to get back on google as well and try to get some visitors from yahoo too. You will do much better if you get traffic from both the search engines. Do whatever you can and have to do in order to get back in google.
I do not understand the logic behind this. Use the Google-Webmastertools for your website and you will see your backlinks. And that they do not send you vistitors is not related with the backlinks Google shows for you but probably with a drop in the SERps.