I can't image that google will not use, users preferences to change search rankings. That they will assume that the search result is not relevant and lower the rank if someone clicks remove. Maybe you can use it make your ranking higher! You can click remove on your competition and promote on your own. Time will tell!