The technology is not illegal but the uploads and downloads of pirated software or contents would be a very sticky issue. There are however cases in which running a torrent search engine can get you in trouble but if you have the financial and legal resources to fight in court, you could probably do it. See http://en.wikipedia.org/wiki/Legal_issues_with_BitTorrent before you decide.
Yes you can it depends where your server is hosted and the policies, content shared, disclaimer, and dedication to keep it as legal as possible. Based on history there has been legal action/tension against Torrent sites/search engines/hosting hosted in the United States. Just look at the infamous case of Torrent Spy being shut-down. The legal ‘heat’ is not as ‘hot’ with torrent and file sharing sites hosting on European, Caribbean, any other foreign land that are not directly/deeply influenced or interfered with the United States government/laws. This is not an advice to support and/or influence copyright infringement, no way, I am just being real and giving you information so you can make decisions. For example, the pirate bay is hosted in Sweden. You can run into legal issues in any territory no matter what so don’t ever feel secure as you’re going up against some pretty big guys when you allow users to upload content they’re “responsible†for. Make sure you can risk your whole pocket in any case because it would be a dog on the other side of the fence terrorizing to get over the fence to you case. All you have to do is look at history, search wikipedia and make your final decisions. Good luck with your ventures if you tread into such a task.