I've been using All In One SEO for awhile and I was thinking about switching to SEO Yoast but if i switch and uninstall All In One will this hurt my SEO?
You can import and export the settings of your current SEO plugin, install SEO by Yoast and import the settings of your old SEO plugin. Everything should be OK, still if you remove your plugin the information will stay in the database which means that if you re-install your old SEO plugin you should get all the settings back immediately (unless you also remove everything from the FTP Server)
I have done the same; moved from AIOSEO to SEO by Yoast and there is no change instead it has helped my blog rank higher for certain keywords. Good Luck