This happened to me totally by accident, but I was wondering if anyone else has done this. I activated a robot.txt file without knowing it. It was only on for 24 hours, but it was enough to make my site disappear off google. However when I removed it, my site was 15 spots higher in ranking, and it has stayed there for the last week. I have been working hard to build links etc. but my site hadn't budged. Has anyone else done this to "force" google to shuffle the order?
I would not say that this is a standard practice to shuffle the positions and I would not recommend anyone to even try this out. The reason your website got bump in the rankings has nothing to do with robots.txt, it is mere coincidence. The only reason your website got better positions is because you had worked hard to build the profiles and results started coming out at the time you removed robots.txt.
Yes, Google doesn't randomly change their algorithm. It must really because of your link building. Sometimes links are being backlinked later than when you made them. it even took few weeks or even a month to have links backlinked. So you hard work must paid off.
Is there really such a science. I just put Index Follow and thats it. Now if you look up luke martinez, i'm the first result. Even when there is a famous basketballer with the same name, I have reclaimed my name
robots.txt is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. You can't use this for boosting your SEO but you can block google to index unwanted directories,etc..
Just to make it clear to you that robots.txt has nothing to do directly with the rankings of your websites. As dir18 said, it is just set of instructions that let crawlers know what sections of a website they should follow and what to avoid. To improve, you should rather focus on the backlinks.