Completely agreed with you. Achieving ranking is all depend on a good neighborhood, though on page optimization is also important but not to that extant as explained by others. Quality backlinks are the most important factor in achieving high rankings. Though, there are so many other factors as well, like: content syndication, blogging, link bait, site architecture, speed of the website, quality of the content, how often you refresh the content of your website, etc.
Use the words that users are typing in the search engine boxes in your blog posts. Do not repeat the same keywords within a line. Don’t change URLs of your old posts. Use dashes to separate keywords in your URLs. You may use a full sentence in Alt tags. Use Wordpress for your blog. Do not alter the content you send to googlebot.