There is great value in being on the first page of Google for a competitive and targeted keyword or key phrase in your space. As a result, there is a cost to ranking well (or not ranking well) for any given competitive keyword. This is because if you are ranking at the top organically, you have likely invested in your website and its search engine optimization (SEO) strategy, whether it be with time or money. If you are ranking as a result of Adwords, then it is obvious how you got there- you paid money for it. Finally, and this is the one that is often overlooked and underestimated by many webmasters, if you are not ranking well you are paying for it in the opportunity you have missed- you have lost potential customers and sales by not being more competitive on that first page of rankings.
Either way, it is clear why SEO is important in order to keep you competitive and keep these costs manageable and the return profitable. One of the issues with the organic ranking strategy is that it is becoming less effective in some ways as Google has over time increased the number of paid listing up top and at the bottom, along with other search page changes including local listings, maps, defining content, etc. taking up valuable real estate and pushing down the organic results. As a result, in some ultra-competitive niches, it may be important to have both organic and paid rankings to maximize your return. Some users will always skip the paid rankings no matter what, so the organic component remains vital, but others may go for what is up top. A balanced approach seems more effective in these scenarios.
The part about ranking in paid results is straightforward- just pay for it and outbid your competition using the Adwords platform. Easy said, easy done. Moving on to organic results, it could not be more the opposite. Organic results are driven by many, many variables, most of which we likely do not and will not ever know about. The weights, the ways they are measured etc. is all a black hole minus the little bit we know from Google’s public announcements (them fully understanding the importance of information asymmetry to help them accomplish their own goals) and what we know from experimentation using imperfect experiments looking at correlations in ranking changes with changes in website characteristics. What we do know is that it is likely some combination of a variety of on-page and off-site factors including having good content, strong internal linking, and a favorable external link and social media profile (without overdoing it or turning to spammy or black hat tactics).
Google uses latent semantic indexing (LSI) which is a set of complicated rules which help the engine judge the content on any given web page. It looks for patterns which may suggest quality content vs. that which may just be generated for search engine gaming. This is great for those who are searching Google because it allows them to get the most likely relevant results. For example, a search for a mac likely will turn up the computer developer instead of macintosh apples. It is important to learn this key component of the Google system in order to be most effective in creating content for it. Ultimately, and especially over the long run, the safe quick rule is to create content that is engaging and demanded by your target audience.
Link building has historically been another important part of the search optimization game, but that is something which is extremely nuanced these days. Large numbers of links from similar, spammy, or websites where you can buy links are likely to have no impact if not a negative one on your rankings. This has been the case for a few years now. You really need highly relevant and quality links- preferably through organic means to have an impact these days.
The change that really kickstarted this was called Penguin by Google. Ever since Penguin, negative SEO has come into play as well. The effect and if it even exists is still a little contentious, but in theory, it is a way to bring down competitors by purposefully doing red flag external things to their websites such as building poor links, etc. You should be on the lookout for potential black hat competitors who are doing this to you and do your best to negate the damage. A great way to stay on top of this is to use the Google Webmasters Console which allows you to see a lot of information about your site including any particularly suspicious activity that Google notes. If you see a warning and it is due to negative SEO, you may be able to contest it or report it and get it removed.
SEO is an ever-changing game, so these key principles will not always be there. It is important to stay educated and vigilant for ultimate success.