Making your site faster isn't *that* easy. Running on a dedicated server, site optimized from here to Tokyo, Google still claims 90+% other sites are faster.
Most likely because that dedicated server is being compared to companies that have multiple connections to the Internet using fiber lines instead of a single T1 or similar connection. Many also create server clusters meaning they have several severs sharing the traffic load instead of a single server. Having more resources they are able to separate web servers from database servers, again reducing the amount of processing that a single machine running both the database and web server would have to deal with.
Larger businesses are also able to dedicate personnel to tuning server side caching that will allow the servers to keep frequently used information in memory rather than having to pull it from hard drives or the database for every visit.
These are just a few examples of ways that larger businesses will gain a huge advantage over equally relevant websites that have less money and resources to compete. I am all for speeding up the Internet, but should a page that loads in 1 second be considered more relevant than a site that loads in 2 seconds?
My assumption is that load times will be just one more piece of the algorithm, not the determining factor in how a site ranks. The question is, how much weight is put into page load times compared to the others factors.