CPU, RAM, bandwidth, etc aren't the only considerations. If you're pulling data from an external source, say Twitter and/or FB, you have to account for the time it takes to fetch that data from another site.
Or say you have a website that does a lot of writes to your database, and you've chosen to use MyISAM as your table type. MyISAM uses table level locking, so you can only perform one write at a time. This can easily make your site grind to a halt even if you have a high-end server. Yet if you use InnoDB, only the affected row is locked, so you can perform multiple writes at the same time. That would perform better even on lower level hardware.
Unoptimized DB queries with complex joins and unions can also be a huge bottleneck causing a site to run very slow when it needn't have to.
Sure, if you're making a simple site with nothing complex, and a a moderate amount of traffic, you don't have to worry about performance optimization very much. But not every project is the same and depending on the site/app, even an over-looked setting can cripple the fastest machines.