Why is so much difference in avg page load time in browsers

It may not be due to something yo are doing, but how the browsers work.
Though some problems in a site can slow some browsers more than others.
For example in this topic, a 404 on a linked css caused a massive slowdown in Firefox only.

If you don’t have any such problems on a site, that it’s probably not something you are doing, but is down to the browser developers. If your scripts or pages are inherently slow, they will be slow in all browsers.