Google have recently announced that page speed will be taken into account as a ranking factor, but it's just one of many. If your site always loads very slowly, then it could be having some effect, but is not necessarily the only reason. SERPs are not static, and many things can affect them.
I would suggest you might want to sort out any speed problems, not so much for the benefit of the search engines as for the benefit of your potential visitors. If your site takes too long to load, some of them are going to get fed up waiting and go elsewhere, so you're losing traffic.