you may also find your site is set to also allow www. or not. e.g http://mysite.com and https://mysite.com as said above .htaccess will allow you to do a 301 permanent redirect to www. and https so google should then know which is the correct link.
as an aside google apparently compensates for the very slight increase in download time for a secure page over a non-secure page so your ranking won’t be affected, as google wants people to move to https