However, there are ways to set it up so that the user stays on your site and sees the results displayed on a page of your site. (I haven't tried this myself, though.)
That's right. In particular, you can customise the result page so that it shows your site's branding, including your logo, colour scheme, etc. - and also navigation to other pages on your site.
If you have a large site, your visitors will want a way to search it. You could add your own search box, with custom code to do the actual searching. Or you could use a third-party service (which might cost money). The advantage of adding a Google search box is that you get the full power of Google's search algorithms without having to write any code yourself, and for free. (The disadvantage is that the results page will carry ads, which might take visitors away from your site. But if you are an AdSense publisher, you can arrange for AdSense ads to appear there instead.)