I may be in too deep- url scraping

[Chri]

I have a customer who wants to use a search script to search outside his site. How is this done? Am I talking about reinventing Google or what?

[adwatson]

Sort of depends on what they want to accomplish - would just using "site:" in a regular google search be good enough, or do they want to display results on their own site? If so, and if the other site has it's own internal search, it may be possible to use a middleman script that sends the query to the other site, grabs the results, and then parses them to display on the client's site.

[Chri]

They want to display on their own site. They want up-to-date results, daily I guess. They want to search within each link for keyword, not just titles or tags. It would be similar to asking someone to search sitepoint for the keyword, "php" and every record that contains php would be on the list.

[longneck]

is there a certain set of sites your customer wants to search? or do they want to search "all of the internet"?

[adwatson]

Perhaps google's ajax search API would help?

http://code.google.com/more/#products-search-ajaxsearch

[tobiaseichner]

Yes, I also would recommend not to write a Perl script but just use one of the big search engines and integrate them into your client's website.

Of course, using a CGI script, you could perform a Google search in the background, then with some regular expressions figure out the search results and display them in a custom format - but very likely this is against Google's use policy ;-) At least, unless you sign a contract with them paying for this service.

[UseShots]

I have a customer who wants to use a search script to search outside his site. How is this done? Am I talking about reinventing Google or what?

Why not use Google? I.e. AdSense for Search. It will search outside the site, it will display results on his site, and the site owner can earn some money. And it only takes about 5 minutes to add it.

[tobiaseichner]

Or does your client want something unlabelled, so used under their own name ? In this way, also Google can be your friend - they sell a search engine appliance. Very expensive, but may work. Check out their website for details.