How to code a crawl script that copies the first paragraph from Wikipedia?

[Sharkyx]

Daniel,

Your code is wonderful and hits the spot, but it’s not there yet.

1. If you type in a more than one word parsing phrase, it won’t return anything.

example “new york” won’t show anything.

then “new_york” will show you that you need to redirect and use proper capitalization.

ultimately, “New_York” will show you the proper display.

2. I’d only like to show the first <p> from the a wikipedia listing. this will put less strain on my server as well as wikipedia’s. and, yes, i’ve downloaded to wikipedia multiple times now, whenever a call out was made.

A perfect example of what I’m trying to replicate can be seen at PhysOrg.com - Science News, Technology, Physics, Nanotechnology, Space Science, Earth Science, Medicine.

Just click on any of their post, check out the right sidebar and click on tag or more.

CLEAR EXAMPLE: PhysOrg.com - magnetic field

thank you everyone for your help.

@Daniel15

In that case I’d check out their API. The “parse” action will parse a page and return the HTML for that page. Example URLs:

http://en.wikipedia.org/w/api.php?action=parse&page=SitePoint&format=php - Data for “SitePoint” page in PHP serialised format - to use with unserialize() in PHP.
http://en.wikipedia.org/w/api.php?action=parse&page=SitePoint&format=xml - Data for “SitePoint” page in XML format

No scraping needed, the data is in an easy-to-use format for you.

I’d strongly recommend donating to Wikipedia if you use its data extensively. High usage of their servers means that your scraping costs them quite a bit of money (bandwidth, server processing time, etc.)

Here’s an example for you (PHP):

<?php
$page = 'SitePoint';
$api_url = 'http://en.wikipedia.org/w/api.php?action=parse&page=%s&format=php';

// MediaWiki API needs a user-agent to be specified
$context = stream_context_create(array('http' => array(
	'user_agent' => 'SitePoint example for topic 748667',
)));

$data = unserialize(file_get_contents(sprintf($api_url, $page), null, $context));

echo $data['parse']['text']['*'];
?>