Logic Behind Article Uniqueness?

[kabatak]

Hello does anybody know what's the logic behind determining the uniqueness of an article in percentage?

For example sentence A:
The quick brown fox jumped over the lazy dog.

When compared to sentence B:
The fast white fox leaped above the stubborn dog.

You could say that sentence B is about 45% unique. But how do you compute that mathematically using PHP perhaps? Any ideas? Thanks.

[Cups]

//For example sentence A:
$a = "The quick brown fox jumped over the lazy dog.";

//When compared to sentence B:
$b = "The fast white fox leaped above the stubborn dog.";

$base = explode(" ", $a) ;

$compare = explode(" ", $b) ;

$diffs = count( array_diff( $compare, $base ) ) ;

$origin = count( $base );

echo ($origin * $diffs)  ; 

= 45

I left it verbose so you can work out this way of doing it, there could be better ways.

[Mark Baker]

Doesn't consider position of words, but:

$sentenceA = 'The quick brown fox jumped over the lazy dog.';
$sA = str_word_count($sentenceA,1);

$sentenceB = 'The fast white fox leaped above the stubborn dog.';
$sB = str_word_count($sentenceB,1);

$xRef = array_intersect($sA,$sB);

$xResult = (count($xRef) / count($sA)) * 100;

print_r($xRef);

echo '<br />'.$xResult.'%'; 

[Trent Reimer]

PHP has some built-in comparison functions you might be able to take advantage of.

e.g. http://www.php.net/manual/en/function.similar-text.php

[Cups]

here could be better ways.

.. and there was. I think mine worked by accident ...

[kabatak]

Thanks guys. I'll work on those examples

[Cups]

SB

echo round($diffs / $origin * 100) ; 

Math brain fart, string b has 5 words different than string a = ~55%, no?

Coincidentally this also seems to work well if you mix the words round too, maybe that is not what you want though;

//For example sentence A:
$a = "The quick brown fox jumped over the lazy dog.";

//When compared to sentence B:
$b = "The fast white dog leaped above the stubborn fox."; 

= 78% wrong

[crmalibu]

http://tartarus.org/~martin/PorterStemmer/
There's a php implementation.

Using that would allow "jump", "jumped", and "jumping" to compare as the same word(but not jumper!). It's pretty cool.

It won't stem jumped and leaped to the same word, but it's probably a move in the right direction still.

[Mark Baker]

It won't stem jumped and leaped to the same word, but it's probably a move in the right direction still.

For that you'd need a porterstemasaurus.

The PHP implementation of the Porter stemming algorithm works pretty well for English language texts: I tested it out with some nice chunks of prose over my Christmas break.
Now you've temped me to try integrating it with a thesaurus to try and identify commonality of meaning even when the words used vary. Not sure of any practical applications, but a pleasant academic exercise.

I've heard rumours that there have been similar techniques developed for stemming in other languages (French has been mentioned several times) but haven't managed to find anything on the web about this other than speculations.