Please check the cURL functions and you can see what these can do ....
You can "forge" referrer , you can set any USER_AGENT you want.
You can parse the whole page and get caller to generate token for you and play the file remotely or even download.
As cURL can be set to accept sessions and cookies.
You will never know if it was user or not.
However there would be problem for cURL if caller.php is hidden in flash movie.
But crackers could decrypt the source code of flash movie and find the link, alter a bit parsed HTML site and call "caller.php" simply with an image this would trigger a session cookie.
Even YouTube can not protect their videos ...