SitePoint Sponsor

User Tag List

Results 1 to 10 of 10
  1. #1
    SitePoint Addict svcghost's Avatar
    Join Date
    Oct 2010
    Posts
    288
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Firefox cURL can't access Yahoo.com but Chrome can

    Hey guys,

    When I try to browse to yahoo.com via cURL and Firefox, it does not work. When I try to browse to yahoo.com through Firefox alone, it does work. When I try to browse via cURL and Chrome, it does work. What gives? Why can't I browse with cURL passing along my actual Firefox useragent info, but I CAN with Chrome's useragent info?

    And why can I browse normally from Firefox just not through cURL?

  2. #2
    Theoretical Physics Student bronze trophy Jake Arkinstall's Avatar
    Join Date
    May 2006
    Location
    Lancaster University, UK
    Posts
    7,062
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Have you tried spoofing the useragent information rather than using the browser?
    Jake Arkinstall
    "Sometimes you don't need to reinvent the wheel;
    Sometimes its enough to make that wheel more rounded"-Molona

  3. #3
    SitePoint Addict svcghost's Avatar
    Join Date
    Oct 2010
    Posts
    288
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Yes. As a matter of fact, after lots of testing, I found that the site loads via cURL on Firefox ONLY when I set the user-agent to NOT include the word "Firefox" in the user-agent string. So
    Code:
    curl_setopt($curl, CURLOPT_USERAGENT, 'blahanything Firefox blah');
    does NOT work, but any user-agent string that does not have the string Firefox in it will work.

    Very very strange. But the weirder thing is, I can visit yahoo.com using Firefox without cURL, which has the string Firefox in it.. obviously.

    I am really just trying to get it so
    Code:
    curl_setopt($curl, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
    works. And as of right now, using Firefox through cURL doesn't work.


    P.S. on Chrome and Safari through cURL, I set the user-agent string to be a valid Firefox string and POOF, all of a sudden it doesn't work. So it definitely has to do with cURL + word "Firefox" in the user-agent string. Why?! I don't know!

  4. #4
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Is cURL outputting the user agent string exactly the same as your browser?

    Reason i ask is yahoo are known for not liking some browsers and if they can't recognise a browser (eg curl screws up the name or puts in a line terminator) then their configuration might not want to serve you / cURL.

    They don't like Seamonkey at all and refuse to allow it access to the new yahoo.
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  5. #5
    SitePoint Addict svcghost's Avatar
    Join Date
    Oct 2010
    Posts
    288
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    100% sure that the user-agent strings are the same. The Firefox through cURL string is the same as the Firefox alone string.

    This just makes no sense!

  6. #6
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by svcghost View Post
    This just makes no sense!

    No it doesn't does it.

    Perhaps you could post the headers from both your firefox and the cURL for us to look at? - By headers i mean the client and server headers. There must be something causing this.
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  7. #7
    SitePoint Addict svcghost's Avatar
    Join Date
    Oct 2010
    Posts
    288
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks for your help thus far in trying to figure this out with me. Is grabbing the client and server headers possible to achieve without using a packet sniffer like Wireshark?


    P.S.
    user-agent string Firefox through cURL:
    Code:
    Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
    user-agent string Firefox only
    Code:
    Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1

  8. #8
    SitePoint Guru aamonkey's Avatar
    Join Date
    Sep 2004
    Location
    kansas
    Posts
    953
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Yahoo seems to be ok as long as the Accept header is set:
    PHP Code:
    $headers = array ("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
    curl_setopt($chCURLOPT_HTTPHEADER$headers); 
    You can view what headers are being sent through the curl request by setting:

    PHP Code:
    curl_setopt($chCURLINFO_HEADER_OUT1); 
    then calling

    PHP Code:
    print_r(curl_getinfo($chCURLINFO_HEADER_OUT)); 
    after calling curl_exec()
    aaron-fisher.com - PHP articles and more

  9. #9
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by svcghost View Post
    Thanks for your help thus far in trying to figure this out with me. Is grabbing the client and server headers possible to achieve without using a packet sniffer like Wireshark?


    P.S.
    user-agent string Firefox through cURL:
    Code:
    Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
    user-agent string Firefox only
    Code:
    Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1
    Thats only the user-agent string. Re-read my previous request asking or both Client AND Server headers. Not header, headers.
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  10. #10
    SitePoint Addict svcghost's Avatar
    Join Date
    Oct 2010
    Posts
    288
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by aamonkey View Post
    Yahoo seems to be ok as long as the Accept header is set:
    PHP Code:
    $headers = array ("Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
    curl_setopt($chCURLOPT_HTTPHEADER$headers); 
    You can view what headers are being sent through the curl request by setting:

    PHP Code:
    curl_setopt($chCURLINFO_HEADER_OUT1); 
    then calling

    PHP Code:
    print_r(curl_getinfo($chCURLINFO_HEADER_OUT)); 
    after calling curl_exec()
    This is great. Thank you very much.
    So I still wonder why this wasn't working before when cURLing w/ Firefox. Maybe when you cURL with Firefox as the user-agent it doesn't have those Accept headers?? But with Chrome and Safari you do I guess. Weird.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •