SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Enthusiast
    Join Date
    Oct 2006
    Posts
    41
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    web scraping from a page that comes after using post method?

    Hi everyone,
    I am not much experienced in web development world and I don't know if it is even possible to even do it. Check out this page http://dsebd.org/news_archive.php
    click "search by symbol".
    The next page that comes up shows a "news database" of the symbol.
    I want to automatically read this information using regular expression and create an offline database of all the symbols.

    Sounds easy, I will run a loop that will go thru each symbol and read the values/ characters of its news page and store it in my database.
    The only problem is, the database page comes up from post method.

    So, I was wondering, pages that come up from after a "submit button push" i.e. "post method" not get method, is there a way to screen scrape that page ?

  2. #2
    Twitter: @AnthonySterling silver trophy AnthonySterling's Avatar
    Join Date
    Apr 2008
    Location
    North-East, UK.
    Posts
    6,111
    Mentioned
    3 Post(s)
    Tagged
    0 Thread(s)
    cURL provides this functionality in PHP, here's a link to the manual and a quick example.

    PHP Code:
    <?php
    function getSymbolData($sSymbol)
    {
        
    $rCurl curl_init('http://dsebd.org/old_news.php');
        
    curl_setopt($rCurlCURLOPT_POST,            true);
        
    curl_setopt($rCurlCURLOPT_RETURNTRANSFER,    true);
        
    curl_setopt($rCurlCURLOPT_POSTFIELDS,        array('cboSymbol'=>$sSymbol));
        return (String)
    curl_exec($rCurl);
    }

    echo 
    getSymbolData('1STBSRS');
    ?>
    @AnthonySterling: I'm a PHP developer, a consultant for oopnorth.com and the organiser of @phpne, a PHP User Group covering the North-East of England.

  3. #3
    SitePoint Enthusiast
    Join Date
    Oct 2006
    Posts
    41
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    woot! thanks buddy.

  4. #4
    Theoretical Physics Student bronze trophy Jake Arkinstall's Avatar
    Join Date
    May 2006
    Location
    Lancaster University, UK
    Posts
    7,062
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    You do have the legal permissions to do this, don't you?
    Jake Arkinstall
    "Sometimes you don't need to reinvent the wheel;
    Sometimes its enough to make that wheel more rounded"-Molona


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •