SitePoint Sponsor

User Tag List

Results 1 to 9 of 9
  1. #1
    SitePoint Enthusiast
    Join Date
    Jan 2006
    Posts
    92
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    PHP Search Engine

    I am currently working on developing a search engine using PHP/MySQL. My goal is to try and develop a search engine similar to that of Google and/or Yahoo. To build my database of search results I want to allow users to submit their URL, and I will have a script that indexes their site and adds it to my database. My database has the following columns: id, title, url, description, and keywords. Does anyone know how I should index sites. Is this to be done with a PHP script or do I need to create an actual program to do this. Any help would be greatly appreciated.
    GeneralForum
    - To hell with the rest...

  2. #2
    Non-Member
    Join Date
    Jul 2005
    Posts
    0
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    you cant do it with php/ mysql, way too server heavy.youll need a spider to index sites

  3. #3
    SitePoint Member
    Join Date
    Nov 2006
    Posts
    8
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Writing a search engine is pretty involved. I suggest looking into open source projects before diving into coding. Google first!. I know phpdig uses mysql as backend engine. I also recommend you read about inverted indexes.

    good luck.

  4. #4
    SitePoint Enthusiast
    Join Date
    Jan 2006
    Posts
    92
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks for the help and also the good luck, but I want to develop some leading edge sites and play a major role in the web design industry so I am cut out for lots of work.
    GeneralForum
    - To hell with the rest...

  5. #5
    SitePoint Member
    Join Date
    Sep 2006
    Location
    Hot Sunny AZ
    Posts
    11
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Good luck and Mysql is the best ..BUT not for that kind of load as mentioned ..talk about slow and time outs ..it will kill your results at best .. Large jobs require larger solutions .. SQL.. Oracle and so on..

  6. #6
    SitePoint Enthusiast
    Join Date
    May 2006
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    first check what is done in open source search engine projects, them grab from them the best techniques and keep going

  7. #7
    SitePoint Enthusiast
    Join Date
    Jan 2006
    Posts
    92
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    So you think I need to use SQL or Oracle and stay away from MySQL for this project.
    GeneralForum
    - To hell with the rest...

  8. #8
    SitePoint Guru
    Join Date
    Jun 2001
    Location
    Australia
    Posts
    676
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    HighCaliber, how many listings are you thinking of getting into your database?

    I spidered 4 million urls last month on my dedicated daul core server. I get about 50,000 searchs a day. Its a small traffic amount but everyone has to start somewhere.

    Don't try and beat google, your only going to hurt yourself, instead make a small goal and go for it, then make a bigger goal etc.

  9. #9
    SitePoint Member
    Join Date
    Feb 2007
    Posts
    1
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dean Wilson, I am curious to learn how u spidered 4 million urls i am also looking to do something samller than this for my project. Cud u help me out here? I have no clue how to proceed here.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •