SitePoint Sponsor

User Tag List

Results 1 to 23 of 23
  1. #1
    SitePoint Guru dragonhawk's Avatar
    Join Date
    Apr 2002
    Location
    Melbourne
    Posts
    707
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Looping mail() for mailing list

    Have a question about the mail function.

    Right now I have a mailing list of around 300 people and this is how I set up my PHP code to mail each person on that list:

    PHP Code:
    $query="SELECT * FROM MailingList";
    $results=mysql_query($query);
    while (
    $row=mysql_fetch_array($results)) {
             
    $recipient "$row[Email]";
             
    $headers ="From: <me@myemail.com>";
             
    $subject "Subject here";
             
    $message "Message here";

             
    mail($recipient$subject$message$headers);

    For a mailing list of 300 people it would take about a 20 seconds to go through each person in the list. I'm just wondering if there is a way to speed something like that up? and not cause a spike in the server when the list is being sent to?

    For example, let's say I get it to 100,000 members or something like that. The server would pretty much crash when I try to send to the entire mailing list. Is there a way to prevent the server from crashing while mailing each member on the list through PHP?

  2. #2
    The short answer is yes... Herbster's Avatar
    Join Date
    Oct 2001
    Location
    Bay City, Oregon
    Posts
    715
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I don't know about speeding up a short list, but you could avoid server overload by putting a conditional sleep statement in there. If the number of records returned is greater than - whatever - sleep for a few seconds between emails.

    Also, you might want to check with your host as some have set policies on how many emails can be sent per day per hour etc.

  3. #3
    SitePoint Guru dragonhawk's Avatar
    Join Date
    Apr 2002
    Location
    Melbourne
    Posts
    707
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I'm on a dedicated server, so I'm not too fussed about the number of emails sent per day etc...

    Can you tell me more about this sleep statement? How would I go about incorporating it into that code?

  4. #4
    SitePoint Enthusiast
    Join Date
    May 2003
    Location
    boston
    Posts
    45
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    http://www.php.net/manual/en/function.sleep.php

    After the mail statement you could
    PHP Code:
    sleep(5); // sleep for five seconds 
    Or whatever value you want.

  5. #5
    SitePoint Guru dragonhawk's Avatar
    Join Date
    Apr 2002
    Location
    Melbourne
    Posts
    707
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    But wouldn't that mean that I have to stay on that same page until all the emails are sent? PHP will time-out on me if I have to wait for 500,000 seconds to deliver to 100,000 members...

  6. #6
    The short answer is yes... Herbster's Avatar
    Join Date
    Oct 2001
    Location
    Bay City, Oregon
    Posts
    715
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Your delay wouldn't have to be that long.
    As you send emails, do you update your db to indicate that the email has been sent? That way you can run the script more than once if it crashes without resending the emails.

  7. #7
    "Of" != "Have" bronze trophy Jeff Lange's Avatar
    Join Date
    Jan 2003
    Location
    Calgary, Canada
    Posts
    2,063
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    You could look into sending to a number of users at a time using multiple recipients and CC'ing the message, so like send to 40 users at a time.
    Who walks the stairs without a care
    It shoots so high in the sky.
    Bounce up and down just like a clown.
    Everyone knows its Slinky.

  8. #8
    SitePoint Enthusiast
    Join Date
    May 2003
    Location
    boston
    Posts
    45
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by dragonhawk
    But wouldn't that mean that I have to stay on that same page until all the emails are sent? PHP will time-out on me if I have to wait for 500,000 seconds to deliver to 100,000 members...
    I'd run the program via command line. Just ssh in and run it.

  9. #9
    SitePoint Zealot vbthanks's Avatar
    Join Date
    May 2001
    Location
    Sydney
    Posts
    170
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I've just written a php bulk mailer that works like this...
    I select the records from the main table and insert them in a 'queue' table with an auto-incrementing batchID. I loop through the queue table (limit 50), mailing each one, then deleting each one, then loop around to the next record, mail it then delete it. When there are zero rows left in the queue table, mail yourself a confirmation, then truncate the queue table to clean it up.
    This script is invoked by CRON every 60 seconds - this avoids the time-out error and free's the browser up to do other shtuff.
    Do some testing to see if it will in-fact send 50 emails per minute, otherwise you will have 2 instances of the send script running and you'll get mashed selects/deletes from the queue table. I am just fine tuning a workaround for this using a flag in a batch table whereby the script won't run if the previous 50 mails have not finished sending.
    Hope that helps - Mick
    "You know what you know - but that's all you know!"

  10. #10
    SitePoint Guru dragonhawk's Avatar
    Join Date
    Apr 2002
    Location
    Melbourne
    Posts
    707
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Yea, I was thinking about doing that, then realized that if there were 100,000 emails, and they are sent at 50 emails per minute, it would take 2,000 minutes to send (over 33 hours)...

  11. #11
    SitePoint Zealot vbthanks's Avatar
    Join Date
    May 2001
    Location
    Sydney
    Posts
    170
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Yes, I was using that as an example, we're doing 1000 per minute. But your mailing speed is basically determined by how many emails per minute the mailserver can pump out. That's where you are going to have to dome some timing testing.
    If you want to send 1000 per minute, you need 1 or maybe even 2 dedicated mail servers. Although I've never been able to get it to work, phpMailer is supposed to work ok and will handle multiple smtp boxes.
    One other thing - to set infinite time on a php script so it won't time out.... set_time_limit(0);
    Mick

    Mick
    "You know what you know - but that's all you know!"

  12. #12
    SitePoint Guru dragonhawk's Avatar
    Join Date
    Apr 2002
    Location
    Melbourne
    Posts
    707
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Yea, i tried it with set_time_limit(0) and the server spikes when mail is sent

    Oh well, I guess there is no real way to do it then... thanks for your help anyway... I'll probably just use set_time_limit(0) an put up a warning or something...

  13. #13
    SitePoint Zealot vbthanks's Avatar
    Join Date
    May 2001
    Location
    Sydney
    Posts
    170
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Because you're on a dedicated server, you have 3 choices...
    1. Do that timing testing I mentioned and set your limit there.
    2. Use the sleep() function to let the server take a breath in conjunction with set_time_limit(0)
    3. Get an additional Ded-Server and set it up as a mail server.
    With one box doing your web, mail, etc - the only way to lighten the load is to offload the mail handling to somewhere else and this can be done in php.ini
    Cheers,
    Mick
    "You know what you know - but that's all you know!"

  14. #14
    SitePoint Zealot vbthanks's Avatar
    Join Date
    May 2001
    Location
    Sydney
    Posts
    170
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Just had a thought - a cheap way to do this woul be to get a $50 per month cable account (with dedicated IP) - setup an old pentium, whack some mail server software on it and relay through that. (Ensuring it doesn't relay those spamming %$$#s from Florida! It should only relay from your webserver)
    Use Linux or Win - there's plenty of quality freeware/open source about. I haven't tried this - just a theory.
    Mike
    "You know what you know - but that's all you know!"

  15. #15
    SitePoint Guru dragonhawk's Avatar
    Join Date
    Apr 2002
    Location
    Melbourne
    Posts
    707
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hey! Thanks for the ideas!!

  16. #16
    SitePoint Member mlin's Avatar
    Join Date
    Apr 2003
    Location
    Inside the Matrix
    Posts
    15
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Work-around with Bcc

    I think there's no urgent need to execute a mail() for each subscriber individually. When looking through SourceForge's code I found a very smart solution for sending out a newsletter which might be lesser CPU-intensive. The entire site's v2.5 source code can be downloaded here:

    http://sourceforge.net/project/showf...lease_id=60003

    Although it's open source, hopefully I'm allowed to post a snippet here directly to give you an idea ..

    PHP Code:
    // ...some MySQL queries come here first...

    for ($i=0$i<$rows$i++) {
        
    $tolist .= db_result($res_mail,$i,'email').', ';
        if (
    $i 25 == 0) {
            echo 
    "\nUser id: ".db_result($res_mail,$i,'user_id');
            
    //spawn sendmail for 25 addresses at a time
            
    $body "To: noreply@$HTTP_HOST".
                
    "\nBCC: $tolist".
                
    "\nSubject: "stripslashes($mail_subject).
                
    "\n\n"stripslashes($mail_message);
            
    exec ("/bin/echo \""util_prep_string_for_sendmail($body) ."\" | /usr/sbin/sendmail -fnoreply@$HTTP_HOST -t -i >& /dev/null &");
            
    usleep(500000);
            print 
    "\nsending to $tolist";
            
    $tolist='';
            
    flush();
        }
    }

    //send the last of the messages.
    //spawn sendmail for 25 addresses at a time
    $body "To: noreply@$HTTP_HOST".
    "\nBCC: $tolist".
    "\nSubject: "stripslashes($mail_subject).
    "\n\n"stripslashes($mail_message);
    exec ("/bin/echo \""util_prep_string_for_sendmail($body) ."\" | /usr/sbin/sendmail -fnoreply@$HTTP_HOST -t -i >& /dev/null &");
    usleep(500000);
    print 
    "\nsending to $tolist";
    $tolist='';
    echo 
    "\n\n\nCOMPLETED SUCCESSFULLY";
    flush(); 

  17. #17
    SitePoint Enthusiast
    Join Date
    Mar 2002
    Posts
    29
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Actually I'm kinda curious... how would you manage to get hundreds, or even thousands of email addresses for these bulk mail development testing purposes?

  18. #18
    SitePoint Addict richard_h's Avatar
    Join Date
    May 2002
    Location
    London
    Posts
    301
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I've been writting something similiar and I'm using a combination of sleep() and set_time_limit(0) to send out the emails. I've posted the code here if you're interested - http://www.sitepointforums.com/showt...hreadid=108917 The fact you have a dedicated server suggests you a fewer problems than my shared environment, I would recommend the phpmailer class and as vbthanks mentioned you can specify multiple smtp servers to spread the load.

    You could look into sending to a number of users at a time using multiple recipients and CC'ing the message, so like send to 40 users at a time.
    Does this method including BCC'ing actual help in anyway? I was on the understanding this was all a myth.

  19. #19
    SitePoint Guru dragonhawk's Avatar
    Join Date
    Apr 2002
    Location
    Melbourne
    Posts
    707
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by arrowhead
    Actually I'm kinda curious... how would you manage to get hundreds, or even thousands of email addresses for these bulk mail development testing purposes?
    Since it's the sending that needs testing, I usually just sent the emails to myself (I create a special box for it). Then once it's all been sent, or it crashes, I delete that email box and re-create another one.

    richard_h, regarding your method of handling this, when you click the button on the browser to send the email to these thousands of members, won't you have to keep the browser open to continue sending the emails? Coz if you close the browser, won't the emails stop sending?

  20. #20
    SitePoint Addict richard_h's Avatar
    Join Date
    May 2002
    Location
    London
    Posts
    301
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by dragonhawk
    richard_h, regarding your method of handling this, when you click the button on the browser to send the email to these thousands of members, won't you have to keep the browser open to continue sending the emails? Coz if you close the browser, won't the emails stop sending?
    Yes this is correct, not having the luxury in this instance of a dedicated server my options are limited. It's obviously very important to be logging the success of the mailing by either writing to a file or email.

  21. #21
    SitePoint Wizard triexa's Avatar
    Join Date
    Dec 2002
    Location
    Canada
    Posts
    2,476
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by mlin
    I think there's no urgent need to execute a mail() for each subscriber individually. When looking through SourceForge's code I found a very smart solution for sending out a newsletter which might be lesser CPU-intensive. The entire site's v2.5 source code can be downloaded here:

    http://sourceforge.net/project/showf...lease_id=60003

    Although it's open source, hopefully I'm allowed to post a snippet here directly to give you an idea ..

    PHP Code:
    // ...some MySQL queries come here first...

    for ($i=0$i<$rows$i++) {
        
    $tolist .= db_result($res_mail,$i,'email').', ';
        if (
    $i 25 == 0) {
            echo 
    "\nUser id: ".db_result($res_mail,$i,'user_id');
            
    //spawn sendmail for 25 addresses at a time
            
    $body "To: noreply@$HTTP_HOST".
                
    "\nBCC: $tolist".
                
    "\nSubject: "stripslashes($mail_subject).
                
    "\n\n"stripslashes($mail_message);
            
    exec ("/bin/echo \""util_prep_string_for_sendmail($body) ."\" | /usr/sbin/sendmail -fnoreply@$HTTP_HOST -t -i >& /dev/null &");
            
    usleep(500000);
            print 
    "\nsending to $tolist";
            
    $tolist='';
            
    flush();
        }
    }

    //send the last of the messages.
    //spawn sendmail for 25 addresses at a time
    $body "To: noreply@$HTTP_HOST".
    "\nBCC: $tolist".
    "\nSubject: "stripslashes($mail_subject).
    "\n\n"stripslashes($mail_message);
    exec ("/bin/echo \""util_prep_string_for_sendmail($body) ."\" | /usr/sbin/sendmail -fnoreply@$HTTP_HOST -t -i >& /dev/null &");
    usleep(500000);
    print 
    "\nsending to $tolist";
    $tolist='';
    echo 
    "\n\n\nCOMPLETED SUCCESSFULLY";
    flush(); 

    The problem with CC or BCC, is that each subscriber is not individually in the To: field... and many readers, like hotmail, will filter this as junk by default.
    AskItOnline.com - Need answers? Ask it online.
    Create powerful online surveys with ease in minutes!
    Sign up for your FREE account today!
    Follow us on Twitter

  22. #22
    Non-Member Icheb's Avatar
    Join Date
    Mar 2003
    Location
    Germany
    Posts
    1,474
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by dragonhawk
    Coz if you close the browser, won't the emails stop sending?
    Not with ignore_user_abort() :-) .
    http://www.php.net/manual/en/functio...user-abort.php

  23. #23
    SitePoint Zealot vbthanks's Avatar
    Join Date
    May 2001
    Location
    Sydney
    Posts
    170
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hello,
    Sorry to revive this thread but thought I would as I've managed to get bulk email sending done and tested via php and mysql without any timeout problems and without being 'junk-filtered'.
    As far as 'how do you test this on bulk recipients' - this is a commercial app I wrote for my employer and has sent over 300,000 recipients without an error. (Not spam - double opt-in, legal, they want it)

    As I described earlier I needed to make sure cron did not invoke another instance of my gomail.php script if it was still sending. Despite resetting the status from 'ready' to 'sending' in the batch table after each 100 emails - there was the slight possibility that this clash would happen and it did. Not a cron clash - but a clash in the millisecond it took for the db update to take place. (sending to ready)
    To overcome this I used LOCK and UNLOCK to make sure the tables could not be queried whilst the status was being updated from 'sending' to 'ready'.
    I hope this gives some hints on how to do it, as described in my post earlier, and here's some help in the script that cron needs to invoke...

    PHP Code:
    <?
    include("conf.php");
    set_time_limit(0);
    $process=1;
    $db mysql_connect($dbhost$dbuser$dbpass);
    mysql_select_db($dbName,$db);

    // --------- CHECK IF THE FIRST BATCH IS READY TO SEND
    // --------- LOCK BATCH TABLE
    $sql = ("LOCK TABLE batch WRITE");
    mysql_query($sql) or die(mysql_error());
    $query = ("SELECT batchID,process FROM batch ORDER BY batchID LIMIT 1"); 
    $result mysql_query($query);
    while (
    $row mysql_fetch_array($result)) {
    $batchID=trim($row['batchID']);
    $process=trim($row['process']);
     }
    if (
    $process != "ready") {
    exit;
     }
    // ---------- BATCH IS READY TO SEND
     
    // ----------- UPDATE BATCH TABLE PROCESS TO SENDING
    $upd = ("UPDATE batch SET process = 'sending' WHERE batchID = $batchID");
    mysql_query($upd) or die(mysql_error());
    // --------- UNLOCK BATCH TABLE
    $unlock = ("UNLOCK TABLES");
    mysql_query($unlock) or die(mysql_error());

    // ---------- GET MESSAGE DATA AND PUT IN VARS
    $mquery = ("SELECT * FROM batch WHERE batchID = $batchID LIMIT 1");
    ......
    You will need 2 tables in addition to your subscriber table - batch (which holds send status, batch number, message, to, from, etc) and queue (which hold the recipients name and email)
    Hope that helps anyone who has ever tried this. It doesn't do the BCC thing, it doesn't do sockets - but depending on what you've got pointing to your SMTP/Mail server in php.ini - it will serve you well.
    Happy Days....
    Mike
    "You know what you know - but that's all you know!"


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •