Try to understand and implement Long Polling

donboe · August 18, 2014, 5:46pm

I am working on a small news website and try to understand and implement Long Polling. I found the following example what actually comes very near to what my requirements are:


<script type="text/javascript" charset="utf-8">
    function addmsg(type, msg){
        $("#messages").append(
            "<div class='msg "+ type +"'>"+ msg +"</div>"
        );
    }
    function waitForMsg(){
        $.ajax({
            type: "GET",
            url: "news.php",
            async: true,
            cache: false,
            timeout:50000,

            success: function(data){
                addmsg("new", data);
                setTimeout(
                    waitForMsg,
                    1000
                );
            },
            error: function(XMLHttpRequest, textStatus, errorThrown){
                addmsg("error", textStatus + " (" + errorThrown + ")");
                setTimeout(
                    waitForMsg,
                    15000);
            }
        });
    };
    $(document).ready(function(){
        waitForMsg();
    });
</script>

And news.php contains the following::


    require_once('connect.php');
    $query  = "SELECT * FROM news WHERE isNew = 1";
    $result = $pdo->query($query);
	
	while($row = $result->fetch()) {
		echo $row['news_contentt'];
	}

And the PHP part is what is breaking me up Since the isNew status will remain 1 untill the mesage is answerd, So the setTimeout function

setTimeout(waitForMsg,15000);}

will be repeated all the time until the isNew status is set to 0. What would be your sollution to solve this?

donboe · August 19, 2014, 5:41pm

I have been searching and trying all kind of things all day but so far without any success. I was wondering if there is no way to pause the waitForMsg() function once the while loop has deliverd the result set?

fretburner · August 19, 2014, 7:39pm

Hey donboe,

If your news items had a timestamp column for the time they are added to the system, you could have your AJAX code submit the timestamp of the last request and simply return anything newer than that. Alternatively, if your news table uses sequential IDs, keep track of the last ID and poll for anything with a greater ID.

donboe · August 19, 2014, 8:31pm

Hi fretburner. Thanks for the reply I have solved the repeated messages issue server side, by doing a update query on the isNew status within the while loop. It indeed shows the messages just one time but somehow the function is stil executing because after a while a scroll bar appears in the browser

I have a timestamp in the table news, called add_date, So how do I integrate that in the function as it is? Because if that solves the former issue I would be more than happy

How and where do I poll for a greater vallue.

I have another question if I may. In a lot of examples I see the use of echo json_encode($data); or echo $json;

Like I said in the opening post this is all very new for me and I really try and like to understand how this all works but it is confusing me somehow. Any advise or tutorial would be really appreciated.

Thank you in advance

fretburner · August 19, 2014, 11:29pm

Fair enough, in this case the problem is that when there is no news returned from the server your polling function is still adding a div to the page. You just need to add a check to see if anything was returned from your AJAX call:

success: function(data){ 
    if (data) {
    	addmsg("new", data);
	}
    setTimeout(
        waitForMsg, 
        1000 
    );
},

json_encode is useful when you want to return arrays or objects to your JS, rather than plain text or HTML. It’s quite common for APIs to return data as JSON, and for apps to generate the HTML client-side.

Here’s how your code might look if you were returning JSON and using the timestamp method of checking for updates:

news.php


&lt;?php

require_once('connect.php');

$query  = "SELECT * FROM news WHERE isNew = 1";

if (empty($_GET['since'])) {
    $result = $pdo-&gt;query($query);
} else {
    $query .= ' AND add_date &gt; :since';
    $stmt = $pdo-&gt;prepare($query);
    $result = $stmt-&gt;execute(array(':since' =&gt; $_GET['since']));
}
 
$data = $result-&gt;fetchAll(PDO::FETCH_ASSOC);

header('Content-Type: application/json');
echo json_encode(array(
        'news'        =&gt; $data,
        'lastChecked' =&gt; time()
    ));

Javascript:

(function() {
    var lastChecked = '';

    function addmsg(type, msg) {
        $("#messages").append(
            "&lt;div class='msg "+ type +"'&gt;"+ msg +"&lt;/div&gt;"
        );
    };

    function waitForMsg() {
        $.ajax({
            type: "GET",
            url: "news.php?since="+lastChecked,
            async: true,
            cache: false,
            timeout:50000,

            success: function(data) { 
                if (data.news.length &gt; 0) {
                    lastChecked = data.lastChecked;
                    data.news.forEach(function(item) {
                        addmsg("new", item.news_contentt);
                    });
                }
                setTimeout(
                    waitForMsg, 
                    1000 
                );
            },
            error: function(XMLHttpRequest, textStatus, errorThrown){
                addmsg("error", textStatus + " (" + errorThrown + ")");
                setTimeout(
                    waitForMsg,
                    15000);
            }
        });
    };

    waitForMsg();
})();

Admittedly, it’s extra complexity for little benefit in this particular example.

donboe · August 20, 2014, 10:56am

Hi fretburner. Again thank you for your reply and explanation, it is realy appreciated.

fretburner:

Fair enough, in this case the problem is that when there is no news returned from the server your polling function is still adding a div to the page. You just need to add a check to see if anything was returned from your AJAX call:
success: function(data){ 
    if (data) {
    	addmsg("new", data);
	}
    setTimeout(
        waitForMsg, 
        1000 
    );
}, 
.

Your explanation is actually quite logical, so I have changed the success function to your suggestion, but for some reason the scroll bar still appears after a certain amount of time. Next to the div that is appended what could be the reason?

fretburner:

json_encode is useful when you want to return arrays or objects to your JS, rather than plain text or HTML. It’s quite common for APIs to return data as JSON, and for apps to generate the HTML client-side.

Here’s how your code might look if you were returning JSON and using the timestamp method of checking for updates:

news.php


&lt;?php

require_once('connect.php');

$query  = "SELECT * FROM news WHERE isNew = 1";

if (empty($_GET['since'])) {
    $result = $pdo-&gt;query($query);
} else {
    $query .= ' AND add_date &gt; :since';
    $stmt = $pdo-&gt;prepare($query);
    $result = $stmt-&gt;execute(array(':since' =&gt; $_GET['since']));
}
 
$data = $result-&gt;fetchAll(PDO::FETCH_ASSOC);

header('Content-Type: application/json');
echo json_encode(array(
        'news'        =&gt; $data,
        'lastChecked' =&gt; time()
    ));

Javascript:

(function() {
    var lastChecked = '';

    function addmsg(type, msg) {
        $("#messages").append(
            "&lt;div class='msg "+ type +"'&gt;"+ msg +"&lt;/div&gt;"
        );
    };

    function waitForMsg() {
        $.ajax({
            type: "GET",
            url: "news.php?since="+lastChecked,
            async: true,
            cache: false,
            timeout:50000,

            success: function(data) { 
                if (data.news.length &gt; 0) {
                    lastChecked = data.lastChecked;
                    data.news.forEach(function(item) {
                        addmsg("new", item.news_contentt);
                    });
                }
                setTimeout(
                    waitForMsg, 
                    1000 
                );
            },
            error: function(XMLHttpRequest, textStatus, errorThrown){
                addmsg("error", textStatus + " (" + errorThrown + ")");
                setTimeout(
                    waitForMsg,
                    15000);
            }
        });
    };

    waitForMsg();
})();

Admittedly, it’s extra complexity for little benefit in this particular example.

I need to have a look at this in more detail because I don’t want to simply copy this and use it, I also would like to understand the functionality.

fretburner · August 20, 2014, 11:15am

It sounds like maybe your PHP script is returning something that evaluates as truthy. Is there more to news.php than you previously posted, or that was the whole script?

donboe · August 21, 2014, 1:23pm

Hi fretburner. maybe I should have mentioned this. At the very top of news.php I have another query which actually holds the same values, but Is only used to extract certain variables which I need on another pages, so this is the complete news.php



require_once('connect.php');
   
$qry = "SELECT MIN(news_id) AS news_id
             , sender_id
             , add_date
          FROM news 
         WHERE isNew = 1";

$stmt = $pdo-&gt;query($qry);
$vars = $stmt-&gt;fetch();

$news_id   = $vars['news_id'];
$sender_id = $vars['sender_id'];
$add_date  = $vars['add_date'];

    
$query  = "SELECT * FROM news WHERE isNew = 1"; 
$result = $pdo-&gt;query($query);
    
while($row = $result-&gt;fetch()) {
    echo "&lt;p&gt;"{$row['news_content']}"&lt;/p&gt;;
}

Could this be of influence on the scroll bar issue or should I approach this completely different?

Thank you in advance

fretburner · August 21, 2014, 2:11pm

This script doesn’t update the isNew field though, so it’ll still be returning the same items every time it’s called?

donboe · August 21, 2014, 3:39pm

Hi fretburner. The update isn’t there as yet, because I am still working on that part See topic

donboe · August 21, 2014, 9:28pm

I now have added the update query to the while loop so now the complete news.php looks like this:


// This query I use to set certain variables which I need elseware
$qry = "SELECT MIN(news_id) AS news_id
             , sender_id
             , add_date
          FROM news 
         WHERE isNew = 1";

$stmt = $pdo-&gt;query($qry);
$vars = $stmt-&gt;fetch();

$news_id   = $vars['news_id'];
$sender_id = $vars['sender_id'];
$add_date  = $vars['add_date'];

// With this query I fetch the oldest news item with status isNew = 1 and all other news items from this sender
$sql = "SELECT news_id
             , sender_id
             , news_content
             , add_date
          FROM news
         WHERE sender_id =
               (SELECT sender_id
                  FROM news
                 WHERE add_date =
                       (SELECT MIN(add_date)
                          FROM news
                         WHERE isNew = 1))
         ORDER 
            BY add_date DESC";

$result = $pdo-&gt;query($sql);

$news_id = 0;
while($row = $result-&gt;fetch()) {
    if ($row['news_id'] &gt; $news_id) {
        $news_id = $row['news_id'];
    } 
    echo "&lt;p&gt;{$row['news_content']}&lt;/p&gt;";
    $qry = "UPDATE news SET isNew = 0 WHERE news_id = ?";
    $stmt = $pdo-&gt;prepare($qry);
    $stmt-&gt;execute(array($news_id));		
}

I still have the issue with the scrollbar though. and as you can see is it just a paragraph I use for the news_content. I still have no idea what could be the reason. And then there is another thing I need to solve. Since I fetch the oldest news item with staus isNew = 1 there will be quite possibly be more recent news items with status isNew = 1 but then from other senders. Like the structure is now, if there are more recent news items with status isNew = 1 from other users, they are loaded as well, while I am looking for a way to limit the listing with news items per sender per time. What would be in your opinion the best way to accomplish that? I have a comment field where other users can leave a comment, could that be a possibility?

Thank you in advance

donboe · August 22, 2014, 9:54am

Hi fretburner

I got this solved I changed:


    if (data) {
    	addmsg("new", data);
	}
    setTimeout(
        waitForMsg,
        1000
    );

into


   if(data) {
        addmsg(data)
   } else {
        setTimeout(waitForMsg, 1000);
   };

And it is working fine. Thank you so much for all the input.

fretburner · August 22, 2014, 10:55am

donboe:

I got this solved I changed:


    if (data) {
    	addmsg("new", data);
	}
    setTimeout(
        waitForMsg, 
        1000 
    );

into


   if(data) { 
        addmsg(data)
   } else {
        setTimeout(waitForMsg, 1000);
   };

And it is working fine. Thank you so much for all the input.

Hmm, but now as soon as a new message comes through the script will stop polling for further updates.

fretburner · August 22, 2014, 11:30am

I’ve got a couple of suggestions regarding the updated news.php script:

Running the update query within the while loop is going to result one DB query per news item. It’d be more efficient to add each news_id to an array and then update the records in a single query, outside of the loop:


$news_ids = array();
while($row = $result->fetch()) {
    $news_ids[] = $row['news_id']; // Add the ID to an array
    echo "<p>{$row['news_content']}</p>";
}

// Update all new records at once
$qry = 'UPDATE news SET isNew = 0 WHERE news_id IN (' . implode(',', $news_id) . ')';
$pdo->query($qry);

Note that it’s more complicated to do a prepared statement using the WHERE … IN () clause, and as the values are record IDs from the DB it’s safe enough to use a plain old query instead.

You can rewrite the main query to avoid the need for one of the nested sub-queries:

SELECT news_id, sender_id, news_content, add_date
FROM news
WHERE sender_id =
   (SELECT sender_id
    FROM news
    WHERE isNew = 1
    ORDER BY add_date ASC
    LIMIT 1)
ORDER BY add_date DESC

donboe · August 23, 2014, 6:52am

Hi fretburner thank you again for the reply :tup:

That is where I actually was after

The news items are listed under a comment form so I needed a listing per sender, so for me this is working just fine.

fretburner:

I’ve got a couple of suggestions regarding the updated news.php script:

Running the update query within the while loop is going to result one DB query per news item. It’d be more efficient to add each news_id to an array and then update the records in a single query, outside of the loop:
$news_ids = array();
while($row = $result-&gt;fetch()) {
    $news_ids[] = $row['news_id']; // Add the ID to an array
    echo "&lt;p&gt;{$row['news_content']}&lt;/p&gt;";    
}

// Update all new records at once
$qry = 'UPDATE news SET isNew = 0 WHERE news_id IN (' . implode(',', $news_id) . ')';
$pdo-&gt;query($qry);
Note that it’s more complicated to do a prepared statement using the WHERE … IN () clause, and as the values are record IDs from the DB it’s safe enough to use a plain old query instead.

Like I said above the with the while(display) loop I fetch the messages from just on sender so I only need to update the isNew status from the last message from that sender.

fretburner:

You can rewrite the main query to avoid the need for one of the nested sub-queries:

SELECT news_id, sender_id, news_content, add_date
FROM news
WHERE sender_id =
   (SELECT sender_id
    FROM news
    WHERE isNew = 1
    ORDER BY add_date ASC
    LIMIT 1)
ORDER BY add_date DESC

This one is/was very helpful thanks a lot

Again thank you for all your input and tips.

donboe · August 27, 2014, 9:38am

donboe:

Hi fretburner
I got this solved I changed:
    if (data) {
    	addmsg("new", data);
	}
    setTimeout(
        waitForMsg, 
        1000 
    );
into
   if(data) { 
        addmsg(data)
   } else {
        setTimeout(waitForMsg, 1000);
   };
And it is working fine. Thank you so much for all the input.

As usual were you right about this!

Let me use a small table to explain what I realy need:

[table=“width: 700, class: grid, align: left”]
[tr]
[td]news_id[/td]
[td]sender_id[/td]
[td]news_content[/td]
[td]add_date[/td]
[td]isNew[/td]
[/tr]
[tr]
[td]1[/td]
[td]1[/td]
[td]Text[/td]
[td]2014-08-16[/td]
[td]0[/td]
[/tr]
[tr]
[td]2[/td]
[td]2[/td]
[td]Text[/td]
[td]2014-08-17[/td]
[td]0[/td]
[/tr]
[tr]
[td]3[/td]
[td]3[/td]
[td]Text[/td]
[td]2014-08-18[/td]
[td]0[/td]
[/tr]
[tr]
[td]4[/td]
[td]1[/td]
[td]Text[/td]
[td]2014-08-19[/td]
[td]0[/td]
[/tr]
[tr]
[td]5[/td]
[td]2[/td]
[td]Text[/td]
[td]2014-08-20[/td]
[td]0[/td]
[/tr]
[tr]
[td]6[/td]
[td]3[/td]
[td]Text[/td]
[td]2014-08-21[/td]
[td]0[/td]
[/tr]
[tr]
[td]7[/td]
[td]1[/td]
[td]Text[/td]
[td]2014-08-22[/td]
[td]1[/td]
[/tr]
[tr]
[td]8[/td]
[td]2[/td]
[td]Text[/td]
[td]2014-08-23[/td]
[td]1[/td]
[/tr]
[tr]
[td]9[/td]
[td]3[/td]
[td]Text[/td]
[td]2014-08-24[/td]
[td]1[/td]
[/tr]
[/table]

As explained would I like to display a listing with messages per sender, rather then displaying all messages with status isNew = 1; With the above table values along with the query as mentioned before I fetch all news items from sender_id 1 (news_id’s: 7, 4 and 1 DESC)


WHERE sender_id =
               (SELECT sender_id
                  FROM news
                 WHERE add_date =
                       (SELECT MIN(add_date)
                          FROM news
                         WHERE isNew = 1))

With the above table values the if/else statement in the AJAX method::


   if(data) { 
        addmsg(data)
   } else {
        setTimeout(waitForMsg, 1000);
   };

was working since there were still 2 news_id’s in the database with status isNew = 1 and this was reason as well why the scroll bar was appearing after a while. But after commenting to those 2 messages (id’s 8 and 9) as well and updating their isNew status to 0 the if/else statement won’t function any longer. It will only function as long as there are news_id’s with status isNew = 1 before the last one was updated to 0.

I hope this makes any sence.

How do I need to change the AJAX method and/or Query to accomplish where I am after.

Thank you in advance

fretburner · August 27, 2014, 10:05am

Hi Don,

OK, so lets say that on first load the server returns records 1, 4, and 7, and then sets isNew to 0 for record 7. Should the next call to the server (via AJAX) return the records for sender 2? Or it should only be looking for new records from the current sender (1)?

donboe · August 27, 2014, 11:47am

Hi fretburner. Thanks for the reply

Yes after a comment (form) to the last messages of sender 1 is submted it should return indeed all records from sender_id 2 since the query is looking for the oldest record with status isNew = 1.

SELECT MIN(verzonden) AS verzonden
						    FROM berichten
						   WHERE   isNieuw = 1

and that is in the example table indeed record 8 from sender 2 etc etc etc

Thank you in advance

fretburner · August 27, 2014, 12:07pm

donboe:

Yes after a comment (form) to the last messages of sender 1 is submted it should return indeed all records from sender_id 2 since the query is looking for the oldest record with status isNew = 1.
SELECT MIN(verzonden) AS verzonden
						    FROM berichten
						   WHERE   isNieuw = 1
and that is in the example table indeed record 8 from sender 2 etc etc etc

OK, I think I follow… let me just check something with you: you say that after a form is submitted, then the server should return records for the next sender - what about if the user opens the page, but doesn’t reply (i.e. doesn’t submit the form)? Then the AJAX call should only return any new messages from the current sender (sender_id 1, in our example) that have been sent since the page was loaded?

donboe · August 27, 2014, 12:13pm

Hi fretburner. Thanks for the reply.

The reader will reply. This is for a so called PTC site, a site where people are paid for reading news messages

Topic		Replies	Views
Long Polling PHP	6	7618	February 19, 2018
Polling! What am I doing wrong JavaScript	16	972	September 10, 2018
Best way to do long polling? JavaScript	9	4706	September 6, 2018
Message system Get Started	8	1381	October 16, 2014
Ajax long polling JavaScript	3	1557	May 1, 2015

Try to understand and implement Long Polling

Related topics