Sql query inside while loop

Hi guys,

I was told not to run sql queries inside while loop as it really slows down the script, but I don’t know if I can do it in any other way since I have to extract data from two separate tables.
Can someone please let me know if I can do it in a way that doesn’t call an sql query inside my while loop, or any other faster way?

Thank you.


$get = mysql_query("SELECT * FROM mainweb WHERE country='$country_form' AND who='host' ORDER BY date DESC");

while ($row = mysql_fetch_assoc($get))
{
 $host_id= $row['member_id'];
 $firstname= strip_tags(stripslashes($row['firstname']));

   $getrep = mysql_query("SELECT * FROM reputation WHERE member_id='$host_id'");
   $x=0;
   while ($row = mysql_fetch_assoc($getrep))
   {
   $reputation= $row['reputation'];
   $y=$x+$reputation;
   $x=$y;
   }
   if (($y>0) && ($y<=6))
   echo "between 0 and 6";
   elseif (($y>6) && ($y<=10))
   echo "between 6 and 10";

echo"$firstname";
}

I was just about to ask a similar question about efficiency (which is the best way?), so I’ll offer you my solution that avoids making a query inside your while loop, and maybe other people can comment on whether it is an efficient solution or not?

I have this function:


function getAgents () {

	 $result = @mysql_query("SELECT * FROM agents");
	
	 if (!$result) {
	   exit('<p>Error performing query: ' .
		   mysql_error() . '</p>');
	 }

$agents = array();

while ($row = mysql_fetch_array($result, MYSQL_ASSOC)) {

foreach ($row as $key => $value)
	{
		$formArray[$key] = $value;
	}
	
	$agentID = $row['agentID'];
	$agents[$agentID] = $formArray;
}
	return $agents;
}

I call it at the top of the page, and then close off:

$agents = getAgents();
mysql_close();

I then have my query and while loop as per usual:

	 $result = @mysql_query("SELECT * FROM residential LIMIT $limitValue, $limit");
	 	 if (!$result) {
	   exit('<p>Error performing query: ' .
		   mysql_error() . '</p>');
	 }

while ($row = mysql_fetch_array($result)) {

Within the loop, I set a variable to indicate which agent, and then access data from my array that the function getAgents() made:

$agent = $row['contact'];
//$row['contact'] would be equivalent to your $host_id
echo $agents[$agent]['name'] . " - " . $agents[$agent]['phone'];

} //end while

Of course I also have lots of other stuff in my while loop to display onscreen, from my query to the “residential” table.

So for you, you’d make your “reputation” query in a function, like I have done for my getAgents(), then access it in your while loop.

Only, you should propbably wait for other people to comment on the efficiency of my code. This solution works well for me for this site, as there are only a very few agents, so the array made by getAgents() isn’t very large. The question I was going to ask was whether that’s an appropriate solution for a larger table with more rows, which it seems like yours would be, or would need the potential to be - I just don’t know how well my code would scale up in terms of efficiency.

Please let me know if there’s anything in that which isn’t clear, and hopefully someone can tell you whether it’s a good solution or if there’s a better one!

Well, like you presumed, my reputation query will have thousands of rows, so I don’t know if your solution is faster. I don’t really understand how to execute the code you gave me but I haven’t meddled with it yet.
I think it’s better I wait to see if others say your solutions is faster, then I’ll try it and ask you for some help if I can’t get it right.

Thank you very much rickibarnes.

Can anyone help me and rickibarnes???

I’ve found that if I have a query that depends on another query, then the most efficient way for me to get the second set of results is to use the mysql IN structure. For example, if you have to get a group of people from one set and get their information from another table, especially if there are times when a person might not exist in the first set (meaning you cannot simply combine the queries into one), then do the following:


$array = array();

$query1 = "SELECT personId, someOtherColumn FROM table1";

$result = mysql_query( $query );

while ( $row = mysql_fetch_array( $result ) ) {

   $array[] = $row['personId'];

}

$query2 = "SELECT name FROM table2 WHERE pID IN (" . implode( ',', $array ) . ")";

It’s a bit brief and not tested, but this is just to give you an idea. Perhaps a more complex join query would give the same results, but without delving too deeply into complex queries, this will get the job done without using a query within a loop and bog down the server doing so. These queries do assume that the personID and pID are integers.

Just one option that will give you and idea of how you can do it.

What is the table structure for the “mainweb” and “reputation” tables?

With the query:

SELECT * FROM reputation WHERE member_id='$host_id'"

how many rows do you expect to be returned by the result set?

how many rows do you expect to be returned by the result set?

For now not many, but in the future probably thousands.

What is the table structure for the “mainweb” and “reputation” tables?

“mainweb” has around 15 columns such as member_id, name, lastname, country. And I echo out most of the information for every user as his profile(which I hope eventually are in the thousands).

Then I use the member_id from that table to extract info from “reputation” table, which includes image, comments, and a numerical value (reputation).
So if the user has many comments from other users, he’ll have many rows.
Hopefully that answers the question.

Why not use the database ability to join table results together?

You currently have the following two queries:


SELECT *
FROM mainweb
WHERE country='$country_form'
    AND who='host'
ORDER BY date DESC

and


SELECT *
FROM reputation
WHERE member_id='$host_id'

What you can do it to use a LEFT OUTER JOIN so that the mainweb results control the reputation results that are retrieved.


SELECT mainweb.*, reputation.*
FROM mainweb
    LEFT OUTER JOIN reputation
        ON mainweb.member_id = reputation.member_id
WHERE mainweb.country='$country_form'
    AND mainweb.who='host'
ORDER BY mainweb.date DESC

You may also find that you have improved database performance, if you only retrieve the fields that you need, for example:


SELECT mainweb.date, mainweb.firstname, reputation.reputation
FROM mainweb
    LEFT OUTER JOIN reputation
        ON mainweb.member_id = reputation.member_id
WHERE mainweb.country='$country_form'
    AND mainweb.who='host'
ORDER BY mainweb.date DESC

[edit]corrected the order of the tables, thanks Rudy.

nice explanation paul, but you got your tables backwards :slight_smile:

it should be

SELECT mainweb.date
     , mainweb.firstname
     , reputation.reputation
  [COLOR="Blue"]FROM mainweb
LEFT OUTER
  JOIN reputation[/COLOR]
    ON reputation.member_id = mainweb.member_id
 WHERE mainweb.country = '$country_form'
   AND mainweb.who = 'host'
ORDER
    BY mainweb.date DESC

Thanks Rudy. I was trying to decide if there was a performance impact on the ordering of the table. Perhaps you can help to clear this up.

Is it better to structure the query so that the potentially smaller number of results is the table that is then joined by others? Or, can many databases perform that type of optimisation themself?

yes it is, and yes they can :slight_smile:

however, that’s not the issue with your LEFT OUTER JOIN

semantically, X left outer join Y means that you want all rows from X, with or without matching rows from Y, while Y left outer join X means that you want all rows from Y, with or without matching rows from X

yours wasn’t an optimization issue, but one of correctness

:slight_smile:

Oh yes, and he did want all of the mainweb ones, not just the mainweb ones that had a reputation. Oops :blush: Thanks for the correction.

I really appreciate it you guys are trying to help me without seeing all my script. I’m sure it’s not easy.I tried what sitepoint author suggested together with paul and I get the reputation fields from the reputation table executed well. However, there are two problems that came up that I’ve been trying to solve.

  1. Mainweb table has one row for each user, but Reputation table might have several rows for every user, since it represents his commented reputation.
    So if Adam has 3 comments on Reputation table, when I run the script, I get Adam’s info from Mainweb three seperate times. But I only need Adam’s info once with the three comments he has.

  2. How do I check num_rows in Reputation table? The one I’m running now seems to be checking both tables, but I only need it to check from Rep table.

I feel this a little bit too advanced for me. Maybe I should stick to running an sql query inside while loop. Would that really be a bad idea?

Thak you guys.

$get = mysql_query("SELECT mainweb.*, reputation.*
  FROM mainweb
LEFT OUTER
  JOIN reputation
    ON reputation.member_id = mainweb.member_id
 WHERE mainweb.country = '$country_form'
   AND mainweb.who = 'host'
ORDER
    BY mainweb.date DESC");

while ($row = mysql_fetch_array($get))
{
  $host_id= $row['member_id'];
  $firstname= strip_tags(stripslashes($row['firstname']));
$lastname= strip_tags(stripslashes($row['lastname']));
$comments= $row[comments]; //this last is taken from rep table

echo"<div style='text-align: center;'>$firstname $lastname</div>";

echo"<h1>Host Reputation</h1>";
$num_row = mysql_num_rows($get);
if ($num_row==0){
echo"The user has not built up a reputaion!<hr size='1'><p />";
}
else{
echo"The user has this: $comments";
}}

yes, it would

:slight_smile:

SELECT mainweb.member_id
     , mainweb.date
     , mainweb.firstname
     , [COLOR="blue"]GROUP_CONCAT(reputation.reputation) AS reputations[/COLOR]
  FROM mainweb 
LEFT OUTER  
  JOIN reputation 
    ON reputation.member_id = mainweb.member_id  
 WHERE mainweb.country = '$country_form' 
   AND mainweb.who = 'host' 
[COLOR="Blue"]GROUP
    BY mainweb.member_id[/COLOR]
ORDER  
    BY mainweb.date DESC

Thank you for the script r937, I learned a lot.
Now my users from mainweb only show up once, but I had to play with the values from the reputation table. As far as I understand, the “GROUP_CONCAT” function you gave me groups the values into a string, but since I need them as an array so I could represent each comment in a separate table, I exploaded them.

And to check for number of rows of comments each user has, I couldn’t use “mysql_num_rows” becaues that seemed to run only on the mainweb and not on the reputation table. So I just checked to see if a value inside reputation table is NULL “if ($rep==NULL)”, given that the GROUP_CONCAT function returns NULL if there are no non-NULL values.

I hope I got it right.

This is how I did it:


SELECT mainweb.*
     , GROUP_CONCAT(reputation.reputation) AS reputation
     , GROUP_CONCAT(reputation.comments) AS comments
  FROM mainweb
LEFT OUTER
  JOIN reputation
    ON reputation.member_id = mainweb.member_id
 WHERE mainweb.country = '$country_form'
   AND mainweb.who = 'host'
GROUP
    BY mainweb.member_id
ORDER
    BY mainweb.date DESC


while ($row = mysql_fetch_array($get))
{
$rep = $row['reputation'];
$comments = $row['comments'];
if ($rep==NULL)
echo"The user has not built up a reputation!";
else{
             $comment = explode(",",$comments);
	$res = explode(",",$rep);

	foreach ($res as $key => $value){
	$e = $res[$key];
	$c = $comment[$key];
echo'<table blablalba><tr><td>'.$c.'</td><td>'.$e.'</td></tr></table>';
}}}

you might run into trouble exploding the comments, if a comment happens to contain a comma

look up the GROUP_CONCAT syntax, you can use a different SEPARATOR to avoid that problem

I added comas in the users comments and realized you were right. It won’t work since there’s a default SEPARATOR that adds commas already. So I made the SEPARATOR add a string set that I don’t think will ever pop up in a wrriten comment (*^).

One question for anyone that’s out there: after hundreds of comments are made, GROUP_CONCAT will end up having really big long values. Will that slow my script down or be harmful in any way?
I know I can set max length but I don’t want to cut comments out…


,GROUP_CONCAT(reputation.comments SEPARATOR'*^') AS comments
, GROUP_CONCAT(reputation.image SEPARATOR'*^') AS image

$comments = $row['comments'];
$image = $row['image'];

$comment = explode("*^",$comments);
$image1 = explode("*^",$image);

Some thoughts… I’m assuming “reputation” is a 5-star type rating field (based on how you’re using it)? If you’re grouping on the member, why not process the average reputation at the database, e.g.

ROUND(AVG(reputation.reputation)) AS reputation

then for your aggregated comments, you should first concat your rep and comments together then group_concat so you can’t get disassociation with 2 separate arrays, e.g.

GROUP_CONCAT(CONCAT(reputation.reputation, ‘^|',reputation.comments) SEPARATOR '|^’) AS comments

Then in your PHP do something like this


if ($rep==NULL) {
    print("The user has not built up a reputation!"); 
} else { 
    print("The user has an average reputation of {$reputation}.");
    $comments = explode("*|^",$comments); 
    foreach ($comments as $comment) { 
        $comment = explode("^|*", $comment); 
        if (count($comment) == 2) {
            print("<table blablalba><tr><td>{$comment[0]}</td><td>{$comment[1]}</td></tr></table>"); 
        }
    }
}

you should first concat your rep and comments together then group_concat so you can’t get disassociation with 2 separate arrays,

I don’t understand what “get disassociation with 2 separate arrays” means. Can you explain it to a beginner or give me an example?

what I did is somehting like this:

     $comment = explode("*^",$comments);
	$res = explode("*^",$rep);
	foreach ($res as $key => $value){
	
	$e = $res[$key];
	$c = $comment[$key];
	if($e==1) echo "user has one rating, and his comments are: $c";
      if($e==2) echo "user has two ratings, and his comments are: $c";//and so on, up to 5
}[/

Transio if GROUP_CONCAT will end up having really big long values. Will that slow my script down or be harmful in any way?