SitePoint Sponsor

User Tag List

Page 3 of 3 FirstFirst 123
Results 51 to 75 of 75
  1. #51
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Dorsey View Post
    Also, keep in mind that *nix is NOT a real-time system. When I wrote real-time systems..
    Daft question, but whats the definition of a real-time system? - I always thought that all OS's would work in real time? You've got me curious now!
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  2. #52
    <?php while(!sleep()){code();} G.Schuster's Avatar
    Join Date
    Mar 2007
    Location
    Germany
    Posts
    428
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    When going down the cron-route I'd rather use lockfiles with locking than just checking for their existence.

    PHP Code:
    $lockFile dirname(__FILE__).'/cron.lock';
    if(
    FALSE !== ($lockFH = @fopen($lockFile'w'))) {
          if(
    TRUE !== @flock($lockFHLOCK_EX LOCK_NB)) {
                exit;
          }
    }
    else {
          exit;
    }

    // Do fancy things...

    @fclose($lockFH);
    @
    unlink($lockFile); 
    This way an existing file (e.g. left by a cancelled run) does no harm as it isn't locked anymore.
    While the script is running it holds the lock and no other process (ok, not exactly, but in PHP terms) could get the lock and thus exits early.
    I've been using this method for years now with transactions running more than a minute and a minute-cron - no problems so far, all is in sync.

  3. #53
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Scallio,
    The problem is that the data is coming 24/7. There is no break. You are right that my reading will lock up my database and then there will be whole long of queue waiting to insert while I am keeping reading. I am also pretty lost first I thought of processing it on 5 minutes basis but come to see of it is the same problem because I will do a read and lock the db so look like I have to find other alternatives.

  4. #54
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    The only other alternative you have is to have different servers receiving information from different GPS devices, Realistically this might be your best option because it will break down the workload as you'll have multiple machines handling their own devices.

    As long as they are all accessible by your main system (for pulling data from them) you'll theoretically be ok.
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  5. #55
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tango,
    How will that help because end of the day I am still having only a single database to read from right it will still lock I guess.

  6. #56
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    No, you are not understanding me correctly.

    If you have multiple machines, each one can process its data and then store it in a database. You then need to become creative about how you pull data from your systems.

    Say you have 50 GPS devices and 10 servers. Each server handles the data for 5 GPS devices. You want to search for data from two devices which are on differeny machines. Simple - You send a query to them all to select data where GPS-device='<device_number>'

    You'll then get the data back from two different machines to your main system where you can analyse it on screen.

    I'll admit it could be a lot of work and ver complex but it could save you if you truly have so much data coming in that one machine can't handle it.

    If you have that much data, then you need to delegate the work to slave systems to handle it. You'll need a master system that can then interrogate them for the required information which can then be used for your purposes.
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  7. #57
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tango,
    So basically you are asking me to load balance at the receiving
    End itself right. Your advice is not to use the php or
    cron because of table locking problem right.

  8. #58
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tango,
    So basically you are asking me to load balance at the receiving
    End itself right. Your advice is not to use the php or
    cron because of table locking problem right.

  9. #59
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tango,
    So in what will the php script you suggested will be best applicable?

  10. #60
    <?php while(!sleep()){code();} G.Schuster's Avatar
    Join Date
    Mar 2007
    Location
    Germany
    Posts
    428
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    MySQL with the InnoDB storage engine does not lock the whole table, it supports row-locking.
    Could you please specify in detail what data and how it is received?

    There are plenty options to handle large amounts of data - but I don't think a "Newbie" (no offence! I'm refering to your nick "newtomysql") will be able to handle them, so the best bet would be to get someone who knows what he's doing.

  11. #61
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Schuster,
    Currently the storage engine is unfortunately is myisam. The data is received from a java listener. The data is gps data e.g latitude,logittude, date time, alerts etc. So in order not to burden the java listener we immediately store the data after receiving. So if I start to process the poi location, geo fence etc I will need to do a lot of select query and calculation etc. So I was planning to then read from the inserted table and process the poi and geo fence etc rather then the java listener. So what is your idea then ?

  12. #62
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Yes i am basically saying that you might be best considering load balancing IF there really is to much data to handle.

    How many devices are you recording data from and how frequently?
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  13. #63
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tangoforce,
    As of now nearly 2k devices but now we have like 3 server doing the job but at times I do find that some data never reach the db do yo think becuase the load is heavy. The iterval is different ranging from 1 to 5 minutes. How to know if is high load on linux? Is the load values give you the indications?

  14. #64
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    No idea about linux.. but wow, thats a lot of devices you have running there.

    How many inserts to the database are there per second?

    What do you mean about the interval being from 1 to 5 minutes? - Is that for each individual device? or for all of them?
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  15. #65
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tangoforce,
    In a minute roughly we get around 4-5K of data that is quite a lot right. The interval is for the device meaning that some send at 1 minute interval some 5 minutes interval. Thank you.

  16. #66
    SitePoint Wizard bronze trophy Immerse's Avatar
    Join Date
    Mar 2006
    Location
    Netherlands
    Posts
    1,661
    Mentioned
    7 Post(s)
    Tagged
    1 Thread(s)
    To check the load on a linux server type 'top' at the command line. It will show you what processes are running, and how much load the machine is experiencing. When you run top, the top-right of the screen will show something like this:

    Code:
    load average: 0.06, 0.12, 0.1
    I'm pretty sure the numbers mean the following:
    The number on the left is the current load (a load of 1.0 means all CPUs are being fully used). The next number is a 5 minute average, and the last number is the average load since the machine was last booted.

  17. #67
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by newtomysql View Post
    In a minute roughly we get around 4-5K of data that is quite a lot right.
    Do you mean Kb of data OR 4-5k queries?

    If you mean 4-5k of queries from your devices then yes that is a lot - i don't honestly think php could handle that many so quickly unless the actual packets of data are very small.

    What is the actual gps data? is it just coordinates? (EG numbers?) Can you show us a sample of the data?
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  18. #68
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Immerse,
    I know about top but how to decide if my server is overloaded or not. Is it if the value is consistently more than 1 is it? I have multiprocessor.

  19. #69
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tango,
    Is 4K queries. Sample data is like this. This is a sample from the device. 10000123,AB100006,180511184513,022550216,000041,0080,000,00000000,89670000'18051118451130460714N10105760E000000*E81A#. So I will split and store into the database fields accordingly. For example this data 89670000'18051118451130460714N10105760E000000 consist of the date,time,lat and long. So now you get a better idea right?

  20. #70
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hmm

    Say 4.5k queries per minute, thats 75 queries per second which although relatively high, doesn't actually seem like much compared to the numer of queries some websites handle per second.

    How are your GPS devices sending their data? How is it received by your servers (EG http, raw TCP/IP, UDP) ?

    I can't help but think that you might have a bottleneck somewhere else in your system. If your code literally just inserts that data and does nothing else with it, then the database should be able to handle 75 inserts per second.. in theory.
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  21. #71
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tango,
    On top of that I also generate reports using this data too. The gps send the raw data using tcp/ip.

  22. #72
    SitePoint Guru
    Join Date
    Aug 2009
    Posts
    669
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Are you generating the reports on the same machine? - This could take a lot of CPU cycles from the processor. How often are these reports generated?
    I'll do anything to avoid working on my own code

    Are you using: if (isset($_POST['submit'])) ?
    IE has a bug and does not always send the value.

  23. #73
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Tango,
    No we have separate machine for the listener,db and web server. The reports are on the web server. The frequency is not fix.

  24. #74
    Utopia, Inc. silver trophy
    ScallioXTX's Avatar
    Join Date
    Aug 2008
    Location
    The Netherlands
    Posts
    9,067
    Mentioned
    153 Post(s)
    Tagged
    2 Thread(s)
    Quote Originally Posted by Immerse View Post
    a load of 1.0 means all CPUs are being fully used
    That's not entirly true. A load of 1.0 means one core is fully used. If you have n cores the maximum load is n. If the loads get above n there is more work that processors to do the work and the system will eventually clog up. So long as the load is on average below n all is fine.
    Rémon - Hosting Advisor

    SitePoint forums will switch to Discourse soon! Make sure you're ready for it!

    Minimal Bookmarks Tree
    My Google Chrome extension: browsing bookmarks made easy

  25. #75
    SitePoint Guru
    Join Date
    Feb 2007
    Posts
    874
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Dear Scallio,
    So any other indicator to be used together to decide on the load. I notice my mysql sometimes have this problem.It will shutdown by itself is it due to heavy load? Then another thing the table read error can be due to heavy inserts?


    110510 21:25:16 [Note] /usr/libexec/mysqld: Normal shutdown

    110510 21:25:21 [Note] /usr/libexec/mysqld: Shutdown complete

    110510 21:25:29 [Note] /usr/libexec/mysqld: ready for connections.
    Version: '5.0.67-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 Source distribution
    110510 21:38:17 [ERROR] Got error 134 when reading table './fms/tripData'
    110510 21:40:02 [ERROR] Got error 134 when reading table './fms/tripData'
    110510 21:47:05 [ERROR] Got error 134 when reading table './fms/tripData'
    110510 21:51:44 [ERROR] Got error 134 when reading table './fms/tripData'


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •