How to Process Large Volumes of Data in JavaScript

Share this article

In my previous posts, we examined JavaScript Execution and Browser Limits and a method which can solve “unresponsive script” alerts using Timer-Based Pseudo-Threading. Today, we’ll look at ways to handle large volumes of data within the browser. A few years ago, developers would never have considered alternatives to complex server-side processing. That perception has changed and many Ajax applications send huge quantities of data between the client and the server. In addition, code may update the DOM which is a particularly time-consuming browser process. However, attempting to analyze that information in one go can make an application unresponsive and throw script alerts. JavaScript timers can help prevent browser locking issues by splitting a long data analysis process into shorter chunks. Here’s the start of our JavaScript function:

function ProcessArray(data, handler, callback) {
The ProcessArray() function accepts three arguments:
  1. data: an array of items to process
  2. handler: a function which processes an individual data item
  3. callback: an optional function called when all processing is complete.
Next, we’ll define configuration variables:

  var maxtime = 100;		// chunk processing time
  var delay = 20;		// delay between processes
  var queue = data.concat();	// clone original array
maxtime specifies the maximum number of milliseconds permitted for each chunk of processing. delay
is the time in milliseconds between processing chunks. Finally, queue is clone the original data array–that won’t be necessary in all cases but, since the array is passed by reference and we’re discarding each item, it’s the safest option. We can now use a setTimeout to start processing:

  setTimeout(function() {

    var endtime = +new Date() + maxtime;

    do {
    } while (queue.length > 0 && endtime > +new Date());
First, an endtime is calculated — this is a future time when processing must cease. The do…while loop processes queued items in turn and continues until every item has completed or endtime has been reached.
note: Why use do…while?
JavaScript supports both while loops and do…while loops. The difference is that do…while loops are guaranteed to perform at least one iteration. If we used a standard while loop, the developer could set a low or negative maxtime, and the array processing would never start or complete.
Finally, we determine whether further items need to be processed and, if necessary, call our processing function after a short delay:

    if (queue.length > 0) {
      setTimeout(arguments.callee, delay);
    else {
      if (callback) callback();

  }, delay);
// end of ProcessArray function
The callback function is executed once every item has been processed. We can test ProcessArray() with a small test case:
// process an individual data item
function Process(dataitem) {

// processing is complete
function Done() {

// test data
var data = [];
for (var i = 0; i < 500; i++) data[i] = i;

// process all items
ProcessArray(data, Process, Done);
The code will work in every browser including IE6+. It’s a viable cross-browser solution, but HTML5 provides a far nicer solution! In my next post, we’ll discuss web workers …

Frequently Asked Questions (FAQs) about JavaScript for Large Data Processing

What are the best practices for handling large data sets in JavaScript?

Handling large data sets in JavaScript can be challenging due to its single-threaded nature. However, there are several best practices you can follow. Firstly, consider using Web Workers. They allow you to run JavaScript in separate background threads, preventing large data processing from blocking the user interface. Secondly, use streaming data processing techniques. Libraries like Oboe.js can help you process data as it arrives, reducing memory usage. Lastly, consider using a database. IndexedDB, a low-level API for client-side storage of significant amounts of structured data, can be used for high performance searches on large data sets.

Can JavaScript be used for data science?

Yes, JavaScript can be used for data science. While it’s not traditionally associated with data science, the rise of full-stack JavaScript and the development of libraries and frameworks for data analysis and visualization have made it a viable option. Libraries like Danfo.js provide data manipulation tools similar to those in Python’s pandas library, while D3.js is a powerful tool for data visualization.

How can I optimize JavaScript for large data processing?

Optimizing JavaScript for large data processing involves several strategies. Firstly, use efficient data structures. JavaScript’s built-in array and object types are not always the most efficient for large data sets. Libraries like Immutable.js provide more efficient alternatives. Secondly, consider using Typed Arrays for handling large amounts of binary data. Lastly, use asynchronous programming techniques to prevent blocking the main thread during data processing.

What are the limitations of using JavaScript for large data processing?

JavaScript has a few limitations when it comes to large data processing. Its single-threaded nature can lead to performance issues when processing large data sets. Additionally, JavaScript’s number type is not ideal for precise numerical computations, which can be a problem in data science applications. Lastly, JavaScript lacks some of the advanced data analysis libraries available in languages like Python and R.

How can I use Web Workers for large data processing in JavaScript?

Web Workers allow you to run JavaScript code in the background, on a separate thread. This can be particularly useful for large data processing tasks that would otherwise block the main thread and cause performance issues. To use a Web Worker, you create a new Worker object and pass it the URL of a script to run in the worker thread. You can then communicate with the worker thread using the postMessage method and the onmessage event handler.

What is streaming data processing in JavaScript?

Streaming data processing is a technique where data is processed as it arrives, rather than waiting for the entire data set to be available. This can be particularly useful for large data sets, as it reduces memory usage and allows processing to start sooner. In JavaScript, you can use libraries like Oboe.js to implement streaming data processing.

How can I use IndexedDB for large data processing in JavaScript?

IndexedDB is a low-level API for client-side storage of significant amounts of structured data. It allows you to store, retrieve, and search large data sets in the user’s browser. To use IndexedDB, you first open a database, then create an object store to hold your data. You can then use transactions to read and write data.

What are Typed Arrays in JavaScript and how can they be used for large data processing?

Typed Arrays are a feature of JavaScript that provides a way to work with binary data. They can be particularly useful for large data processing tasks, as they allow you to work with data in a more memory-efficient way. To use a Typed Array, you first create an ArrayBuffer to hold your data, then create a view onto that buffer using one of the Typed Array types.

What are some libraries I can use for data visualization in JavaScript?

There are several libraries available for data visualization in JavaScript. D3.js is one of the most powerful and flexible, allowing you to create a wide range of visualizations. Chart.js is another popular option, providing a simpler API for creating common types of charts. Other options include Highcharts, Google Charts, and Plotly.js.

How does asynchronous programming help with large data processing in JavaScript?

Asynchronous programming allows JavaScript to perform other tasks while waiting for data processing to complete. This can be particularly useful for large data processing tasks, as it prevents the main thread from being blocked, leading to a smoother user experience. JavaScript provides several features for asynchronous programming, including callbacks, promises, and async/await.

Craig BucklerCraig Buckler
View Author

Craig is a freelance UK web consultant who built his first page for IE2.0 in 1995. Since that time he's been advocating standards, accessibility, and best-practice HTML5 techniques. He's created enterprise specifications, websites and online applications for companies and organisations including the UK Parliament, the European Parliament, the Department of Energy & Climate Change, Microsoft, and more. He's written more than 1,000 articles for SitePoint and you can find him @craigbuckler.

Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week