Key Takeaways
- JavaScript has grown beyond the browser and can be used effectively for desktop scripting, including writing shell scripts. Node.js is a prerequisite for this process.
- JavaScript can be used to recursively iterate over files in a directory and identify occurrences of a specified string. This is done using Node’s native File System module, which comes with a readdir method, and the lstatSync method to identify directories.
- Shell scripts can be written to identify specific file types, such as PHP files, and search for occurrences of specific strings within these files. This can be done using regular expressions and JavaScript’s match method.
- JavaScript for shell scripting can be taken further by automating processes using replace() and fs.writeFileSync, or using child_process.exec to open files for editing. JavaScript files can also be made executable, and the language can be used for a range of tasks outside of the browser, including automating scripting tasks.
“How to Write Shell Scripts with JavaScript” is the editorial from our latest JavaScript newsletter.
This week I had to upgrade a client’s website to use SSL. This wasn’t a difficult task in itself — installing the certificate was just the click of a button — yet once I had made the switch, I was left with a lot of mixed content warnings. Part of fixing these meant that I had to go through the theme directory (it was a WordPress site) and identify all of the files in which assets were being included via HTTP.
Previously, I would have used a small Ruby script to automate this. Ruby was the first programming language I learned and is ideally suited to such tasks. However, we recently published an article on using Node to create a command-line interface. This article served to remind me that JavaScript has long since grown beyond the browser and can (amongst many other things) be used to great effect for desktop scripting.
In the rest of this post, I’ll explain how to use JavaScript to recursively iterate over the files in a directory and to identify any occurrences of a specified string. I’ll also offer a gentle introduction to writing shell scripts in JavaScript and put you on the road to writing your own.
Set Up
The only prerequisite here is Node.js. If you don’t have this installed already, you can head over to their website and download one of the binaries. Alternatively, you can use a version manager such as nvm. We’ve got a tutorial on that here.
Your First Shell Script
So where to begin? The first thing we need to do is iterate over all of the files in the theme directory. Luckily Node’s native File System module comes with a readdir method we can use for that. It takes the directory path and a callback function as parameters. The callback gets two arguments (err
and entries
) where entries
is an array of the names of the entries
in the directory excluding .
and ..
— the current directory and the parent directory, respectively.
const fs = require('fs');
function buildTree(startPath) {
fs.readdir(startPath, (err, entries) => {
console.log(entries);
});
}
buildTree('/home/jim/Desktop/theme');
If you’re following along with this, save the above in a file named search_and_replace.js
and run it from the command line using node search_and_replace.js
. You’ll also need to adjust the path to whichever directory you are using.
Adding Recursion
So far so good! The above script logs the directory’s top level entries to the console, but my theme folder contained subdirectories which also had files that needed processing. That means that we need to iterate over the array of entries and have the function call itself for any directories it encounters.
To do this, we first need to work out if we are dealing with a directory. Luckily the File System module has a method for that, too: lstatSync. This returns an fs.Stats object, which itself has an isDirectory
method. This method returns true
or false
accordingly.
Note that we’re using the synchronous version of lstat
here. This is fine for a throwaway script, but the asynchronous version should be preferred if performance matters.
const fs = require('fs');
function buildTree(startPath) {
fs.readdir(startPath, (err, entries) => {
console.log(entries);
entries.forEach((file) => {
const path = `${startPath}/${file}`;
if (fs.lstatSync(path).isDirectory()) {
buildTree(path);
}
});
});
}
buildTree('/home/jim/Desktop/theme');
If you run the script, you will now see that it prints a list of files and folders for the current directory and every subdirectory that it contains. Success!
Identifying Files to Process
Next, we need to add some logic to identify any PHP files, open them up and search them for any occurrences of the string we are looking for. This can be done using a simple regular expression to check for file names that end in “.php”, then calling a processFile
function if that condition is met, passing it the current path as an argument.
Let’s also make a small improvement to how the pathname is constructed. Until now we’ve been using string interpolation, but this will only work in a Unix environment due to the forward slash. Node’s path module however, offers a join method, which will take the separator into account.
const fs = require('fs');
const Path = require('path');
function processFile(path) {
console.log(path);
}
function buildTree(startPath) {
fs.readdir(startPath, (err, entries) => {
entries.forEach((file) => {
const path = Path.join(startPath, file);
if (fs.lstatSync(path).isDirectory()) {
buildTree(path);
} else if (file.match(/\.php$/)) {
processFile(path);
}
});
});
}
buildTree('/home/jim/Desktop/theme');
If you run the script at this point, it should recurse a directory tree and print out the path of any php files it might find.
Searching for Text within a File
All that remains to do is to open up the files that the script finds and to process them. This can be done using Node’s readFileSync method which accepts the file path and its encoding (optional) as parameters. If the encoding is specified then this function returns a string. Otherwise it returns a buffer.
Now we can read the contents of a file into a variable, which we can then split on every newline character and iterate over the resulting array. After that, it’s a simple matter of using JavaScript’s match method to look for the word or phrase we want:
function processFile(path) {
const text = fs.readFileSync(path, 'utf8');
text.split(/\r?\n/).forEach((line) => {
if (line.match('http:\/\/')) {
console.log(line.replace(/^\s+/, ''));
console.log(`${path}\n`);
}
});
}
If you run the script now, it’ll print out every line where it finds a match as well as the name of the file.
Taking It Further
In my particular case this was enough. The script spat out a handful of occurrences of “http” which I was able to fix by hand. Job done! It would however, be simple to automate the process using replace()
and fs.writeFileSync to alter every occurrence and write the new contents back to a file. You could also use child_process.exec to open up the files in Sublime ready for editing:
const exec = require('child_process').exec;
...
exec(`subl ${path}`)
This kind of scripting lends itself to a whole bunch of tasks, not just manipulating text files. For example, maybe you want to batch rename a bunch of music tracks, or delete every Thumbs.db
file from a directory. Maybe you want to fetch data from a remote API, parse a CSV file, or generate files on the fly. The list goes on …
You can also make the JavaScript files executable, so that they run when you click on them. Axel Rauschmayer goes into this on his post Write your shell scripts in JavaScript, via Node.js.
Conclusion
And there we have it. I’ve demonstrated how to use JavaScript to recurse through a directory tree and manipulate a subset of the files contained within. It’s a simple example, but it serves to emphasize the point that JavaScript can be used for a whole host of tasks outside of the browser, desktop scripting being one of them.
Now its over to you. Do you automate scripting tasks with JavaScript? If not do you have a different preferred language, or are you a bash purist? What kind of tasks do you automate? Let me know in the comments below.
Frequently Asked Questions (FAQs) about Shell Scripts in JavaScript
What are the benefits of using JavaScript for shell scripting?
JavaScript is a versatile language that is primarily used for web development. However, its use is not limited to just that. It can also be used for shell scripting. The main advantage of using JavaScript for shell scripting is that it allows you to use the same language across different parts of your project, reducing the learning curve and increasing productivity. Moreover, JavaScript has a rich ecosystem with numerous libraries and frameworks, which can be leveraged to write more efficient and powerful shell scripts.
How can I start writing shell scripts in JavaScript?
To start writing shell scripts in JavaScript, you need to have Node.js installed on your system. Once you have Node.js, you can use the ‘fs’ and ‘child_process’ modules to interact with the file system and execute shell commands respectively. You can also use libraries like ShellJS or Google’s zx to make your scripts more powerful and easier to write.
What is the difference between ShellJS and Google’s zx?
Both ShellJS and Google’s zx are libraries that provide utilities for shell scripting in JavaScript. However, they have some differences. ShellJS is a portable implementation of Unix shell commands on top of the Node.js API. It provides a simple way to run shell commands with JavaScript, and it works on both Unix and Windows systems. On the other hand, Google’s zx is a newer library that provides a more modern and promise-based API. It also includes additional features like top-level await, built-in fetch, and markdown-like syntax for shell commands.
Can I use ES6 features in my shell scripts?
Yes, you can use ES6 features in your shell scripts. Node.js, which is used to run JavaScript shell scripts, supports most of the ES6 features. This includes let and const for variable declarations, arrow functions, template literals, destructuring assignment, and more. Using these features can make your scripts more concise and easier to read.
How can I handle errors in my shell scripts?
Error handling is an important part of writing shell scripts. In JavaScript, you can use try-catch blocks to handle errors. If an error occurs in the try block, the control is passed to the catch block where you can handle the error. In addition to this, libraries like ShellJS and zx provide their own mechanisms for error handling. For example, in ShellJS, you can use the ‘.error’ property to check if a command resulted in an error, and in zx, you can use the ‘ function which throws an exception if a command exits with a non-zero exit code.
Can I write asynchronous shell scripts in JavaScript?
Yes, you can write asynchronous shell scripts in JavaScript. Node.js supports asynchronous programming with callbacks, promises, and async-await. Moreover, libraries like zx are built with async-await in mind, allowing you to write asynchronous shell scripts more easily.
How can I debug my shell scripts?
Debugging shell scripts in JavaScript can be done in a similar way to debugging regular JavaScript code. You can use console.log statements to print out values at different points in your script. Additionally, you can use the Node.js debugger or a tool like Chrome DevTools for a more powerful debugging experience.
Can I use npm packages in my shell scripts?
Yes, you can use npm packages in your shell scripts. This is one of the advantages of using JavaScript for shell scripting. You can leverage the vast npm ecosystem to use packages that provide additional functionality, make your scripts more powerful, and save you from having to reinvent the wheel.
How can I run my shell scripts?
To run your shell scripts, you can use the ‘node’ command followed by the path to your script. For example, ‘node myscript.js’. If your script is meant to be run as a command-line tool, you can add a shebang line at the top of your script (‘#!/usr/bin/env node’), make your script file executable, and then run it directly like ‘./myscript.js’.
Can I write cross-platform shell scripts in JavaScript?
Yes, you can write cross-platform shell scripts in JavaScript. Node.js runs on multiple platforms including Windows, Linux, and macOS. Moreover, libraries like ShellJS provide a cross-platform API for shell commands, allowing you to write scripts that work on different platforms. However, you should be aware of the differences and potential issues when running scripts on different platforms, especially when dealing with file paths and environment variables.
Network admin, freelance web developer and editor at SitePoint.