Create Your Own Yeoman-Style Scaffolding Tool with Caporal.js

Starting a new project (especially as a JavaScript developer) can often be a repetitive and tedious process. For each new project, we normally need to add a package.json file, pull in some standard dependencies, configure them, create the correct directory structure, add various other files … The list goes on.

But we’re lazy developers, right? And luckily we can automate this. It doesn’t require any special tools or strange languages — if you already know JavaScript, the process is actually quite simple.

In this tutorial, we are going to use Node.js to build a cross-platform command-line interface (CLI). This will allow us to quickly scaffold out a new project using a set of predefined templates. It will be completely extensible so that you can easily adapt it to your own needs and automate away the tedious parts of your workflow.

Why Roll Your Own?

Although there are plenty of similar tools for this task (such as Yeoman), by building our own we gain knowledge, experience and can make it totally customizable. You should always consider the idea of creating your tools over using existing ones, especially if you are trying to solve specialized problems. This might sound contrary to the common practice of always reusing software, but there are cases where implementing your own tool can be highly rewarding. Gaining knowledge is always helpful, but you can also come up with highly personalized and efficient tools, tailored especially to your needs.

Saying that, we won’t be reinventing the wheel entirely. The CLI itself is going to be built using a library called Caporal.js. Internally it will also use prompt to ask for user data and shellJS that will provide us with some Unix tools right in our Node.js environment. I selected these libraries mostly because of their ease of use, but after finishing this tutorial, you’ll be able to swap them out for alternatives that best fit your needs.

As ever, you can find the completed project on Github: https://github.com/sitepoint-editors/node-scaffolding-tool

Now let’s get started …

Up and Running with Caporal.js

First, create a new directory somewhere on your computer. It is recommended to have a dedicated directory for this project that can stay untouched for a long time since the final command will be called from there every time.

Once in the directory, create a package.json file with the following content:

{
  "name": "scaffold",
  "version": "1.0.0",
  "main": "index.js",
  "bin": {
    "scaffold": "index.js"
  },
  "dependencies": {
    "caporal": "^0.3.0",
    "colors": "^1.1.2",
    "prompt": "^1.0.0",
    "shelljs": "^0.7.7"
  }
}

This already includes everything we need. Now to install the packages execute npm install and all the marked dependencies will be available in our project. The versions of these packages are the latest at the time of writing. If newer versions become available in the meantime, you might consider updating them (paying attention to any API changes).

Note the scaffold value in bin. It indicates the name of our command and the file that is going to be called every time we enter that command in our terminal (index.js). Feel free to change this value as you need.

Building the Entry Point

The first component of our CLI is the index.js file which contains a list of commands, options and the respective functions that are going to be available to us. But before writing this file, let’s start by defining what our CLI is going to do in a little more detail.

• The main (and only) command is create, which allow us to create a project boilerplate of our choice.
• The create command takes a mandatory template argument, that indicates which template we want to use.
• It also takes a --variant option that allows us to select a specific variation of our template.
• If no specific variant is supplied, it will use a default one (we will define this later).

Caporal.js allows us to define the above in a compact way. Let’s add the following content to our index.js file:

#!/usr/bin/env node

const prog = require('caporal');

prog
  .version('1.0.0')
  .command('create', 'Create a new application')
  .argument('<template>', 'Template to use')
  .option('--variant <variant>', 'Which <variant> of the template is going to be created')
  .action((args, options, logger) => {
    console.log({
      args: args,
      options: options
    });
  });

prog.parse(process.argv);

The first line is a Shebang to indicate that this is a Node.js executable.

The shebang included here only works for Unix-like systems. Windows has no shebang support, so if you want to execute the file directly on Windows you will have to look for a workaround. Running the command via npm (explained at the end of this section) will work on all platforms.

Next, we include the Caporal.js package as prog and we start defining our program. Using the command function, we define the create command as the first parameter and a little description as the second one. This will be shown in the automatically-generated help option for our CLI (using --help).

Then, we chain the template argument inside the argument function, and because it is a required argument we wrap it inside angular brackets (< and >).

We can define the variant option by writing --variant <variant> inside the option function. It means that the option for our command is called --variant and the value will be stored in a variant variable.

Finally, in the action command we pass another function that will handle the current command. This callback will be called with three arguments:

• passed arguments (args)
• passed options (options)
• a utility object to show things on screen (logger).

At this point, we are going to log out the values of the passed arguments and options, so we can get an idea of how to get the necessary information to perform an action from the CLI.

The last line passes the information from the scaffold command to the Caporal.js parser which will do the heavy lifting.

Make the CLI Available Globally

We can now test our application to see if everything is going according to plan. To do this, we need to make it globally available to our system using npm’s link command. Execute the following from the project root:

npm link

After the process is complete, we will be able to execute scaffold in our terminal inside any directory without having to make an explicit reference to our index.js file:

scaffold create node --variant mvc

And you should get this in response:

{ args: { template: 'node' }, options: { variant: 'mvc' } }

That’s a sample of the information that we will use next to create projects from templates.

Building a Template

Our templates will consist of the files and directory structure we need to get up and running with a certain type of project. Every template will have a package.json file with some placeholder values, which we can fill with our real data.

To start, create a templates directory in your project and a node directory inside of that. In the node directory, create a default directory (which will be used if we don’t provide a variant option) and a second directory called mvc (to create a Node.js project using the MVC architecture).

The final structure should look like this:

.
└── templates
    └── node
        ├── default
        └── mvc

Now we need to fill our default and mvc folders with project files. You can either create some of your own, or you can use those provided in the sample app.

Next we can proceed to put variable identifiers where we want dynamic values. Each template folder should contain a package.json file. Open these up and include any variables in capital letters (no spaces) and square brackets.

This is the package.json file inside our default template:

 {
  "name": "[NAME]",
  "version": "[VERSION]",
  "description": "[DESCRIPTION]",
  "main": "server.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node server.js",
    "start:dev": "nodemon server.js"
  },
  "author": "[AUTHOR]",
  "license": "[LICENSE]",
  "dependencies": {
    "dotenv": "^2.0.0",
    "hapi": "^16.1.0",
    "hoek": "^4.1.0"
  },
  "devDependencies": {
    "nodemon": "^1.11.0"
  }
}

After creating all the variables, put them inside a _variables.js file in the same template directory, like this:

/*
 * Variables to replace
 * --------------------
 * They are asked to the user as they appear here.
 * User input will replace the placeholder  values
 * in the template files
 */

module.exports = [
  'name',
  'version',
  'description',
  'author',
  'license'
];

The names in the exported array are the same in the files but in lowercase and without the square brackets. We will use this file to ask for each value in the CLI.

Now we can proceed to build the function for the create command that will do all the work.

Building the “Create” Function

In our index.js file, we were previously passing a simple function to action() that logged the values received by the CLI. Now we are going to replace that function with a new one that will copy the template files to the directory where the scaffold command is executed. We will also replace the placeholder variables with values obtained via user input.

Inside a lib directory (so to keep things organized), add a create.js file and put the following content inside:

module.exports = (args, options, logger) => {

};

We are going to put all the logic of our application inside this function, which means we need to alter our index.js file accordingly:

#!/usr/bin/env node

const prog = require('caporal');
const createCmd = require('./lib/create');

prog
  .version('1.0.0')
  .command('create', 'Create a new application')
  .argument('<template>', 'Template to use')
  .option('--variant <variant>', 'Which <variant> of the template is going to be created')
  .action(createCmd);

prog.parse(process.argv);

Importing Dependencies and Setting Variables

Now going back to the create.js file, we can put the following at the beginning of the file to make the required packages available:

const prompt = require('prompt');
const shell = require('shelljs');
const fs = require('fs');
const colors = require("colors/safe");

// Set prompt as green and use the "Replace" text
prompt.message = colors.green("Replace");

Note the customization setting for the prompt messages. This is entirely optional.

Inside the exported function, the first thing that we are going to add are some variables:

const variant = options.variant || 'default';
const templatePath = `${__dirname}/../templates/${args.template}/${variant}`;
const localPath = process.cwd();

As you can see, we are grabbing the variant option passed to the scaffold command and setting it to 'default' if this option was omitted. The variable templatePath contains the complete path for the specified template and localPath contains a reference to the directory where the command was executed.

Copying the Template Files

The process of copying the files is very straightforward using the cp function from shellJS. Below the variables we just included, add the following:

if (fs.existsSync(templatePath)) {
  logger.info('Copying files…');
  shell.cp('-R', `${templatePath}/*`, localPath);
  logger.info('✔ The files have been copied!');
} else {
  logger.error(`The requested template for ${args.template} wasn't found.`)
  process.exit(1);
}

First, we make sure that the template exists, if not we will exit the process showing an error message using the logger.error() function from Caporal.js. If the template exists, we will show a notification message using logger.info() and we will copy the files using shell.cp(). The -R option indicates that it should copy files recursively from the template path to the path where the command is being executed. Once the files are copied, we show a confirmation message. And because shellJS functions are synchronous, we don’t have to use callbacks, promises or anything similar — we just have to write code in a procedural way.

Replacing Variables

Although the idea of replacing variables in files sounds like a complicated thing to do, it is quite simple if we use the right tools. One of them is the classic sed editor from Unix systems which can transform text dynamically. ShellJS provides us with this utility which will work on both Unix systems (Linux and MacOS) as well as Windows.

To do all the replacements, add the following piece of code in your file, below the code we created before:

const variables = require(`${templatePath}/_variables`);

if (fs.existsSync(`${localPath}/_variables.js`)) {
  shell.rm(`${localPath}/_variables.js`);
}

logger.info('Please fill the following values…');

// Ask for variable values
prompt.start().get(variables, (err, result) => {

  // Remove MIT License file if another is selected
  // Omit this code if you have used your own template
  if (result.license !== 'MIT') {
    shell.rm(`${localPath}/LICENSE`);
  }

  // Replace variable values in all files
  shell.ls('-Rl', '.').forEach(entry => {
    if (entry.isFile()) {
      // Replace '[VARIABLE]` with the corresponding variable value from the prompt
      variables.forEach(variable => {
        shell.sed('-i', `\\[${variable.toUpperCase()}\\]`, result[variable], entry.name);
      });

      // Insert current year in files
      shell.sed('-i', '\\[YEAR\\]', new Date().getFullYear(), entry.name);
    }
  });

  logger.info('✔ Success!');
});

We start off by reading the , and variables is set to the content of the template’s _variables.js file that we created previously.

Then, because we have copied all the files from the template, the first if statement will remove the _variables.js file from our local directory since it is only needed in the CLI itself.

The value of each variable is obtained using the prompt tool, passing the array of variables to the get() function. In this way, the CLI will ask us a value for each item in this array and will save the result in an object called result which is passed to the callback function. This object contains each variable as a key and the entered text as the value.

The next if statement is only necessary if you are using the included templates in the repository since we also include a LICENSE file. Nonetheless, it is useful to see how we can retrieve a value for each variable, in this case from the license property using result.license. If the user enters a license other than MIT, then we delete the LICENSE file from the directory using the rm() function of ShellJS.

Now we get to the interesting part. By using the ls function from ShellJS, we can get a list of all the files in the current directory (.) where we are going to replace the variables. We pass it the -Rl option, so it becomes recursive and returns a file object instead of the file name.

We loop over the list of file objects using forEach() and for each one, we check if we are receiving a file using the isFile() function. If we get a directory, we don’t do anything.

Then for each file we get, we loop over all the variables and execute the sed function like this:

shell.sed('-i', `\\[${variable.toUpperCase()}\\]`, result[variable], entry.name);

Here we are passing the -i option which allows us to replace the text, then we pass a regex string that will match the variable identifier in uppercase and wrapped in square brackets ([ and ]). Then, each match of that regex will be replaced by the value for the corresponding variable (result[variable]) and finally we pass the name of the file that we are replacing from the forEach() function (entry.name).

The second sed is completely optional. This one is just to replace [YEAR] occurrences with the current year. Useful for LICENSE or README.md files.

And that’s it! We can now execute our command again in an empty directory to see how it generates a project structure and replaces all the variables with new values:

// To generate a Node.js MVC project
scaffold create node --variant mvc

// To generate a default Node.js project
scaffold create node

After executing the command it should start asking you for the value of the variables, and once the process is finished, it will show a success message. To check if everything went as expected, open a file containing variables, and you should see the text you entered during the CLI process instead of the identifiers in uppercase.

If you’ve used the templates from [the repo](https://github.com/sitepoint-editors/node-scaffolding-tool ) to follow along, you should also have generated working Node projects, which can be fired up by running npm install followed by npm start.

What to Do Next

We have successfully created a CLI tool to create new Node.js projects from templates, but we don’t have to stop here. Because we are building our tool from scratch, we have absolute freedom in what it can do. You can take the following ideas as inspiration:

• Extend the variables to replace code blocks instead of simple words; you can use more complicated regular expressions and capture groups in the sed function to achieve this.
• Add more commands to create specific files for each kind of project, like new models for the MVC template.
• Include commands to deploy the project to a server, which can be achieved by using libraries for rsync and remote commands via SSH.
• If you have a complicated setup, you can also try to add commands to build static assets or source files, which can be useful in case of a static site.
• Use the mv function to rename files from variable names.

Conclusion

In this tutorial, I demonstrated how we could build a CLI to start new projects in a quick way and in a familiar environment. But this is not a single-use project — you can extend it as you need. The creation of automatized tools is what characterizes developers. If you find yourself doing repetitive tasks, just stop and think if you can automatize it. Most of the time it is possible, and the long-term benefit can be huge.

Now it’s over to you? Do you love automating away repetitive and tedious work? What’s your toolkit of choice? Let me know in the comments below.

This article was peer reviewed by Joan Yin, Camilo Reyes and and Tim Severien. Thanks to all of SitePoint’s peer reviewers for making SitePoint content the best it can be!

Frequently Asked Questions (FAQs) about Caporal.js and Scaffolding Tools