Easy Multi-Language Twig Apps with Gettext

Share this article

There are many approaches for adding new languages to your application’s UI. Though some userland solutions like symfony/translation are arguably simpler to use, they’re slower than the good old native gettext by an order of several magnitudes.

In this tutorial, we’ll modify an English-only application to use gettext. Through this, we’ll demonstrate that getting internationalization up and running in an already existing app is not only possible, but relatively easy.

The application in question will be our own nofw – a ready-to-use skeleton app.

Do you speak English?

Key Takeaways

  • Gettext provides a more efficient solution for multi-language support in applications compared to other userland solutions like symfony/translation, due to its native implementation.
  • The tutorial guides through modifying an existing English-only application to support multiple languages using Gettext, demonstrating the process is straightforward and manageable.
  • Essential steps covered include setting up the environment, extracting translatable strings with xgettext, and generating language-specific .po and .mo files for use in the application.
  • Integration with Twig is achieved through the i18n extension, enabling the use of trans blocks within Twig templates to handle translations effectively.
  • Additional utility scripts are provided to streamline the management of language files and simplify the process of adding new languages to the application.

Bootstrapping and Basics

We’ll be using our trusty Homestead Improved as always as an environment – if you’d like to follow along, please fire it up. Our box already has gettext installed and activated. We’ll see how to manually install it for deployment purposes at the end of this tutorial.

Since nofw uses Twig, we’ll need the i18n extension. To start the project off right, here’s the full process:

git clone https://github.com/swader/nofw
cd nofw
git checkout tags/2.93 -b 2.93
composer require twig/extensions

Note: the above commands clone an older version of nofw – one without the internationalization features built in – so that readers can follow along with the tutorial.

This will install both Twig’s extensions, and all the project’s dependencies. Follow the procedure from the README to set up the rest of the nofw app (the database end), then return to this post.

The app should be up and running now.

Nofw working

The syntax for getting a translatable string is gettext("string") or its alias: _("string") – that is, _() is the function we call and "string" is the string we’re translating. If a translation for "string" isn’t found, then the original (which is considered a placeholder) value is returned. Placeholders are usually full strings in the most popular language for the site’s audience, so that if translation fails for some reason, readable text is still rendered.

Let’s try and make this work on a bogus PHP file, one that isn’t being powered by Twig, just to make sure everything is in working order. We’ll use the example from the old gettext post series. In the root of the project, we’ll make a file called i18n.php and give it the contents:

<?php

$language = "en_US.UTF-8";
putenv("LANGUAGE=" . $language); 
setlocale(LC_ALL, $language);

$domain = "messages"; // which language file to use
bindtextdomain($domain, "Locale"); 
bind_textdomain_codeset($domain, 'UTF-8');

textdomain($domain);

echo _("HELLO_WORLD");

In the same folder, let’s create a folder structure like this one:

Folder structure

Describing the code above, we first set the OS environment’s language as US English, then save that as an environment variable. PHP’s setlocale function uses the LC_ALL constant to switch all contexts to the given locale – so PHP will try to convert dates, numeric formatting, even currency to the locale we give it. Naturally, LC_ALL includes our custom translated messages, too.

The $domain field is there to tell PHP which language file to use – the language file will be called messages.po in its raw, editable form, and messages.mo in its compiled, machine readable form. bindtextdomain merely sets the path of the language file, which as we know is inside the Locale folder, and bind_textdomain_codeset will set the language character set. UTF-8 is a pretty universally safe bet here.

Finally, textdomain sets the active domain to be used.

Running this test script in the command line would echo the placeholder: HELLO_WORLD. Obviously, it’s missing the actual language file. It’s time to create it.

Extraction

Gettext comes with a handy tool for extracting placeholder strings from files. In the root of the project, we’ll execute:

xgettext --from-code=UTF-8 -o Locale/messages.pot public/i18n.php

Above, xgettext will use the UTF-8 encoding to output (-o) harvested strings from public/i18n.php into the given file. Inspecting the resulting messages.pot file now gives:

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2016-04-10 10:44+0000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=CHARSET\n"
"Content-Transfer-Encoding: 8bit\n"

#: public/i18n.php:13
msgid "HELLO_WORLD"
msgstr ""

.pot stands for portable object template. These template files are used to build other language files. If we decide to add Japanese to our app later on, the .pot file will be used to generate a Locale/ja_JP/LC_MESSAGES/messages.po which will, in turn, be used to generate the respective messages.mo file. Let’s use this approach to generate the en_US messages file now:

msginit --locale=en_US --output-file=Locale/en_US/LC_MESSAGES/messages.po --input=Locale/messages.pot

This process needs to be repeated for every new language we want added to the app.

The .po file is very similar to the .pot file from before, only it contains actual translation strings we can edit:

# English translations for PACKAGE package.
# Copyright (C) 2016 THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# vagrant <vagrant@homestead>, 2016.
#
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2016-04-10 10:44+0000\n"
"PO-Revision-Date: 2016-04-10 10:58+0000\n"
"Last-Translator: vagrant <vagrant@homestead>\n"
"Language-Team: English\n"
"Language: en_US\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=ASCII\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"

#: public/i18n.php:13
msgid "HELLO_WORLD"
msgstr "HELLO_WORLD"

After replacing the msgstr value of HELLO_WORLD with Howdy, we should compile the .po file into a .mo file Gettext can read:

msgfmt -c -o Locale/en_US/LC_MESSAGES/messages.mo Locale/en_US/LC_MESSAGES/messages.po

Adding a new language

To be sure things work, let’s add a new language – hr_HR (Croatian).

  1. First, we install the new locale onto the OS with:

    sudo locale-gen hr_HR hr_HR.UTF-8
    sudo update-locale
    sudo dpkg-reconfigure locales
    
  2. We then generate new .po files from the .pot files:

    mkdir -p Locale/hr_HR/LC_MESSAGES
    msginit --locale=hr_HR --output-file=Locale/hr_HR/LC_MESSAGES/messages.po --input=Locale/messages.pot
    
  3. Next, we change the HELLO_WORLD value into Zdravo, then generate the .mo file:

    msgfmt -c -o Locale/hr_HR/LC_MESSAGES/messages.mo Locale/hr_HR/LC_MESSAGES/messages.po
    
  4. Finally, we change the locale setting in the PHP file to hr_HR.UTF-8 and test.

Everything should be working fine.

Note: a restart of the web server and / or PHP-FPM might be necessary to clear the gettext cache.

Twig

Now that we know that gettext works fine and we’re able to add new languages on a whim, let’s see how it behaves in conjunction with Twig. First, let’s add the following into app/config/config_web.php, at the very top:

$language = "hr_HR.UTF-8";
putenv("LANGUAGE=" . $language);
setlocale(LC_ALL, $language);

$domain = "messages"; // which language file to use
bindtextdomain($domain, __DIR__."/../../Locale");
bind_textdomain_codeset($domain, 'UTF-8');

textdomain($domain);

For Twig to work with translatable strings, it needs the i18n extension we installed during the bootstrapping section. Then, in the templates, we use the trans block:

{% trans %}
    Hello {{ name }}!
{% endtrans %}

Of course, gettext has no idea what {{name}} is supposed to mean so Twig’s extension automatically compiles this into the gettext-friendly Hello %name%!. One caveat is that xgettext isn’t equipped to extract twig strings, so we need an alternative as per the docs.

We’ll compile our view templates into the system’s temporary folder, and then xgettext those, like regular PHP files!

First, let’s add a translatable message to one of the files. For example, somewhere into Standard/Views/home.twig, we can put:

    {% trans %}
        This is translatable
    {% endtrans %}

Then, in app/bin, we’ll create a new file: twigcache.php:

<?php

require __DIR__.'/../../vendor/autoload.php';
$shared = require __DIR__.'/../config/shared/root.php';

$tplDir = dirname(__FILE__) . '/templates';
$tmpDir = '/tmp/cache/';
$loader = new Twig_Loader_Filesystem($shared['site']['viewsFolders']);

// force auto-reload to always have the latest version of the template
$twig = new Twig_Environment(
    $loader, [
    'cache' => $tmpDir,
    'auto_reload' => true,
]
);
$twig->addExtension(new Twig_Extensions_Extension_I18n());
// configure Twig the way you want

// iterate over all your templates
foreach ($shared['site']['viewsFolders'] as $tplDir) {
    foreach (new RecursiveIteratorIterator(
                 new RecursiveDirectoryIterator($tplDir),
                 RecursiveIteratorIterator::LEAVES_ONLY
             ) as $file) {
        // force compilation
        if ($file->isFile()) {
            $twig->loadTemplate(str_replace($tplDir . '/', '', $file));
        }
    }
}

This file pulls in the common root.php configuration file in which view folders are defined, and as such we only need to update them in one place. Executing the script with php app/bin/twigcache.php now produces a directory tree with PHP cache files:

/tmp/cache
├── 1a
│   └── 1ad38dfd106734cda72279c3bbd83dd4c64d93ff9c713afb1e74904144018347.php
├── 1c
│   └── 1ca70331199383cea2ce308ab09447cebd7e5e81f2a7f5caa319d577f3a66682.php
...
├── df
│   └── df75e14ad2cb55315ab205872c8b8590ffde333912ec5c89e44c365479bfe457.php
└── f4
    └── f444ff725954cd5a9ec29ceb56a9cbf7eda8a273cea96c542c35a271e0f57c7e.php

We can unleash xgettext on this collection now:

xgettext -o Locale/messages.pot --from-code=UTF-8 -n --omit-header /tmp/cache/*/*.ph

Inspecting Locale/message.pot now reveals entirely new contents:

#: /tmp/cache/d0/d006e63c5a4c4e6a700d9273d4523dd0cf419105fa4b00cf6b89918c67df4b2b.php:56
msgid "This is translatable"
msgstr ""

As before, we can now create the .po files for our two pre-installed languages.

msgmerge -U Locale/en_US/LC_MESSAGES/messages.po Locale/messages.pot
msgmerge -U Locale/hr_HR/LC_MESSAGES/messages.po Locale/messages.pot

The msgmerge command merges the changes from messages.pot into the defined messages.po file. We use msgmerge instead of msginit here for convenience, but we could have also used msginit to start a new language file. Merge has an added bonus, though: seeing as xgettext no longer looked for translatable strings in the i18n.php from the example above, the newly updated .po files actually have the previously used string-value pair commented out:

#: /tmp/cache/d0/d006e63c5a4c4e6a700d9273d4523dd0cf419105fa4b00cf6b89918c67df4b2b.php:56
msgid "This is translatable"
msgstr "Yes, this is totally translatable"

#~ msgid "HELLO_WORLD"
#~ msgstr "Howdy"

This makes it easy to track deprecated translations without actually losing the effort it took to make them.

Assuming we changed some translation values, let’s compile to .mo now and test:

msgfmt -c -o Locale/hr_HR/LC_MESSAGES/messages.mo Locale/hr_HR/LC_MESSAGES/messages.po
msgfmt -c -o Locale/en_US/LC_MESSAGES/messages.mo Locale/en_US/LC_MESSAGES/messages.po

Working Croatian translation

Notice our translated string at the bottom there – everything works as expected!

Granted, the configuration we pasted to the top of config_web.php could use some work – like detecting the desired language through routes etc, but for brevity, this works fine.

Now all that’s left is hunting down all the strings in all the views and turning them into {% trans %} blocks!

Bonus: Scripts!

While the process above isn’t exactly complicated, it’d be simpler not to have to type out those long commands for every little thing. With more languages, things get more convoluted and confusing rather quickly, and it becomes ever easier to make a typo when punching in those shell commands. That’s why we can put together some shortcut bash scripts to help us out.

Note: if you’re not using nofw and don’t intend to, feel free to skip this section and/or just harvest what you think is useful from it. Likewise, note that all these scripts are meant to be run from the root folder of the project.


We’ll put all these scripts into app/bin/i18n/ and make them executable on the command line:

touch app/bin/i18n/{addlang.sh,update-pot.sh,update-mo.sh,config.sh}
chmod +x app/bin/i18n/*.sh

Config

LOCALE_FOLDER="Locale"
REGULAR_USER="forge"

[[ -f app/bin/i18n/config_local.sh ]] && source app/bin/i18n/config_local.sh

This script will be included by other scripts, which allows users to change the desired folder for the locales. Likewise, it contains the name of the non-sudo user. As it’s generally a bad idea to have many sudo commands inside a bash script, and we’ll certainly need to execute a lot of them with root privileges, we’ll opt to execute the whole script with sudo and then just drop privileges to the regular user on those commands that don’t need sudo. The user defaults to “forge” because that’s the user Laravel Forge sets up.

This script also includes another config script if it exists (because it’s in .gitignore and won’t exist on live servers) in which the username can be overridden. This is useful for local development. For example, when using Homestead Improved, everything will be run from the vagrant user’s perspective and the forge user doesn’t exist.

New language / refresh languages script

#!/usr/bin/env bash

# addlang.sh

source app/bin/i18n/config.sh

if [[ $EUID -ne 0 ]]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi

if [ -z "$1" ]; then
for folder in $(find ${LOCALE_FOLDER} -maxdepth 1 -type d | awk -F/ '{print $NF}')
do
    if [ "${folder}" != ${LOCALE_FOLDER} ]; then
        echo "Executing locale-gen ${folder} ${folder}.UTF-8"
        locale-gen ${folder} ${folder}.UTF-8
    fi
done
echo "Executing updates..."
update-locale
dpkg-reconfigure locales
fi

if [ -n "$1" ]; then
echo "Executing locale-gen $1 $1.UTF-8"
locale-gen $1 $1.UTF-8

echo "Executing updates..."
update-locale
dpkg-reconfigure locales

echo "Creating folder: ${LOCALE_FOLDER}/$1/LC_MESSAGES"
sudo -u ${REGULAR_USER} mkdir -p ${LOCALE_FOLDER}/$1/LC_MESSAGES
fi

This will immediately install any locale passed in as the first argument, like so:

sudo app/bin/i18n/addlang.sh ja_JP

It will also create the appropriate language folder in the Locale folder.

If no parameter was passed in, then this script will look for expected locales by traversing the Locale folder, and auto-installing each of the locales as per the subfolder name.

I.e., if there are folders Locale/en_US and Locale/hr_HR, it will be as if we had run sudo app/bin/i18n/addlang.sh en_US and sudo app/bin/i18n/addlang.sh hr_HR. This helps auto-install locales during deployment.

This script needs to be run as root because the locale-related commands require elevated privileges.

Refresh pot script

#!/usr/bin/env bash

# update-potpo.sh

source app/bin/i18n/config.sh

echo "Regenerating cache"
php app/bin/twigcache.php

echo "Running xgettext on the cached files"
xgettext -o ${LOCALE_FOLDER}/messages.pot --from-code=UTF-8 -n --omit-header /tmp/cache/*/*.php

for folder in $(find ${LOCALE_FOLDER} -maxdepth 1 -type d | awk -F/ '{print $NF}')
do
    if [ "${folder}" != ${LOCALE_FOLDER} ]; then
        if [[ -f ${LOCALE_FOLDER}/${folder}/LC_MESSAGES/messages.po ]]; then
            echo "Merging for ${folder}"
            msgmerge -U Locale/${folder}/LC_MESSAGES/messages.po ${LOCALE_FOLDER}/messages.pot
        else
            echo "Initializing for ${folder}"
            msginit --locale=${folder} --output-file=${LOCALE_FOLDER}/${folder}/LC_MESSAGES/messages.po --input=${LOCALE_FOLDER}/messages.pot
        fi
    fi
done

This regenerates the view cache, unleashes xgettext on it, and merges the result with the current .pot file, if any. It then uses the refreshed .pot file to update the .po files. Notice it uses msginit if the language hasn’t been initialized yet, and msgmerge otherwise.

Recompile script

#!/usr/bin/env bash

# update-mo.sh

source app/bin/i18n/config.sh

for folder in $(find ${LOCALE_FOLDER} -maxdepth 1 -type d | awk -F/ '{print $NF}')
do
    if [ "${folder}" != ${LOCALE_FOLDER} ]; then
        echo "Compiling .mo for ${folder}"
        msgfmt -c -o Locale/${folder}/LC_MESSAGES/messages.mo ${LOCALE_FOLDER}/${folder}/LC_MESSAGES/messages.po
    fi
done

The recompilation script is supposed to be run after edits to .po files have been made. It makes the edits ready for use, and allows the translations to appear on the site.

Deploying

Deployment of these language-specific upgrades will depend on the deployment approach applied to the app. We could be using Deployer, we could be using Forge, or something else entirely. What ever the case, before we even try out our scripts above we’ll need to make sure that:

  1. gettext is installed and activated
  2. the necessary locales have been generated on the OS

On Ubuntu, this is easily skipped by making sure the following commands run at the end of the deployment process:

sudo apt-get install gettext
sudo app/bin/i18n/addlang.sh

The rest is automatic, seeing as .pot, .po and .mo files are meant to be committed alongside the application’s source code.

Note that you’ll need to modify both the installation command and the shell scripts above if you’re using something other than Ubuntu

Conclusion

In this tutorial, we looked at adding internationalization features to an existing application powered by Twig. We demonstrated the use of gettext on a mock no-Twig file, made sure everything worked, and then went through a step by step integration with Twig. Finally, we wrote some shortcut scripts that can help tremendously when sharing the project or deploying it to production.

Do you use gettext? Or do your apps take a different approach? Let us know in the comments!

Frequently Asked Questions (FAQs) about Multi-Language Twig Apps with Gettext

How does Gettext work with Twig to create multi-language apps?

Gettext is a powerful internationalization (i18n) and localization (l10n) system commonly used in open source projects. It works with Twig, a flexible, fast, and secure template engine for PHP, to create multi-language apps. Gettext uses PO (Portable Object) files to store translated strings and MO (Machine Object) files for the actual use in the software. When integrated with Twig, it allows developers to easily switch between different languages in their apps, making the app more user-friendly for a global audience.

What are the benefits of using Gettext for localization in Twig apps?

Gettext offers several benefits for localization in Twig apps. It provides a standardized way of handling i18n and l10n, making it easier for developers to manage and update translations. It also supports plural forms and context, which are essential for accurate translations. Moreover, Gettext has wide support in various programming languages and tools, making it a versatile choice for developers.

How can I install and configure Gettext for my Twig app?

To install Gettext, you can use the package manager for your operating system. For example, on Ubuntu, you can use the command sudo apt-get install gettext. After installing Gettext, you need to configure it for your Twig app. This involves setting up the locale on your server, creating PO and MO files for each language, and integrating Gettext with Twig. You can use tools like Poedit to create and manage your PO and MO files.

How can I use Gettext functions in my Twig templates?

You can use Gettext functions in your Twig templates to mark strings for translation. The most commonly used functions are _() and gettext(), which translate a single string, and ngettext(), which translates a string with plural forms. You can use these functions in your Twig templates like this: {{ _('Hello, world!') }}.

How can I switch between different languages in my Twig app?

To switch between different languages in your Twig app, you need to change the locale on your server. You can do this using the setlocale() function in PHP. For example, to switch to French, you can use the command setlocale(LC_ALL, 'fr_FR.utf8'). After changing the locale, all Gettext functions will return translations in the new language.

How can I manage and update translations in my Twig app?

You can manage and update translations in your Twig app using PO and MO files. PO files are human-readable files that contain the original strings and their translations. You can edit these files to update translations. MO files are binary files that are used by Gettext at runtime. After updating a PO file, you need to compile it into a MO file for the changes to take effect.

What tools can I use to work with Gettext and Twig?

There are several tools you can use to work with Gettext and Twig. Poedit is a popular GUI tool for creating and managing PO and MO files. There are also various text editors and IDEs that support Gettext, such as Sublime Text, Atom, and PhpStorm. For integrating Gettext with Twig, you can use the Twig-Gettext-Extractor tool, which extracts translatable strings from Twig templates and generates PO files.

Can I use Gettext with other template engines or programming languages?

Yes, Gettext is a versatile system that can be used with various template engines and programming languages. Besides Twig and PHP, Gettext also supports languages like C, C++, Java, Python, and many others. This makes it a great choice for developers working on multi-language projects in different programming environments.

Are there any limitations or challenges in using Gettext for localization?

While Gettext is a powerful and flexible system, it does have some limitations. For example, it requires the locale to be installed on the server, which may not be possible in some hosting environments. It also doesn’t support right-to-left languages out of the box, although there are workarounds for this. Despite these challenges, Gettext remains a popular choice for localization due to its wide support and standardized approach.

Where can I find more resources to learn about Gettext and Twig?

There are many resources available online to learn about Gettext and Twig. The official documentation for both Gettext and Twig is a great starting point. There are also various tutorials, blog posts, and forums where you can find helpful information and examples. Additionally, you can check out open source projects that use Gettext and Twig to see how they handle i18n and l10n.

Bruno SkvorcBruno Skvorc
View Author

Bruno is a blockchain developer and technical educator at the Web3 Foundation, the foundation that's building the next generation of the free people's internet. He runs two newsletters you should subscribe to if you're interested in Web3.0: Dot Leap covers ecosystem and tech development of Web3, and NFT Review covers the evolution of the non-fungible token (digital collectibles) ecosystem inside this emerging new web. His current passion project is RMRK.app, the most advanced NFT system in the world, which allows NFTs to own other NFTs, NFTs to react to emotion, NFTs to be governed democratically, and NFTs to be multiple things at once.

bashbash scriptingBrunoSgettexti18ninternationalizationmultilanguagemultilingualnofwOOPHPPHPtwig
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week