At Bitfalls.com, we also use WordPress for now, and use the same peer review approach for content as we do at SitePoint.
We decided to build a tool which automatically pulls content from merged pull requests into articles, giving us the ability to fix typos and update posts from Github, and see the changes reflected on the live site. This tutorial will walk you through the creation of this tool, so you can start using it for your own WordPress site, or build your own version.
Key Takeaways
- Utilize Git and WordPress together to automatically pull content from merged pull requests into WordPress posts, ensuring updates and typo corrections are seamlessly integrated.
- Implement a hook processor in your WordPress setup to detect pushes to the master branch, automatically converting Markdown content to HTML and merging multilingual content in WPGlobus format before saving to the database.
- Set up a local testing environment using ngrok to pipe GitHub hook triggers to your machine, allowing for local debugging and testing of the webhook integration without deployment.
- Process the webhook data in WordPress using a custom PHP script that bypasses typical WordPress processing to handle incoming data more efficiently and securely.
- Use Parsedown for converting Markdown to HTML, maintaining consistency with the Markdown flavor used during content creation, and ensuring smooth integration into WordPress posts.
- Manage updates directly via WP-CLI commands for a streamlined process, enabling updates from file to handle special characters and automate content updates without manual intervention in the WordPress backend.
The Plan
The first part is identifying the problem and the situation surrounding it.
- we use WPGlobus for multi-language support, which means content gets saved like this:
{:en}English content{:}{:hr}Croatian content{:}
. - authors submit PRs via Github, the PRs are peer reviewed and merged, and then (currently) manually imported into WP’s Posts UI through the browser.
- every post has the same folder layout:
author_folder/post_folder/language/final.md
- this is slow and error prone, and sometimes mistakes slip by. It also makes updating posts tedious.
The solution is the following:
- add a hook processor which will detect pushes to the master branch (i.e. merges from PRs)
- the processor should look for a meta file in the commit which would contain information on where to save the updated content
- the processor automatically converts the MD content to HTML, merges the languages in the WPGlobus format, and saves them into the database
Bootstrapping
If you’d like to follow along (highly recommended), please boot up a good virtual machine environment, install the newest version of WordPress on it, and add the WPGlobus plugin. Alternatively, you can use a prepared WordPress box like VVV. Additionally, make sure your environment has ngrok installed – we’ll use that to pipe Github hook triggers to our local machine, so we can test locally instead of having to deploy.
Hooks
For this experiment, let’s create a new repository. I’ll call mine autopush.
In the settings of this repository, we need to add a new hook. Since we’re talking about a temporary Ngrok URL, let’s first spin that up. In my case, entering the following on the host machine does the trick:
ngrok http homestead.app:80
I was given the link http://03672a64.ngrok.io
, so that’s what goes into the webhook, with an arbitrary suffix like githook
. We only need push events. The json
data type is cleaner, so that’s selected as a preference, and the final webhook setup looks something like this:
Let’s test this now.
git clone https://github.com/swader/autopush
cd autopush
touch README.md
echo "This is a README file" >> README.md
git add -A
git commit -am "We're pushing for the first time"
git push origin master
The ngrok log screen should display something like this:
POST /githook/ 404 Not Found
This is fine. We haven’t made the /githook
endpoint yet.
Processing Webhooks
We’ll read this new data into WordPress with custom logic. Due to the spaghetti-code nature of WP itself, it’s easier to circumvent it entirely with a small custom application. First, we’ll create the githook
folder in the WordPress project’s root, and an index.php
file inside it. This makes the /githook/
path accessible, and the hook will no longer return 404, but 200 OK.
According to the docs, the payload will have a commits
field with a modified
field in each commit. Since we’re only looking to update posts, not schedule them or delete them – those steps are still manual, for safety – we’ll only be paying attention to that one. Let’s see if we can catch it on a test push.
First, we’ll save our request data to a text file, for debugging purposes. We can do this by modifying our githook/index.php
file:
<?php
file_put_contents('test.txt', file_get_contents('php://input'));
Then we’ll create a new branch, add a file, and push it online.
git checkout -b test-branch
touch testfile.md
git add testfile.md
git commit -am "Added test file"
git push origin test-branch
Sure enough, our test.json
file is filled with the payload now. This is the payload I got. You can see that we have only one commit, and that commit’s modified
field is empty, while the added
field has testfile.md
. We can also see this happened on refs/heads/test-branch
, ergo, we’re not interested in it. But what happens if we make a PR out of this branch and merge it?
Our payload looks different. Most notably, we now have refs/heads/master
as the ref
field, meaning it happened on the master
branch and we must pay attention to it. We also have 2 commits instead of just one: the first one is the same as in the original PR, the adding of the file. The second one corresponds to the change on the master branch: the merging itself. Both reference the same added
file.
Let’s do one final test. Let’s edit testfile.md
, push that, and do a PR and merge.
echo "Hello" >> testfile.md
git add testfile.md
git commit -am "Added test file"
git push origin test-branch
Ahh, there we go. We now have a modified file in the payload.
Now let’s do a “real” scenario and simulate an update submission. First we’ll create a post’s default folder, and then we’ll PR an update into it.
git checkout master
git pull
mkdir -p authors/some-author/some-post/{en_EN,hr_HR,images}
echo "English content" >> authors/some-author/some-post/en_EN/final.md
echo "Croatian content" >> authors/some-author/some-post/hr_HR/final.md
touch authors/some-author/some-post/images/.gitkeep
git add -A
git commit -am "Added some author"
git push origin master
Then we do the edit.
git checkout -b edit-for-some-post
echo "This is a new line" >> authors/some-author/some-post/en_EN/final.md
git add -A
git commit -am "Added an update on the English version of the post"
git push origin edit-for-some-post
If we turn this into a pull request in the Github web UI and merge the PR, we’ll get this payload.
If we follow the path from the modified files in the payload, we can easily discern the folder we’re talking about. Let’s modify the index.php
file from before.
$payload = json_decode($json, true);
$last_commit = array_pop($payload['commits']);
$modified = $last_commit['modified'];
$prefix = 'https://raw.githubusercontent.com/';
$repo = 'swader/autopush/master/';
$lvl = 2;
$folders = [];
foreach ($modified as $file) {
$folder = explode('/', $file);
$folder = implode('/', array_slice($folder, 0, -$lvl));
$folders[] = $folder;
}
$folders = array_unique($folders);
var_dump($folders);
We fetch the last commit in the payload, extract its modified files list, and find the parent folder of each modified file. The parent is dictated by the $lvl
variable – in our case it’s 2 because the folder is 2 levels up: one extra for language (en_EN
).
And there we have it – the path of the folder that holds the files that need to be updated. Now all we have to do is fetch the contents, turn the Markdown of those files into HTML, and save it into the database.
Processing Markdown
To process MarkDown, we can use the Parsedown package. We’ll install these dependencies in the githooks
folder itself, to make the app as standalone as possible.
composer require erusev/parsedown
Parsedown is the same flavor of Markdown we use at Bitfalls while writing with the Caret editor, so it’s a perfect match.
Now we can modify index.php
again.
$payload = json_decode($json, true);
$last_commit = array_pop($payload['commits']);
$modified = $last_commit['modified'];
$prefix = 'https://raw.githubusercontent.com/';
$repo = 'swader/autopush/';
$branch = 'master/';
$languages = [
'en_EN' => 'en',
'hr_HR' => 'hr'
];
$lvl = 2;
$folders = [];
foreach ($modified as $file) {
$folder = explode('/', $file);
$folder = implode('/', array_slice($folder, 0, -$lvl));
$folders[] = $folder;
}
$folders = array_unique($folders);
foreach ($folders as $folder) {
$fullFolderPath = $prefix.$repo.$branch.$folder.'/';
$content = '';
foreach ($languages as $langpath => $key) {
$url = $fullFolderPath.$langpath.'/final.md';
$content .= "{:$key}".mdToHtml(getContent($url))."{:}";
}
if (!empty($content)) {
// Save to database
}
}
function getContent(string $url): string {
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url.'?nonce='.md5(microtime()));
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
$data = curl_exec($ch);
$code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ($code != 200) {
return '';
}
curl_close($ch);
return $data;
}
function mdToHtml(string $text): string {
$p = new Parsedown();
$p->setUrlsLinked(true);
return $p->parse($text);
}
We made some really simple functions to avoid repetition. We also added a mapping of language folders (locales) to their WPGlobus keys, so that when iterating through all the files in a folder, we know how to delimit them in the post’s body.
Note: we have to update all language versions of a post when doing an update to just one, because WPGlobus doesn’t use an extra field or a different database row to save another language of a post – it saves them all in one field, so the whole value of that field needs to be updated.
We iterate through the folders that got updates (there might be more than one in a single PR), grab the contents of the file and convert it to HTML, then store all this into a WPGlobus-friendly string. Now it’s time to save this into the database.
Note: we used a nonce at the end of the URL to invalidate a possible cache issue with raw github content.
Saving Edited Content
We have no idea where to save the new content. We need to add support for meta files.
First, we’ll add a new function which gets this meta file:
function getMeta(string $folder): ?array {
$data = getContent(trim($folder, '/').'/meta.json');
if (!empty($data)) {
return json_decode($data, true);
}
return null;
}
Simple, if it exists, it’ll return its contents. The meta files will be JSON, so all the parsing we’ll ever need is already built into PHP.
Then, we’ll add a check to our main loop so that the process skips any folder without a meta file.
foreach ($folders as $folder) {
$fullFolderPath = $prefix.$repo.$branch.$folder.'/';
$meta = getMeta($fullFolderPath);
if (!$meta) {
continue;
}
// ...
We’ll use the WP CLI to make updates. The CLI can be installed with the following commands:
curl -O https://raw.githubusercontent.com/wp-cli/builds/gh-pages/phar/wp-cli.phar
sudo mv wp-cli.phar /usr/local/bin/wp
sudo chmod +x /usr/local/bin/wp
This downloads the WP-CLI tool, puts it into the server’s path (so it can be executed from anywhere), and adds “executable” permission to it.
The post update
command needs a post ID, and the field to update. WordPress posts are saved into the wp_posts
database table, and the field we’re looking to update is the post_content
field.
Let’s try this out in the command line to make sure it works as intended. First we’ll add an example post. I gave it an example title of “Example post” in English and “Primjer” in Croatian, with the body This is some English content for a post!
for the English content, and Ovo je primjer!
for the Croatian content. When saved, this is what it looks like in the database:
In my case, the ID of the post is 428. If your WP installation is fresh, yours will probably be closer to 1.
Now let’s see what happens if we execute the following on the command line:
wp post update 428 --post_content='{:en}This is some English content for a post - edited!{:}{:hr}Ovo je primjer - editiran!{:}'
Sure enough, our post was updated.
This looks like it might become problematic when dealing with quotes which would need to be escaped. It’s better if we update from file, and let this tool handle the quotes and such. Let’s give it a try.
Let’s put the content :en}This is some English 'content' for a post - edited "again"!{:}{:hr}Ovo je 'primjer' - editiran "opet"!{:}
into a file called updateme.txt
. Then…
wp post update 428 updateme.txt
Yup, all good.
Okay, now let’s add this into our tool.
For now, our meta file will only have the ID of the post, so let’s add one such file to the content repo.:
git checkout master
git pull
echo '{"id": 428}' >> authors/some-author/some-post/meta.json
git add -A
git commit -am "Added meta file for post 428"
git push origin master
Note: update the ID to match yours.
At this point, our content repo should look like this (version saved as release, feel free to clone).
Replace the // Save to database
line in the code from before and its surrounding lines with:
if (!empty($content) && is_numeric($meta['id'])) {
file_put_contents('/tmp/wpupdate', $content);
exec('wp post update '.$meta['id'].' /tmp/wpupdate', $output);
var_dump($output);
}
We make sure that both content and the ID of the post to be updated are somewhat valid, and then we write the contents into a temporary file, from which we then feed it to the wp cli
tool.
We should also add some more checks to the beginning of the script to make sure we only execute the updates we want to execute:
// ...
$payload = json_decode($json, true);
if (empty($json)) {
header("HTTP/1.1 500 Internal Server Error");
die('No data provided for parsing, payload invalid.');
}
if ($payload['ref'] !== 'refs/heads/master') {
die('Ignored. Not master.');
}
$last_commit = array_pop($payload['commits']);
// ...
The full index.php
file looks like this now:
<?php
require_once 'vendor/autoload.php';
$json = file_get_contents('php://input');
file_put_contents('test.json', $json);
$payload = json_decode($json, true);
if (empty($json)) {
header("HTTP/1.1 500 Internal Server Error");
die('No data provided for parsing, payload invalid.');
}
if ($payload['ref'] !== 'refs/heads/master') {
die('Ignored. Not master.');
}
$last_commit = array_pop($payload['commits']);
$modified = $last_commit['modified'];
$prefix = 'https://raw.githubusercontent.com/';
$repo = 'swader/autopush/';
$branch = 'master/';
$languages = [
'en_EN' => 'en',
'hr_HR' => 'hr'
];
$lvl = 2;
$folders = [];
foreach ($modified as $file) {
$folder = explode('/', $file);
$folder = implode('/', array_slice($folder, 0, -$lvl));
$folders[] = $folder;
}
$folders = array_unique($folders);
foreach ($folders as $folder) {
$fullFolderPath = $prefix.$repo.$branch.$folder.'/';
$meta = getMeta($fullFolderPath);
if (!$meta) {
continue;
}
$content = '';
foreach ($languages as $langpath => $key) {
$url = $fullFolderPath.$langpath.'/final.md';
$content .= "{:$key}".mdToHtml(getContent($url))."{:}";
}
if (!empty($content) && is_numeric($meta['id'])) {
file_put_contents('/tmp/wpupdate', $content);
exec('wp post update '.$meta['id'].' /tmp/wpupdate', $output);
var_dump($output);
}
}
function getContent(string $url): ?string {
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url.'?nonce='.md5(microtime()));
curl_setopt($ch, CURLOPT_FRESH_CONNECT, TRUE);
$data = curl_exec($ch);
$code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
if ($code != 200) {
return null;
}
curl_close($ch);
return $data;
}
function mdToHtml(string $text): string {
$p = new Parsedown();
$p->setUrlsLinked(true);
return $p->parse($text);
}
function getMeta(string $folder): ?array {
$data = getContent(trim($folder, '/').'/meta.json');
if (!empty($data)) {
return json_decode($data, true);
}
return null;
}
At this point, we can test things. Perfect chance for a brand new branch, too.
git checkout -b post-update
echo 'Adding a new line yay!' >> authors/some-author/some-post/en_EN/final.md
git add -A; git commit -am "Edit"; git push origin post-update
Let’s check our post out.
It works – deploying this script now is as simple as deploying the WP code of your app itself, and updating the webhook’s URL for the repo in question.
Conclusion
In true WordPress fashion, we hacked together a tool that took us less than an afternoon, but saved us days or weeks in the long run. The tool is now deployed and functioning adequately. There is, of course, room for updates. If you’re feeling inspired, try adding the following:
- modify the post updating procedure so that it uses
stdin
instead of a file, making it compatible with no-writable-filesystem hosts like AWS, Heroku, or Google Cloud. - custom output types: instead of fixed
{:en}{:}{:hr}{:}
, maybe someone else is using a different multi-language plugin, or doesn’t use one at all. This should be customizable somehow. - auto-insertion of images. Right now it’s manual, but the images are saved in the repo alongside the language versions and could probably be easily imported, auto-optimized, and added into the posts as well.
- staging mode – make sure the merged update first goes through to a staging version of the site before going to the main one, so the changes can be verified before being sent to master. Rather than having to activate and deactivate webhooks, why not make this programmable?
- a plugin interface: it would be handy to be able to define all this in the WP UI rather than in the code. A WP plugin abstraction around the functionality would, thus, be useful.
With this tutorial, our intention was to show you that optimizing workflow isn’t such a big deal when you take the time to do it, and the return on investment for sacrificing some time on getting automation up and running can be immense when thinking long term.
Any other ideas or tips on how to optimize this? Let us know!
Frequently Asked Questions about Git and WordPress
How can I use Git with WordPress for version control?
Git is a powerful tool for version control. It allows you to track changes, create branches, and collaborate with others. To use Git with WordPress, you need to first install Git on your local machine. Once installed, you can initialize a new Git repository in your WordPress directory using the command git init
. After making changes to your WordPress files, you can commit these changes using git commit -m "Your message here"
. You can then push these changes to a remote repository using git push origin master
.
What is a pull request in Git and how does it work with WordPress?
A pull request in Git is a way of proposing changes to a project. It allows you to submit your changes for review and then merge them into the project. With WordPress, you can use pull requests to update posts. When you make changes to a post, you can create a pull request and then merge it to update the post.
How can I use the wp_update_post
function in WordPress?
The wp_update_post
function in WordPress is used to update the content of a post. You can use it by passing an array of arguments to it. The array should include the ID of the post you want to update and the new content. For example, you can use it like this: wp_update_post(array('ID' => $post_id, 'post_content' => $new_content));
.
How can I contribute to the WordPress project on GitHub?
You can contribute to the WordPress project on GitHub by submitting pull requests. First, you need to fork the WordPress repository to your own GitHub account. Then, you can clone the repository to your local machine, make changes, commit them, and then push them to your forked repository. Finally, you can create a pull request to propose your changes to the WordPress project.
How can I use ElectronJS for pull requests?
ElectronJS provides a guide on how to submit pull requests. You need to first fork the repository you want to contribute to, clone it to your local machine, create a new branch, make changes, commit them, and then push them to your forked repository. You can then create a pull request to propose your changes.
What is the difference between wp_update_post
and wp_insert_post
?
The wp_update_post
function is used to update an existing post, while the wp_insert_post
function is used to create a new post. Both functions take an array of arguments, but the arguments for wp_insert_post
do not need to include an ID.
How can I use Hostinger for WordPress hosting?
Hostinger provides a user-friendly platform for WordPress hosting. You can use it by signing up for a hosting plan, installing WordPress, and then managing your site through the Hostinger control panel. They also provide tutorials on how to use various WordPress functions.
How can I collaborate with others on a WordPress project using Git?
Git allows you to collaborate with others on a WordPress project by using branches and pull requests. You can create a new branch for each feature or bug fix, make changes, commit them, and then create a pull request to propose your changes. Your collaborators can then review your changes and merge them into the project.
How can I use Git for WordPress theme development?
Git can be a powerful tool for WordPress theme development. You can use it to track changes, create branches, and collaborate with others. You can also use it to roll back changes if something goes wrong. To use Git for theme development, you need to initialize a new Git repository in your theme directory and then commit and push your changes as you work on the theme.
How can I automate the process of updating posts in WordPress using Git?
You can automate the process of updating posts in WordPress using Git by setting up a continuous integration (CI) pipeline. This pipeline can automatically pull changes from your Git repository, update your posts using the wp_update_post
function, and then push the changes to your WordPress site. You can set up this pipeline using tools like Jenkins or Travis CI.
Bruno is a blockchain developer and technical educator at the Web3 Foundation, the foundation that's building the next generation of the free people's internet. He runs two newsletters you should subscribe to if you're interested in Web3.0: Dot Leap covers ecosystem and tech development of Web3, and NFT Review covers the evolution of the non-fungible token (digital collectibles) ecosystem inside this emerging new web. His current passion project is RMRK.app, the most advanced NFT system in the world, which allows NFTs to own other NFTs, NFTs to react to emotion, NFTs to be governed democratically, and NFTs to be multiple things at once.