Key Takeaways
- Elasticsearch, an open-source storage engine built on Lucene, can be used with AngularJS to create a search engine for a recipe website.
- Elasticsearch uses JSON, making it compatible with OpenRecipes, an open-source project that provides recipes in JSON format.
- Elasticsearch provides a RESTful backend, allowing easy interaction with various environments. The JavaScript driver is used in this example.
- Angular is used to build the user interface for the recipe search site. It interacts with the Elasticsearch backend to fetch and display search results.
- When deploying the recipe search site, it’s important to secure the Elasticsearch instance as it doesn’t inherently have user or permission concepts. A proxy can be used to prevent unauthorized access.
Building a Recipe Search Engine
In this article, you’ll learn how to use Elasticsearch with AngularJS to create a search engine for recipes, just like the one at OpenRecipeSearch.com. Why recipes?- OpenRecipes exists, which makes our job a lot easier.
- Why not?
cd
to the directory you just unzipped, and run bin/elasticsearch
(bin/elasticsearch.bat
on Windows). Ta-da! You’ve just started your very own elasticsearch instance. Leave that running while you follow along.
One of the great features of Elasticsearch is its out-of-the-box RESTful backend, which makes it easy to interact with from many environments. We’ll be using the JavaScript driver, but you could use whichever one you like; the code is going to look very similar either way. If you like, you can refer to this handy reference (disclaimer: written by me).
Now, you’ll need a copy of the OpenRecipes database. It’s just a big file full of JSON documents, so it’s straightfoward to write a quick Node.js script to get them in there. You’ll need to get the JavaScript Elasticsearch library for this, so run npm install elasticsearch
. Then, create a file named load_recipes.js
, and add the following code.
var fs = require('fs');
var es = require('elasticsearch');
var client = new es.Client({
host: 'localhost:9200'
});
fs.readFile('recipeitems-latest.json', {encoding: 'utf-8'}, function(err, data) {
if (err) { throw err; }
// Build up a giant bulk request for elasticsearch.
bulk_request = data.split('\n').reduce(function(bulk_request, line) {
var obj, recipe;
try {
obj = JSON.parse(line);
} catch(e) {
console.log('Done reading');
return bulk_request;
}
// Rework the data slightly
recipe = {
id: obj._id.$oid, // Was originally a mongodb entry
name: obj.name,
source: obj.source,
url: obj.url,
recipeYield: obj.recipeYield,
ingredients: obj.ingredients.split('\n'),
prepTime: obj.prepTime,
cookTime: obj.cookTime,
datePublished: obj.datePublished,
description: obj.description
};
bulk_request.push({index: {_index: 'recipes', _type: 'recipe', _id: recipe.id}});
bulk_request.push(recipe);
return bulk_request;
}, []);
// A little voodoo to simulate synchronous insert
var busy = false;
var callback = function(err, resp) {
if (err) { console.log(err); }
busy = false;
};
// Recursively whittle away at bulk_request, 1000 at a time.
var perhaps_insert = function(){
if (!busy) {
busy = true;
client.bulk({
body: bulk_request.slice(0, 1000)
}, callback);
bulk_request = bulk_request.slice(1000);
console.log(bulk_request.length);
}
if (bulk_request.length > 0) {
setTimeout(perhaps_insert, 10);
} else {
console.log('Inserted all records.');
}
};
perhaps_insert();
});
Next, run the script using the command node load_recipes.js
. 160,000 records later, we have a full database of recipes ready to go. Check it out with curl
if you have it handy:
$ curl -XPOST http://localhost:9200/recipes/recipe/_search -d '{"query": {"match": {"_all": "cake"}}}'
Now, you might be OK using curl
to search for recipes, but if the world is going to love your recipe search, you’ll need to…
Build a Recipe Search UI
This is where Angular comes in. I chose Angular for two reasons: because I wanted to, and because Elasticsearch’s JavaScript library comes with an experimental Angular adapter. I’ll leave the design as an exercise to the reader, but I’ll show you the important parts of the HTML. Get your hands on Angular and Elasticsearch now. I recommend Bower, but you can just download them too. Open yourindex.html
file and insert them wherever you usually put your JavaScript (I prefer just before the closing body
tag myself, but that’s a whole other argument):
<script src="path/to/angular/angular.js"></script>
<script src="path/to/elasticsearch/elasticsearch.angular.js"></script>
Now, let’s stop to think about how our app is going to work:
- The user enters a query.
- We send the query as a search to Elasticsearch.
- We retrieve the results.
- We render the results for the user.
- HTML attributes starting with
ng
are Angular directives. - The dynamic parts of your Angular applications are enclosed with an
ng-app
and anng-controller
.ng-app
andng-controller
don’t need to be on the same element, but they can be. - All other references to variables in the HTML refer to properties on the
$scope
object that we’ll meet in the JavaScript. - The parts enclosed in
{{}}
are template variables, like in Django/Jinja2/Liquid/Mustache templates.
<div ng-app="myOpenRecipes" ng-controller="recipeCtrl">
<!-- The search box puts the term into $scope.searchTerm
and calls $scope.search() on submit -->
<section class="searchField">
<form ng-submit="search()">
<input type="text" ng-model="searchTerm">
<input type="submit" value="Search for recipes">
</form>
</section>
<!-- In results, we show a message if there are no results, and
a list of results otherwise. -->
<section class="results">
<div class="no-recipes" ng-hide="recipes.length">No results</div>
<!-- We show one of these elements for each recipe in $scope.recipes.
The ng-cloak directive prevents our templates from showing on load. -->
<article class="recipe" ng-repeat="recipe in recipes" ng-cloak>
<h2>
<a ng-href="{{recipe.url}}">{{recipe.name}}</a>
</h2>
<ul>
<li ng-repeat="ingredient in recipe.ingredients">{{ ingredient }}</li>
</ul>
<p>
{{recipe.description}}
<a ng-href="{{recipe.url}}">... more at {{recipe.source}}</a>
</p>
</article>
<!-- We put a link that calls $scope.loadMore to load more recipes
and append them to the results.-->
<div class="load-more" ng-hide="allResults" ng-cloak>
<a ng-click="loadMore()">More...</a>
</div>
</section>
Now, we can start writing our JavaScript. We’ll start with the module, which we decided above would be called myOpenRecipes
(via the ng-app
attribute).
/**
* Create the module. Set it up to use html5 mode.
*/
window.MyOpenRecipes = angular.module('myOpenRecipes', ['elasticsearch'],
['$locationProvider', function($locationProvider) {
$locationProvider.html5Mode(true);
}]
);
For those new to Angular, the ['$locationProvider', function($locationProvider) {...}]
business is our way of telling Angular that we’d like it to pass $locationProvider
to our handler function so we can use it. This system of dependency injection removes the need for us to rely on global variables (except the global angular
and the MyOpenRecipes
we just created).
Next, we’ll write the controller, named recipeCtrl
. We need to make sure to initialize the recipes
, allResults
, and searchTerm
variables used in the template, as well as providing search()
and loadMore()
as actions.
/**
* Create a controller to interact with the UI.
*/
MyOpenRecipes.controller('recipeCtrl', ['recipeService', '$scope', '$location', function(recipes, $scope, $location) {
// Provide some nice initial choices
var initChoices = [
"rendang",
"nasi goreng",
"pad thai",
"pizza",
"lasagne",
"ice cream",
"schnitzel",
"hummous"
];
var idx = Math.floor(Math.random() * initChoices.length);
// Initialize the scope defaults.
$scope.recipes = []; // An array of recipe results to display
$scope.page = 0; // A counter to keep track of our current page
$scope.allResults = false; // Whether or not all results have been found.
// And, a random search term to start if none was present on page load.
$scope.searchTerm = $location.search().q || initChoices[idx];
/**
* A fresh search. Reset the scope variables to their defaults, set
* the q query parameter, and load more results.
*/
$scope.search = function() {
$scope.page = 0;
$scope.recipes = [];
$scope.allResults = false;
$location.search({'q': $scope.searchTerm});
$scope.loadMore();
};
/**
* Load the next page of results, incrementing the page counter.
* When query is finished, push results onto $scope.recipes and decide
* whether all results have been returned (i.e. were 10 results returned?)
*/
$scope.loadMore = function() {
recipes.search($scope.searchTerm, $scope.page++).then(function(results) {
if (results.length !== 10) {
$scope.allResults = true;
}
var ii = 0;
for (; ii < results.length; ii++) {
$scope.recipes.push(results[ii]);
}
});
};
// Load results on first run
$scope.loadMore();
}]);
You should recognize everything on the $scope
object from the HTML. Notice that our actual search query relies on a mysterious object called recipeService
. A service is Angular’s way of providing reusable utilities for doing things like talking to outside resources. Unfortunately, Angular doesn’t provide recipeService
, so we’ll have to write it ourselves. Here’s what it looks like:
MyOpenRecipes.factory('recipeService', ['$q', 'esFactory', '$location', function($q, elasticsearch, $location) {
var client = elasticsearch({
host: $location.host() + ':9200'
});
/**
* Given a term and an offset, load another round of 10 recipes.
*
* Returns a promise.
*/
var search = function(term, offset) {
var deferred = $q.defer();
var query = {
match: {
_all: term
}
};
client.search({
index: 'recipes',
type: 'recipe',
body: {
size: 10,
from: (offset || 0) * 10,
query: query
}
}).then(function(result) {
var ii = 0, hits_in, hits_out = [];
hits_in = (result.hits || {}).hits || [];
for(; ii < hits_in.length; ii++) {
hits_out.push(hits_in[ii]._source);
}
deferred.resolve(hits_out);
}, deferred.reject);
return deferred.promise;
};
// Since this is a factory method, we return an object representing the actual service.
return {
search: search
};
}]);
Our service is quite barebones. It exposes a single method, search()
, that allows us to send a query to Elasticsearch’s, searching across all fields for the given term. You can see that in the query
passed in the body of the call to search
: {"match": {"_all": term}}
. _all
is a special keyword that lets us search all fields. If instead, our query was {"match": {"title": term}}
, we would only see recipes that contained the search term in the title.
The results come back in order of decreasing “score”, which is Elasticsearch’s guess at the document’s relevance based on keyword frequency and placement. For a more complicated search, we could tune the relative weights of the score (i.e. a hit in the title is worth more than in the description), but the default seems to do pretty well without it.
You’ll also notice that the search accepts an offset
argument. Since the results are ordered, we can use this to fetch more results if requested by telling Elasticsearch to skip the first n results.
Some Notes on Deployment
Deployment is a bit beyond the scope of this article, but if you want to take your recipe search live, you need to be careful. Elasticsearch has no concept of users or permissions. If you want to prevent just anyone from adding or deleting recipes, you’ll need to find some way to prevent access to those REST endpoints on your Elasticsearch instance. For example, OpenRecipeSearch.com uses nginx as a proxy in front of Elasticsearch to prevent outside access to all endpoints butrecipes/recipe/_search
.
Congratulations, You’ve Made a Recipe Search
Now, if you openindex.html
in a browser, you should see an unstyled list of recipes, since our controller fetches some randomly for you on page load. If you enter a new search, you’ll get 10 results relating to whatever you searched for, and if you click “More…” at the bottom of the page, some more recipes should appear (if there are indeed more recipes to fetch).
That’s all there is to it! You can find all the necessary files to run this project on GitHub.
Frequently Asked Questions (FAQs) on Building a Recipe Search Site with Angular and Elasticsearch
How Can I Implement Pagination in Angular and Elasticsearch?
Implementing pagination in Angular and Elasticsearch involves using the ‘from’ and ‘size’ parameters in your Elasticsearch query. The ‘size’ parameter determines the number of search hits to return, while the ‘from’ parameter allows you to define the starting point for the search. In your Angular service, you can create a function that accepts the current page and items per page as parameters, then calculates the ‘from’ parameter and includes it in the Elasticsearch query.
How Can I Optimize My Elasticsearch Queries for Better Performance?
Optimizing Elasticsearch queries can significantly improve the performance of your application. One way to optimize your queries is by using filters instead of queries whenever possible, as filters are cacheable and often faster. You can also use the ‘explain’ API to understand how Elasticsearch is scoring your queries, and adjust them accordingly. Additionally, consider using the ‘profile’ API to identify slow parts of your queries.
How Can I Handle Errors in Angular and Elasticsearch?
Error handling is crucial for any application. In Angular, you can use the ‘catchError’ operator from RxJS to catch and handle errors in your HTTP requests. In Elasticsearch, you can use the ‘error_trace’ parameter in your queries to get more detailed information about any errors that occur. You can then display these errors to the user in a user-friendly way, or log them for further investigation.
How Can I Secure My Elasticsearch Instance?
Securing your Elasticsearch instance is crucial to protect your data. Elasticsearch provides several security features, such as encryption, role-based access control, and audit logging. You can also use a reverse proxy to add an additional layer of security. In addition, always ensure that your Elasticsearch instance is not publicly accessible and that you regularly update it to the latest version to benefit from the latest security patches.
How Can I Test My Angular and Elasticsearch Application?
Testing is an essential part of software development. For your Angular application, you can use tools like Jasmine and Karma for unit testing, and Protractor for end-to-end testing. For Elasticsearch, you can use the Elasticsearch Test Framework to write integration tests for your queries. You can also use tools like Postman to manually test your API endpoints.
How Can I Deploy My Angular and Elasticsearch Application?
Deploying your Angular and Elasticsearch application involves several steps. First, you need to build your Angular application for production using the ‘ng build –prod’ command. Then, you can serve your application using a web server like Nginx or Apache. For Elasticsearch, you can use the Elasticsearch Service on Elastic Cloud, which allows you to easily deploy, manage, and scale your Elasticsearch cluster.
How Can I Implement Real-Time Search with Angular and Elasticsearch?
Implementing real-time search with Angular and Elasticsearch can be achieved using the ‘search_after’ parameter in your Elasticsearch queries. This allows you to retrieve fresh results without having to re-execute the entire query. In your Angular application, you can use RxJS and the ‘debounceTime’ operator to delay the execution of the search function until the user has stopped typing, providing a smoother user experience.
How Can I Use Elasticsearch Aggregations with Angular?
Elasticsearch aggregations allow you to group your data in various ways, providing powerful analytics capabilities. You can use the ‘aggs’ parameter in your Elasticsearch queries to define your aggregations. In your Angular application, you can then process the aggregation results and display them to the user in a meaningful way, such as in charts or tables.
How Can I Implement Autocomplete with Angular and Elasticsearch?
Implementing autocomplete with Angular and Elasticsearch can be achieved using the ‘suggest’ feature of Elasticsearch. This allows you to provide real-time suggestions to the user as they type. In your Angular application, you can use the ‘map’ operator from RxJS to process the suggestions and display them to the user.
How Can I Monitor the Performance of My Elasticsearch Cluster?
Monitoring the performance of your Elasticsearch cluster is crucial to ensure its health and performance. Elasticsearch provides several monitoring tools, such as the Elasticsearch Monitoring API and Kibana’s Monitoring app. These tools provide detailed metrics about your cluster, such as CPU usage, memory usage, and query latency, allowing you to identify and resolve any performance issues.
Adam makes a lot of websites. He presently develops full-time for Tapstream, freelances occasionally and posts unsaleable articles on his blog.