Master Mobile-Friendly Image Manipulation: Emboss Images Using Android’s Renderscript API
During your Android development career, you may have to write apps that blur, sharpen, convert to grayscale, emboss, or otherwise process images. Even if your latest app isn’t necessarily focused on manipulating imagery, you can still employ interface design techniques such as blurring background content to focus the user on important alerts in the foreground, or using grayscale to provide contrast between available and unavailable options within the interface. To perform these operations quickly while remaining portable, your Android app must use Renderscript.
This article focuses on embossing images via Renderscript. It first introduces you to the embossing algorithm, then it presents a Renderscript primer, and finally it shows you how to build a simple app that embosses an image in a Renderscript context.
Understanding the Embossing Algorithm
Books written in Braille, documents sealed with hot wax to guarantee their authenticity, and coins illustrate embossing, the act of raising content in relief from a surface. Embossing helps the blind read via the Braille system of raised dots, and it provides authentication for stamped, sealed documents or minted coins. (In the past, these seals guaranteed the weights and values of precious metals used in the coins.) Along with these real-world uses, embossing also lends a three-dimensional chiseled look to images. Figure 1 presents an example.
Figure 1: An embossed image also appears somewhat metallic.
Think of an image as mountainous terrain. Each pixel represents an elevation: brighter pixels represent higher elevations. When an imaginary light source shines down on this terrain, the “uphills” that face the light source are lit, while the “downhills” that face away from the light source are shaded. An embossing algorithm captures this information.
The algorithm scans an image in the direction that a light ray is moving. For example, if the light source is located to the image’s left, its light rays move from left to right, so the scan proceeds from left to right. During the scan, adjacent pixels (in the scan direction) are compared. The difference in intensities is represented by a specific level of gray (from black, fully shaded, to white, fully lit) within the destination image.
There are eight possible scanning directions: left-to-right, right-to-left, top-to-bottom, bottom-to-top, and four diagonal directions. To simplify the code, the algorithm usually scans left-to-right and top-to-bottom, and chooses its neighbors as appropriate. For example, if the light source is above and to the left of the image, the algorithm would compare the pixel above and to the left of the current pixel with the current pixel during the left-to-right and top-to-bottom scan. The comparison is demonstrated in Listing 1’s embossing algorithm pseudocode:
FOR row = 0 TO height-1
FOR column = 0 TO width-1
SET current TO src.rgb[row]
SET upperLeft TO 0
IF row > 0 AND column > 0
SET upperLeft TO
src.rgb[row-1][column-1]
SET redIntensityDiff TO
red (current)-red (upperLeft)
SET greenIntensityDiff TO
green (current)-green (upperLeft)
SET blueIntensityDiff TO
blue (current)-blue (upperLeft)
SET diff TO redIntensityDiff
IF ABS(greenIntensitydiff) > ABS(diff)
SET diff TO greenIntensityDiff
IF ABS(blueIntensityDiff) > ABS(diff)
SET diff TO blueIntensityDiff
SET grayLevel TO
MAX(MIN(128+diff, 255), 0)
SET dst.rgb[row] TO grayLevel
NEXT column
NEXT rowListing 1: This algorithm embosses an image in a left-to-right and top-to-bottom manner.
Listing 1’s embossing pseudocode identifies the image to be embossed as src and identifies the image to store the results of the embossing as dst. These images are assumed to be rectangular buffers of RGB pixels (rectangular buffers of RGBA pixels where the alpha component is ignored could be accommodated as well).
Because pixels in the topmost row don’t have neighbors above them, and because pixels in the leftmost column don’t have neighbors to their left, there is a bit of unevenness along the top and left edges of the embossed image, as shown in Figure 1. You can eliminate this unevenness by drawing a solid border around the image.
After obtaining the red, green, and blue intensities for each of the current pixels and their respective upper-left neighbors, the pseudocode calculates the difference in each intensity. The upper-left neighbor’s intensities are subtracted from the current pixel’s intensities because the light ray is moving in an upper-left to lower-right direction.
The pseudocode identifies the greatest difference (which might be negative) between the current pixel and its upper-left neighbor’s three intensity differences. That’s done to obtain the best possible “chiseled” look. The difference is then converted to a level of gray between 0 and 255, and that gray level is stored in the destination image at the same location as the current pixel in the source image.
Renderscript Primer
Renderscript combines a C99-based language with a pair of Low Level Virtual Machine (LLVM) compilers, and a runtime that provides access to graphics and compute engines. You can use Renderscript to write part of your app in native code to improve 3D graphics rendering and/or data processing performance, all while keeping your app portable and avoiding use of the tedious Java Native Interface.
Note: Google introduced Renderscript in Android 2.0, but did not make an improved version publicly available until the Android 3.0 release. Because of subsequent developer feedback that favored working directly with OpenGL, Google deprecated the graphics engine portion of Renderscript starting with Android 4.1.
Embossing and other image processing operations are performed with Renderscript’s compute engine. Although you could leverage Android’s GPU-accelerated (as of Android 3.0) 2D Canvas API to perform these 2D tasks, Renderscript lets you move beyond the Dalvik virtual machine, write faster native code, and run this code via multiple threads on available CPU cores (and GPU and DSP cores in the future).
When introducing Renderscript to your app, you will need to write Java code and C99-based Renderscript code, which is known as a script. The Java code works with types located in the android.renderscript package. At minimum, the types you will need to access are Renderscript (for getting a context) and Allocation (for allocating memory on behalf of Renderscript):
Renderscriptdeclares astatic RenderScript create(Context ctx)method that is passed a reference to your current activity, and that returns a reference to aRenderscriptobject. This object serves as a context that must be passed to other APIs.Allocationdeclares methods for wrapping data on the Java side and binding this data to the Renderscript runtime, so that the script can access the data. Methods are also available for copying script results back to the Java side.
During the Android build process, LLVM’s front-end compiler compiles the script and creates a ScriptC_-prefixed class whose suffix is taken from the script file’s name. Your app instantiates this class and uses the resulting instance to initialize and invoke the script, and to retrieve script results. You don’t have to wait for script results because Renderscript takes care of this for you.
The C99 code is typically stored in a single file that is organized as follows:
- A pair of
#pragmadirectives must be specified and typically appear at the top of the file. One#pragmaidentifies the Renderscript version number (currently 1) to which the file conforms. The other#pragmaidentifies the Java package of the app to which the file associates. - A pair of
rs_allocationdirectives typically follow and identify the input and output allocations. The input allocation refers to the data that is to be processed (and which is wrapped by oneAllocationinstance), and the output allocation identifies where results are to be stored (and is wrapped by the otherAllocationinstance). - An
rs_scriptdirective typically follows and identifies theScriptC_-prefixed instance on the Java side, so that Renderscript can communicate results back to the script. - Additional global variables can be declared that are either local to the script or bound to the Java code. For example, you could declare
int width;andint height;, and pass an image’s width and height from your Java code (via calls toset_-prefixed methods on theScriptC_-prefixed object) to these variables. - A
root()function typically follows that is executed on each CPU core for each element in the input allocation. - A
noargument initfunction typically follows that performs any necessary initialization and invokes one of the overloadedrsForEach()functions. This function is responsible for identifying the number of cores, creating threads that run on these cores, and more. Each thread invokesroot().
You can place these items wherever you want in the file. However, rs_allocation-based, rs_script-based, and any other global variables must be declared before they are referenced.
Embossing Meets Renderscript
Now that you have a basic understanding of the embossing algorithm and Renderscript, let’s bring them together. This section first introduces you to the Java and C99 sides of an app (a single activity and an embossing script) that demonstrates embossing via Renderscript, and then shows you how to build this app.
Exploring the Java Side of an Embossing App
Listing 2 presents the EmbossImage activity class:
package ca.tutortutor.embossimage;
import android.app.Activity;
import android.os.Bundle;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.renderscript.Allocation;
import android.renderscript.RenderScript;
import android.view.View;
import android.widget.ImageView;
public class EmbossImage extends Activity
{
boolean original = true;
@Override
public void onCreate(Bundle savedInstanceState)
{
super.onCreate(savedInstanceState);
final ImageView iv = new ImageView(this);
iv.setImageResource(R.drawable.leopard);
setContentView(iv);
iv.setOnClickListener(new View.OnClickListener()
{
@Override
public void onClick(View v)
{
if (original)
drawEmbossed(iv, R.drawable.leopard);
else
iv.setImageResource(R.drawable.leopard);
original = !original;
}
});
}
private void drawEmbossed(ImageView iv, int imID)
{
Bitmap bmIn = BitmapFactory.decodeResource(getResources(), imID);
Bitmap bmOut = Bitmap.createBitmap(bmIn.getWidth(), bmIn.getHeight(),
bmIn.getConfig());
RenderScript rs = RenderScript.create(this);
Allocation allocIn;
allocIn = Allocation.createFromBitmap(rs, bmIn,
Allocation.MipmapControl.MIPMAP_NONE,
Allocation.USAGE_SCRIPT);
Allocation allocOut = Allocation.createTyped(rs, allocIn.getType());
ScriptC_emboss script = new ScriptC_emboss(rs, getResources(),
R.raw.emboss);
script.set_in(allocIn);
script.set_out(allocOut);
script.set_script(script);
script.invoke_filter();
allocOut.copyTo(bmOut);
iv.setImageBitmap(bmOut);
}
}Listing 2: The embossing app’s activity communicates with the embossing script via script.
The overriding void onCreate(Bundle bundle) method is responsible for setting up the activity’s user interface, which consists of a single android.widget.ImageView instance to which a click listener is registered. When the user clicks this view, either the embossed or the original version of this image is displayed, as determined by the current value of boolean variable original.
Communication with the script occurs in the void drawEmbossed(ImageView iv, int imID) method. This method first obtains an android.graphics.Bitmap object containing the contents of the image to be embossed, and creates a Bitmap object containing an empty bitmap with the same dimensions and configuration as the other object’s bitmap.
Next, a Renderscript context is obtained by executing RenderScript.create(this). The resulting Renderscript object is passed as the first argument to each of the two Allocation factory methods along with the ScriptC_emboss constructor. This context is analogous to the android.content.Context argument passed to widget constructors such as ImageView(Context context).
The Dalvik virtual machine and surrounding framework control memory management. Objects and arrays are always allocated on the Java side and subsequently made available to the script by wrapping them in Allocation objects. This class provides several factory methods for returning Allocation instances, including the following pair:
static Allocation createFromBitmap(RenderScript rs, Bitmap b, Allocation.MipmapControl mips, int usage)creates an allocation that wraps aBitmapinstance with specific mipmap behavior and usage.bmInis passed tobidentifying the inputBitmapinstance (containing the image) as the instance to be wrapped.Allocation.MipmapControl.MIPMAP_NONEis passed tomips, indicating that no mipmaps are used.Allocation.USAGE_SCRIPTis passed tousage, indicating that the allocation is bound to and accessed by the script.Allocation createTyped(RenderScript rs, Type type)creates an allocation for use by the script with the size specified bytypeand no mipmaps generated by default.allocIn.getType()is passed totype, indicating that the layout for the memory to be allocated is specified by the previously created input allocation.
Following creation of the input and output allocations, the ScriptC_emboss class (created by LLVM during the app build process) is instantiated by invoking ScriptC_emboss(rs, getResources(), R.raw.emboss). Along with the previously created Renderscript context, the following arguments are passed to the constructor:
- a reference to an
android.content.res.Resourcesobject (for accessing app resources). This reference is returned by invokingContext‘sResources getResources()method. R.raw.emboss(the resource ID of the embossing script). At build time, the LLVM front-end compiler compiles the script into a file (with a.bcextension) containing portable bitcode that is stored in the APK under theres/rawhierarchy. At runtime, the LLVM back-end compiler extracts this file and compiles it into device-specific code (unless the file has been compiled and cached on the device).
The LLVM front-end compiler generates a set_-prefixed method for each non-static variable that it finds in the script. Because the embossing script contains two rs_allocation directives (named in and out) and an rs_script directive (named script), set_in(), set_out(), and set_script() methods are generated and used to pass allocIn, allocOut, and script to the script.
A script is executed by invoking an invoke_-prefixed method on an instance of the ScriptC_-prefixed class. The suffix for this method is taken from the name of the noargument init function (with a void return type) that appears in the script. In this case, that name is filter.
Because Renderscript handles threading issues, allocOut.copyTo(bmOut); can be executed immediately after the invocation, to copy the script results to the output bitmap, and the resulting image is subsequently displayed via iv.setImageBitmap(bmOut);.
Exploring the C99 Side of an Embossing App
Listing 3 presents the embossing script.
#pragma version(1)
#pragma rs java_package_name(ca.tutortutor.embossimage)
rs_allocation out;
rs_allocation in;
rs_script script;
void root(const uchar4* v_in, uchar4* v_out, const void* usrData, uint32_t x,
uint32_t y)
{
float4 current = rsUnpackColor8888(*v_in);
float4 upperLeft = { 0, 0, 0, 0 };
if (x > 0 && y > 0)
upperLeft = rsUnpackColor8888(*(uchar*) rsGetElementAt(in, x-1, y-1));
float rDiff = current.r-upperLeft.r;
float gDiff = current.g-upperLeft.g;
float bDiff = current.b-upperLeft.b;
float diff = rDiff;
if (fabs(gDiff) > fabs(diff))
diff = gDiff;
if (fabs(bDiff) > fabs(diff))
diff = bDiff;
float grayLevel = fmax(fmin(0.5f+diff, 1.0f), 0);
current.r = grayLevel;
current.g = grayLevel;
current.b = grayLevel;
*v_out = rsPackColorTo8888(current.r, current.g, current.b, current.a);
}
void filter()
{
rsDebug("RS_VERSION = ", RS_VERSION);
#if !defined(RS_VERSION) || (RS_VERSION < 14)
rsForEach(script, in, out, 0);
#else
rsForEach(script, in, out);
#endif
}Listing 3: The filter() function is the embossing script’s entry point.
The script begins with a pair of #pragmas that identify 1 as the Renderscript version to which the script targets and ca.tutortutor.embossimage as the APK package. A pair of rs_allocation globals and an rs_script global follow, providing access to their Java counterparts (passed to the script via the previously mentioned set_-prefixed method calls).
The subsequently declared void root(const uchar4* v_in, uchar4* v_out, const void* usrData, uint32_t x, uint32_t y) function is invoked for each pixel (known as an element within this function) and on each available core. (Each element is processed independently of the others.) It declares the following parameters:
v_incontains the address of the current input allocation element being processed.v_outcontains the address of the equivalent output allocation element.usrDatacontains the address of additional data that can be passed to this function. No additional data is required for embossing.xandyidentify the zero-based location of the element passed tov_in. For a 1D allocation, no argument would be passed toy. For a bitmap, arguments are passed to both parameters.
The root() function implements Listing 1’s embossing algorithm for the current input element, which is a 32-bit value denoting the element’s red, green, blue, and alpha components. This element is unpacked into a four-element vector of floating-point values (each between 0.0 and 1.0) via Renderscript’s float4 rsUnpackColor8888(uchar4 c) function.
Next, Renderscript’s const void* rsGetElementAt(rs_allocation, uint32_t x, uint32_t y) function is called to return the element at the location passed to x and y. Specifically, the element to the northwest of the current element is obtained, and then unpacked into a four-element vector of floating-point values.
Renderscript provides fabs(), fmax(), and fmin() functions that are equivalent to their Listing 1 algorithm counterparts. These functions are called in subsequent logic. Ultimately, Renderscript’s uchar4 rsPackColorTo8888(float r, float g, float b, float a) function is called to pack the embossed element components into a 32-bit value that is assigned to*v_out.
Note: For a complete list of Renderscript functions, check out the online documentation or the .rsh files in your Android SDK installation.
The script concludes with the void filter() function. The name is arbitrary and corresponds to the suffix attached to the ScriptC_-prefixed class’s invoke_-prefixed method. This function performs any needed initialization and executes the root() function on multiple cores, by calling one of Renderscript’s overloaded rsForEach() functions.
In lieu of initialization, this function invokes one of Renderscript’s overloaded rsDebug() functions, which is useful for outputting information to the device log, to be accessed by invoking adb logcat at the command line, or from within Eclipse. In this case, rsDebug() is used to output the value of the RS_VERSION constant.
filter() uses an #if–#else–#endif construct to choose an appropriate rsForEach() function call to compile based on whether the script is being compiled in the context of the Android SDK or Eclipse. RS_VERSION can contain a different value in each of these compiling contexts, and this value determines which overloaded rsForEach() functions are legal to call, as follows:
00134 #if !defined(RS_VERSION) || (RS_VERSION < 14)
00135 extern void __attribute__((overloadable))
00136 rsForEach(rs_script script, rs_allocation input,
00137 rs_allocation output, const void * usrData,
00138 const rs_script_call_t *sc);
00142 extern void __attribute__((overloadable))
00143 rsForEach(rs_script script, rs_allocation input,
00144 rs_allocation output, const void * usrData);
00145 #else
00146
00165 extern void __attribute__((overloadable))
00166 rsForEach(rs_script script, rs_allocation input, rs_allocation output,
00167 const void * usrData, size_t usrDataLen, const rs_script_call_t *);
00171 extern void __attribute__((overloadable))
00172 rsForEach(rs_script script, rs_allocation input, rs_allocation output,
00173 const void * usrData, size_t usrDataLen);
00177 extern void __attribute__((overloadable))
00178 rsForEach(rs_script script, rs_allocation input, rs_allocation output);
00179 #endifThis code fragment is an excerpt from Renderscript’s rs_core.rsh header file as viewed online. The filter() function invokes the appropriate rsForEach() function with the minimal number of arguments based on the current value of RS_VERSION, as revealed by this excerpt.
Building and Running EmbossImage
Let’s build and run EmbossImage. For brevity, this section only shows you how to build and install this app’s APK in a command-line environment. Although it doesn’t also show the Eclipse equivalent, it isn’t difficult to extrapolate the instructions for the Eclipse environment.
This app was developed in the following context:
- Windows 7 is the development platform. Change backslashes into forward slashes and remove the
C:drive designator for a Linux environment. - The project build directory is
C:prjdev. Replace theC:prjdevoccurrence with your preferred equivalent. - Android SDK Revision 20 has been installed and an Android 4.1-based AVD associated with target ID 1 has been created. Execute
android list targetsto identify your equivalent target ID.
Complete the following steps to create and build an EmbossImage project:
- Create the
EmbossImageproject by executingandroid create project -t 1 -p C:prjdevEmbossImage -a EmbossImage -k ca.tutortutor.embossimage. - Replace the skeletal contents of
EmbossImagesrccatutortutorembossimageEmbossImage.javawith Listing 2. - Introduce an
emboss.rsfile with Listing 3’s contents into thesrcdirectory. - Create a
drawablesubdirectory ofEmbossImageres. Copy theleopard.jpgfile that’s included in this article’s code archive to this directory. - With
EmbossImageas the current directory, executeant debugto build the app.
The final step may fail with the following warning message (spread across multiple lines for readability):
WARNING: RenderScript include directory
'C:prjdevGrayScale${android.renderscript.include.path}'
does not exist!
[llvm-rs-cc.exe] :2:10: fatal: 'rs_core.rsh' file not foundFortunately, Issue 34569 in Google’s Android issues database provides a remedy. Simply add the following property (spread across multiple lines for readability) to the build.xml file that’s located in the toolsant subdirectory of your Android SDK home directory to fix the problem:
<property name="android.renderscript.include.path"
location="${android.platform.tools.dir}/renderscript/include:
${android.platform.tools.dir}/renderscript/clang-include"/>Specifically, place this <property> element after the following <path> element:
<!-- Renderscript include Path -->
<path id="android.renderscript.include.path">
<pathelement location="${android.platform.tools.dir}/renderscript/include" />
<pathelement location="${android.platform.tools.dir}/renderscript/clang-include" />
</path>Rexecute ant debug and the build should succeed.
If successful, you should discover an EmbossImage-debug.apk file in the EmbossImagebin directory. You can install this APK onto your emulated or real device by executing adb install binEmbossImage-debug.apk.
If installation succeeds, go to the app launcher screen and locate the generic icon for the EmbossImage APK. Click this icon and the app should launch. Figure 2 shows you what the initial screen should look like on the Android emulator with an Android 4.1 AVD.
Figure 2: The activity displays an unembossed leopard at startup.
Click the leopard image (which is courtesy of Vera Kratochvil at PublicDomainPictures.net), and you should observe the embossed equivalent that’s shown in Figure 3.
Figure 3: Clicking the activity causes an embossed leopard to emerge.
Conclusion
Embossing and other image processing operations are fairly easy to write in Renderscript. As an exercise, enhance the embossing script to support the light source coming from a direction other than the northwest (such as the northeast). This technique can be invaluable for using color, contrast, and the inherent “elevation” of pixels to build intuitive interfaces and enable advanced image manipulation.
