Crouching Javascript, Hidden PHP [1]

Tweet

Jason has been pondering the options of getting Javascript talking to PHP or, put another way, how to get a web page to load data without reloading (or at least without appearing to reload).

Why do this? Because you can give the impression of a rich client (a desktop GUI) vs. a typical web-based thin client where simple user interaction often requires extensive page reloads and waiting around.

By way of example, consider a simple web-based administration app where the administrator has a list of users and can edit each user in the list. How can we avoid reloading the list of users between each edit to a single user?

At first glance it’s a simple problem to solve – or perhaps it should be a simple problem to solve. Look a little further and you discover it’s one of those grey areas of web development that leaves you thinking “It’s 2004. Someone must have solved this once-and-for-all by now.”

Anyway, this is a long and not-particularly-focused rant which also follows on from Seperating Browser from Resource and Serializing PHP data structures for Javascript.

Feel free to flame / correct. The mission is to try to define the “problem” as fully as possible. Perhaps this needs to be transplanted to a WIKI in due course.

Mechanism / Payload / Population

Think there’s three main technical areas to consider when thinking about how to get Javascript talking to PHP;

- the mechanism you use to transport information between Javascript and a remote server

- the payload you actually exchange over that mechanism

- how you populate the user interface with the payload you’ve received

Note deliberately I’ve avoided words like “transport” and “protocol” because I think they cloud the issue, particularly when dealing with stuff like XML-RPC which amounts to a protocol sitting on a protocol.

Mechanism

First a quick word on terminology;

Synchronous vs. Asynchronous Calls
In the context of this subject, when you call some function in Javascript, which instructs the browser to fetch something from from a remote server, does code which executed the call sit and wait for the function to return (a synchronous call) or can execution continue, allowing the function to execute in the background (an asynchronous call)?

From the point of view of an end user, when they click on that “Get Data” button, does the user interface freeze until the data fetching operation is complete (which will be the case with a synchronous call) or can they continue messing around while the browser is busy fetching stuff from the remote server (asynchronous)?

Side note – related to this in the notion of blocking or non-blocking sockets. See also stream_set_blocking() in PHP.

Available Mechanisms
Considering the options you have to get Javascript talking to a remote server, realistically (i.e. ignore stuff like Mozilla’s SOAP implementation) it seems to boil down to one of two choices;

- Tricks with HTML src attributes

Given a tag with which refers to an external resource and can return something parse-able, you can vary the src attribute using Javascript, allowing the main page to remain resident in the browser.

The most common way to do this is with a frame or iframe – see Remote Scripting with IFRAME (as the first obvious link from Google).

Another approach is varying the src attribute of a script tag, which is illustrated nicely by Javascript Serializer, a library to get PHP and Javascript talking, which implies loading executable Javascript from the remote server.

In general I guess this is probably the sanest mechanism, if you want something you can make work quickly, with minimum “risk”. It is fundamentally “clunky” though, IMO. You have little means to identify and handle errors such as the remote server being down at the time the request was made plus you’ve got to embed stuff in your HTML which really shouldn’t be there (and is just waiting from someone to break it while modifying the page). You’re also forced into asynchronous calls which isn’t a disaster but sometimes you want your app to make the user wait until the response to an action is complete.

- XMLHttpRequest

XMLHttpRequest, available with more-or-less similar APIs in recent versions of Mozilla / Firefox, IE and Safari gives you a Javascript object you can treat as an HTTP client.

XMLHttpRequest gives you a lot of control, allowing your code to be aware of server responses (like 404) for error handling as well as providing synchronous and asynchronous modes e.g. (Mozilla example);


var XmlHttp = new XMLHttpRequest();

// Async - execution continues after this
XmlHttp.open('GET', 'http://localhost/slowscript.php', true);

// Sync - execution waits until HTTP request is complete
XmlHttp.open('GET', 'http://localhost/slowscript.php', false);

There’s no need to embed dubious tags in your HTML

The obvious point against XMLHttpRequest is the browsers which don’t support it but, as in Jason’s case, the type of application where you want rich-client behaviour is often deployed to a known user group (e.g. an Intranet app) where you can reasonably demand they get with the program. Otherwise documentation and experience with XMLHttpRequest is still relatively thin on the ground (but getting better) – if you’re not confident in Javascript that may be a problem.

Payload

The next question is what do you actually exchange? Much of the discussion has been thrashed out with SOAP and the ideas of RPC vs. REST and encoded vs. literal. Avoiding long discussions of semantics, the choices involved looked something like this;

RPC vs. REST

- RPC (Remote Procedure Calls)

Suits programmers; instead of using a some library locally, you know a network address for that library and call it remotely.

For the web, “network address”, typically means a single URL, data exchanged using the HTTP POST method.

To see web based RPC in it’s rawest sense (avoiding clouding the issue with XML), take a look at MIME-RPC, particularly the examples ( head to Kev’s Advanced Email in PHP tutorial as a starting point for understanding MIME plus scan over Methods GET and POST in HTML forms – what’s the difference? if you’re not sure ).

From a PHP point of view you might have a URL like;

http://www.site.com/rpc-server.php

In Javascript you’d POST stuff to this URL (either using the form hidden in a frame like Jason suggests or with XMLHttpRequest).

The simplest view of the problems with web-based RPC are “can you interact with the server using just your browser?” (easy to debug?). The implication is the “src” based mechanism above won’t work as it’s restricted purely to GET based requests.

- REST

The “REST way” (see here for a good summary), in essence, gives preference to the HTTP GET method.

Put another way, it should be possible to surf your remote server with a browser e.g. http://www.tekool.net/php/js_serializer/exemple_calculator.php?1,+,1 – you can modify the URL manually.

REST suits the src attribute based approach nicely and is generally easy to debug. If your server generates something like XML, it’s also easy to re-use for different types of operations (e.g. throw in some XSLT).

The problem (or the place where smart decisions need to be made) is how you map a function call (plus it’s arguments) in Javascript to a given URL. And what about sending a complex (or large) data structure, like a multidimensional array, to the server (you probably need POST at this point).

Encoded vs. Literal

The next question is what form should the data being exchanged actually take? And will a request contain data in the same form as the response?

- Encoded

Encoding the data means representing native variables (in Javascript or PHP) in some intermediate form which both the client and server can understand. XML-RPC is a well known approach to encoding both client requests and server responses in an XML format that describes the data types directly e.g. the server response;





12
Egypt
0
-31

…might correspond to the following in PHP;


$response = array (
12,
"Egypt",
false,
-31,
);

…and in Javascript;


var response = [
12,
"Egypt",
false,
-31
];

XML-RPC also pretty much decides that you’re going to use RPC (vs. REST) but still represents one of the “sanest” approaches, given numerous and mature implementations in PHP and Javascript (see list below).

There are alternative ways to encode the data, including WDDX and SOAP encoding. WDDX is probably the next-sanest way to go, after XML-RPC, when considering PHP and Javascript ( PHP WDDX extension and a Javascript implementation).

You might also encode a request in a different manner to the response, perhaps using URL encoding (variables in the GET query string) for the request while even generating Javascript for the response.

Also, a nice trick for PHP (I think), for dealing with more complex data types in a client request, is to serialize in Javascript variables into a string ready for PHP’s unserialize() function (see growing user contributed notes plus my attempt here).

Meanwhile the response could contain literal Javascript ready for eval() – my attempt at a code generator for this is here while Javascript Serializer has a more straightforward implementation.

The main problem with encoding is you can get bogged down, both in terms of development and performance, in translating between native data and encoded data.

- Literal

The literal approach means moving information around without the information itself containing a description of how it translates to native language data types. You might have something extra, like an XML schema document, which describes how the document should be “translated” but it’s up to you whether you use it.

Perhaps the most well known form of this is Amazon’s XML service. If consider a URL like this, it contains information as to what you should translate the document into in your chosen language / platform. That said it’s pretty clear that this;



php

Books

0321186486
Ullman, Larry

067232525X
Welling, Luke

Would look something like this in PHP;


$catalog = array (
'keyword' => 'php',
'product_group' => 'books',
'products' => array (
array (
'title'=>'PHP and MySQL for Dynamic Web Sites',
'asin'=>'0321186486',
'author'=>'Ullman, Larry',
),
array (
'title'=>'PHP and MySQL Web Development',
'asin'=>'067232525X',
'author'=>'Welling, Luke',
),
),
);

This fits nicely with something like PEAR::XML_Serializer, as I’ve covered before here. Unfortunately have yet to find a ready-to-roll equivalent to XML_Serializer in Javascript (although it wouldn’t be too hard to write one). Do we want to the overhead of parsing XML?

Another, very literal approach is what Christian Stocker has done with LiveSearch, the output being a chunk of HTML you can insert directly into a page. So far no-one’s asked questions like “where’s the doctype?” ;)

One general problem (or key decision point) is how client requests should be sent – how does a client request all the books from three different categories, for example – should requests be encoded while responses are literal?

Also, especially if the server side developers work independently (remotely) from those building clients, how do they agree on a common format? And is it easy to parse? All the joys of screen shaping…

Population

Once we’ve reached the point of having received something back from the server, whatever it is, the next question is how to “blend” it with the user interface?

The choices seems to be between manipulation with DOM, XSLT or somewhat dodgy (but probably easier to hack) visual tricks with frames.

GMail seems to have gone for the later, using frames (I don’t have an account so I’m guessing from what Mark Pilgrim says here). In other words there’s no effort client side at all – the response is delivered in ready-to-view form.

XSLT could be a slicker way to do things. Then again…

Or using DOM you can manipulate anything you like… at a price – you need to write the code which populates that table with the result set you just received from the server. This can get pretty painful and may take you into the realm of templating (already a disaster area on the server side). And life gets even more interesting when you want to do something like update only a single row in that list of users, after editing a single user.

The Way to Go?

No idea ;)

Probably the safest is what GMail is doing – let’s not be clever. That said, reckon it can lead to it’s own development nightmares. More on that another time.

Rant Over

There’s a second part to this focused more on practical development issues e.g. should building an XML-RPC server in PHP look like this;


include 'xmlrpc.inc';
include 'xmlrpcs.inc';

function sumAndDifference ($params) {

// Parse our parameters.
$xval = $params->getParam(0);
$x = $xval->scalarval();
$yval = $params->getParam(1);
$y = $yval->scalarval();

// Build our response.
$struct = array(
'sum' => new xmlrpcval($x + $y, 'int'),
'difference' => new xmlrpcval($x - $y, 'int')
);
$return_val = & new xmlrpcval($struct, 'struct');

return new xmlrpcresp($return_val);
}

new xmlrpc_server(
array(
'sample.sumAndDifference' => array(
'function' => 'sumAndDifference'
)
)
);
?>

or like this;


include('IXR_Library.inc.php');

function sumAndDifference ($params) {
$x = $args[0];
$y = $args[1];

return array(
'sum' => $x + $y,
'difference' => $x - $y,
);
}

$server = new IXR_Server(
array(
'sample.sumAndDifference' => 'sumAndDifference',
),
);

>>

Another point – how easy is it to break the client-side when the server-side changes? What can be done with generated Javascript?

XML-RPC in PHP and Javascript

A choice few;

- PHP

+ XML-RPC – the ever-experimental extension. Nice if you’ve got it compiled on your server although painful to work with when in comes to introspection and awkward to use with classes defined in PHP.

+ Incutio XML-RPC – excellent library from Simon. Simon’s given up on it sadly. Meanwhile I’m slowly hacking a (now somewhat bloated) fork here. Note: I highly recommend pulling Incutio XML-RPC apart and seeing how it works.

+ PEAR::XML-RPC – traces it’s roots back to the original usefulinc library. Personally not a big fan (e.g. count the globals). If you use it, make sure you find out what the XML_RPC_encode() function does.

+ Keith’s XML-RPC implementation – it doesn’t get any more straightforward.

- Javascript

+ JavaScript o lait – primed for use with XMLHttpRequest. Haven’t tried it but code looks good and it’s what makes this possible. Getting pretty big but also actively developed.

+ Scott Andrew’s XML-RPC message builder – gives you XML-RPC messages – up to you what mechanism you use. Lightweight, which is nice.

+ vcXMLRPC – pretty complete implementation although development now dead.

+ nsXmlRpcClient – Mozilla only over XMLHttpRequest – watch out for the permissions minefield (see here).

Free book: Jump Start HTML5 Basics

Grab a free copy of one our latest ebooks! Packed with hints and tips on HTML5's most powerful new features.

  • msyKYLE

    Smiley
    The solution I proposed would not tie anyone to PHP, as you could use whatever you want to generate the XML (or static XML files)

    The Smarty format would be a simpler XSLT, written in a format that most PHP developers (which is what this group is) are familiar with. XSLT would be ideal.

  • arborint

    I have to agree with Alan. Having dealt with this implementing SCORM using javascript and several server side languages, I can say that if you want to keep any kind of continuous data conversation going then stick with POST/GET strings. XML from my experience just adds an extra layer of error checking: 1st check XML for errors, 2nd check you data for errors. The overhead of XML is just not worth it right now when trying to keep up a back and forth data exchange.

    Maybe on intranets where you can prescribe the client and know the perfomance you can get away with more. But, today on the internet with a 4.0 minimum browser spec and unknown connection, XML and the various protocols only work reliably for submitting and retrieving a chunk at a time.

  • Pierre Charbonneau

    I still run for my life the second that somebody mentions Java script or any idea of a client side equivalent. The trouble is the client side execution. You just don’t know what will happen. You can control your server side output pretty well.

    Until browsers get their act together, I shall stick with good old html and won’t dream about anything fancy.

    I have a key phrase that goes: “If it doesn’t work in Mozilla, try it in IE”.

  • http://maetl.coretxt.net.nz/ maetl

    Kind of a side issue, but I’m just wondering about unit testing JavaScript. It’s now well taken care of in PHP, and I have had thoughts about extending SimpleTest with a ‘JSReporter’, but so far I’ve never really found any good resources for actually testing or debugging javascript…

  • http://www.phppatterns.com HarryF

    When javascript (especially if you are risking life and limb trying to get a MS implementation to work) comes in, it’s alot easier to stick with the basics.. KISS / POST/GET/ strings and XML as needed..

    Very true!

    Kind of a side issue, but I’m just wondering about unit testing JavaScript.

    The best framework I’ve found so far was JsUnit. It pretty much got it all, except Mock Objects. Integrating with SimpleTest would be nice but don’t think there’s much value until PHP has a Javascript interpreter (e.g. Spidermonkey) and even then, would be some gaps to fill.

  • http://sproutworks.com sproutworks

    I built a very simple chat system using XMLHTTPRequest. http://sproutworks.com/chat.php

    All it does is gives the server the latest message id it has displayed, and if the server reponds with a greater id, it makes another request for all new messages.

    http://sproutworks.com/chat.js

  • marcoax

    I’ve used the XMLHTTPRequest in this online form
    http://www.smiwrap.it/modulo.php?IdPage=Offerta&Id=&lang=it to fill some combo boxes after the user has select a Paper Type. It is very fast.

  • lively

    sproutworks: that chat system doesn’t appear to work in Opera 7.54 or FireFox 0.9. I hope that whenever anyone implements these systems that they always provide fully gracefully degradable non-javascript server-side code …

  • isao

    One more option: Flash Remoting. It is Javascript-ish, and implemented more or less identically across thick client browsers. There are PHP layers that abstract the Flash client communication. That said, I’ve only dabbled in it so far…

  • Anonymous

    De plus, Communication-Qu

  • Anonymous

    I have used a system in the past with javascript/images/cookies/php.

    First I create a image object in javascript and setting src property to a php script. The php script changes the content type as a jpeg, set’s some cookies and then outputs some jpeg binary (only small as it’s not used). Back in javascript the cookie is read and used accordingly. Information can be passed to the php script via the query string.

  • mike smiff

    Here is a wddx implementation…

    http://www.prescientsoftware.com/JPX_WDDX/

  • Rush

    Can someone plesase tell me how can I pick up an xml file and post it to a php webpage? man, i’ve been researching for days and nothing concrete!

  • Alan Knowles

    I can tell you having worked with SOAP, various RPC layers, and simple GET/POST / XML parses, that most of the time RPC tends to suck big time, It’s great for the first 3 test calls, but as a project grows, there is a point where it doesnt work as expected. You then enter one of the worst nightmares in hell to debug.

    When javascript (especially if you are risking life and limb trying to get a MS implementation to work) comes in, it’s alot easier to stick with the basics.. KISS / POST/GET/ strings and XML as needed..

  • http://blog.casey-sweat.us/ sweatje

    Man, when you say you are fired up to respond, you mean it ;)

    Nice summary of the evaluation criteria for the problem. I had not quite formalized my thinking, but was certainly poking around all of those areas.

    Now I will just wait for someone to read you post and respond with the “magic bullet” ;)

  • dotvoid

    There are simpler approaches to this as the more complex solution might be overkill. I have described an alternative method at http://www.dotvoid.com/view.php?id=13

  • Jan Korbel

    I tried messing with the XMLHttpRequest and it looks promising. For anyone interested here a couple links I bookmarked:
    http://jibbering.com/2002/4/httprequest.html
    http://developer.apple.com/internet/webcontent/xmlhttpreq.html
    http://blog.bitflux.ch/p1735.html

  • msyKYLE

    I was just thinking about this yesterday. Here’s what I would do: I would implement client-side Smarty templates.

    Here’s how it works: On page load, the controller page uses xmlHTTP to grab a list of templates, and caches them. Then it displays the default page. Pages are a combination of a template, and a PHP file that returns XML, which is parsed into a variable structure for the Smarty template.

    So perhaps this link:
    [PHP]
    Go
    [/PHP]
    loads data.php?id=2 with xmlHTTP, which returns the XML:
    [PHP]

    Blah, blah.

    Apple
    Bear


    [/PHP]
    So the controller digs up template1, which looks like:
    [PHP]

    {$title}

    {$contents}

    {$arrayexample.a} | {$arrayexample.b}



    [/PHP]
    parses it, fills in the variables, and displays it.

    Issues:
    * It’d be a B$%#$ to port Smarty to Javascript.
    * How much client resources would this take up, between initial downloads, parsing and rendering templates?

    Oh well, whatdya think?

  • http://www.realityedge.com.au mrsmiley

    I’m currently playing with a library called Sarissa which is a cross browser implementation for XMLHttpRequest plus some other useful features that either Moz or IE lack that the other one has.

    Anyway the basics of this project allow me to generate server side XML that I can apply XSLT to to generate either client side HTML for the end user, or client side HTML for configuring the component that I’m generating for admin type users.

    By using XSLT it should be a snap to apply skins/themese to the site without changing the backend or client side output. And as the XSLT is generating valid XHTML, it could even be changed to output for say WAP and hand held devices. Using Sarissa, all I do it load the remote document/component, and then check to see if its an error, if its not, then I show the component straight to the screen.

    So long as the client has Javascript enabled and is mostly DOM compliant, I’m really anticipating any big problems.

    So long as you know what your target clients are, the choices are not really that hard. Just pick the solution you best understand and find easiest to work with and maintain. Its when you start designing thin, rich clients for any abitrary device that things start getting tricky. From past research, XSLT seems to provide most of the answer to this though.

    On the topic of client side smarty templates, how do you propose to deal with interop between different backends? Rich clients do not necessarily pull all their information from the one backend source, so tying yourself to PHP and Smarty would be very restrictive.

  • Treehouse Cityguide Maintainer
  • sdfsjfhsjkh

    KHJHFK JKFSHJKHFKJFSDFSFSFDSF

  • sskmscss25@yahoo.co.in

    how to retrieve javascript values in php script in the same file?

    Like……

    a=new Date();document.cookie=a.getTime();alert(document.cookie);var b=a.getTime();”;
    echo “”;
    $res=……….//here i have to retrieve getTime value
    ?>

  • Goshe

    Like Rush, I am in a similar quest. Yes, there is definitely more than one way to send XML back to client that is previously set up some sort of parser (as in the ajax case, via JavaScript and XMLHttp), but what if the client sends a request back to the server-side script in the form of an XML? How to get to the sent file in the PHP script? Directly assigning a variable to the received file?
    Like:

  • Adam Kennedy

    There are all manner of different frameworks, and using XMLHttpRequest can be… well… interesting.

    But when it comes to sane, standalone, you absolutely cannot go past the JSAN (http://openjsan.org/) module HTTP.Request

    It just does the request, but it’s had the heck thrashed out of it on every browser and plain Just Works.

  • Renmiri

    Ajax does this, right ?

  • http://www.edu11622358495043.net.in/ Sofia

    How old are you?

  • http://www.ind.in1162307077811.ind.in/ Gustavo

    Howdy Folks. I was wondering if anyone had any details on the Eastern re-union that is being held in Moncton in 2007. Cheers, Dan

  • webmaster

    HCL ISD is known as the pioneer of Offshore Remote Infrastructure Management market in India and is today the leader in this field. The National Stock Exchange (NSE), our first IMS deal, involves managing the IT Infrastructure for Asia’s largest stock exchange, has instilled a strong DNA of “Real Time Operations” in HCL ISD’s ethos