Programming
Article

WS-DeadParrot

By Harry Fuecks

It’s rant time again (apologies – move along unless you’re feeling fiesty). It’s that REST vs. SOAP thing. There’s another round of debate traversing various blogs. This time it seems to be SOAP’s final death throes. If SOAP and all that makes little sense, know that you’re not alone and that you can happily ignore it and focus on smarter options.

Anyway, David summarizes from a distance while Tim Bray nailed the detail nicely a few weeks back in WS-Angst, which began with this post (which boils down to: “show me your data structures” vs. “show me your apis“) and finally ends up in this;

If you have Microsoft saying “well, the best approach is to make this elaborate infrastructure we’ve spent billions of dollars building out optional”, then the debate is over.

And make no mistake – it’s truly over. Forget whatever bizarre use cases WS-* support; when the real world starts doing this it’s time to move on. You want message routing? Try this…


< ?php
$in = 'http://api.search.yahoo.com/WebSearchService'. 
    '/V1/webSearch?appid=YahooDemo&results=10&output=php'. 
    '&query=SOAP';
    
$data = unserialize(file_get_contents($in));

file_put_contents('/dev/null',serialize($data));
?>

WS-Headless

On the one hand this is all forgivable human failing – too many architects spoil the WS-Broth. Design by committee means supporting strange edge cases and unnecessary complexity. Let’s move along to REST at last.

On the other, how about some accountability? Heads need to roll. This wasn’t just an amusing intellectual exercise to help boost flagging egos and payrolls – this was buzzed at high level and now real people are taking the flak when WS-projects fail, despite being dictated the choices leading to failure. The WS-Bandwagon is still in full steam even though the drivers have jumped clear.

At the very least, we need to prevent anything like this from happening again. Some ideas;

  • If you employ architects, make them spend 4 out of 5 days writing code that someone actually has a use for. Programmers away from the keyboard get weird ideas – reality is a great reminder.
  • If you’ve got a Great New Architecture, build a working prototype in Perl, before discussing it with anyone. You may use CPAN, which gives you a pretty good chance (call it learning from Roy).
  • In describing your Great New Architecture, if you need to invent new words and acroynms, return to your keyboard.
  • If you can’t clearly describe your Great New Architecture in under 60 seconds to a normal person, return to your keyboard. If you made them reach for a dictionary while doing so, please contribute 4 weeks voluntary service to a good cause.
  • Do not, ever, form a committee.
  • Do not invent use cases. If no one is doing it today, they won’t be doing it tomorrow either.
  • Have at least two tales (from real experience) of how easy configuration, troubleshooting and debugging is. Better yet if you can produce the sysadmins you inflicted it on.

What remains now is to bring REST out of the WS-* debate and back down to earth, so we can solve the real problems of people using HTTP, like getting authentication working.

We also need to take note of what those doing real work are saying;

The best advice is just basically to keep everything as simple as possible—simple processes, simple SKUs, simple engineering. These systems get to be very big very fast. I don’t think there’s really any one particularly hard, gnarly problem, but when you add them all up, there are lots and lots of little problems. As long as you can keep each of those pieces simple, that seems to be the key. It’s more of a philosophy, I think, than anything else.

  • http://www.lopsica.com BerislavLopac

    Of course, we should not forget JSON.

  • stepz

    JSON vs. serialize($data) is completely beside the point. Different flavours of the same thing. JSON’s pretty much only advantage is that you don’t have to deploy the parser to the browsers, it’s already there.

  • http://www.lopsica.com BerislavLopac

    But what if we try to connect different applications? What good it does us to have a PHP-specific serialization if we expect people to connect using, say, ASP.NET or Java?

    SOAP’s prime advantage is that it’s a standard, even an overblown one, and all platforms have some kind of client/server implementation of it. This is also true ror XML-RPC, while REST defines how to call for a service and not how to encode the transfered data.

  • http://www.phppatterns.com HarryF

    But what if we try to connect different applications? What good it does us to have a PHP-specific serialization if we expect people to connect using, say, ASP.NET or Java?

    Yahoo have a number of different serializations. If you want PHP developers to use your service though, why not provide something specific for that group, given that it costs little to do so.

    The same effectively goes for AJAX / JSON – because XMLHttpRequest is restricted to a single domain and we know for sure that we’ll always be using Javascript on the client side, you can at least use and encoding that’s specific to Javascript or even go further, given the server-side for a single application is also a known, and use an encoding that’s specific to both client and server side. There’s some interesting discussion of alternatives here.

    What’s being learnt is the cost to use “X” encoding in dynamic languages is very low – inherent flexibility means supporting new encodings can be done with relatively little code. Less code means easier debugging etc.

    SOAP’s prime advantage is that it’s a standard, even an overblown one, and all platforms have some kind of client/server implementation of it. This is also true ror XML-RPC, while REST defines how to call for a service and not how to encode the transfered data.

    Well on the standard front, these days if you talk to anyone experienced with SOAP, they’re saying interop between different platforms basically hasn’t happened. In theory it can be done but in practice it needs someone with deep knowledge of SOAP specs and the ability to read WSDL by eye. The “common wisdom” has been reduced to J2EE and .NET SOAP don’t mix – use all one or all the other.

    On the encoding front, perhaps that is one useful part of SOAP – the encoded data types. Except that it’s geared for strongly typed languages, given the backdrop of xml schema. XML-RPC does at least as good a job of encoding, given that’s it’s fairly easy to implement but probably needs a specific “NULL” data type to help remove doubt.

    All that said though, given tools like simplexml, SDO, XML_Serializer, XML::Simple and E4X with help from stuff like Data::Dumper or var_dump, it costs little extra to work with ad-hoc XML structures.

    Meanwhile, when you need to make the structure a little more rigid, Relax NG is a great deal easier to support and use than XML Schema.

    Really this whole issue needs to put dynamic languages as the first class consumers and providers, and leave static languages as the second class citizens (which is the reverse of what happened with SOAP). Doing so would result in simple specs (if any) and a focus on stuff that works.

  • stepz

    I’m following the ongoing REST vs SOAP discussion with great interest as I’m currently working on revamping my employers internal between-services communications. I have some (successful) experience with SOAP and WSDL using it for communications between internal systems and between external systems. (as a freelance job I actually even did a SOAP webservice implementation in database stored procedures using perl for one client with a peculiar vision for their architecture :) ) I was going to go with SOAP+WSDL but nothing is yet set in stone.

    So a brief overview of the requirements that I’ll have to satisfy:
    * Current system interconnects are anything from NFS transfered xml files to somewhat RESTy HTTP GET access to an imagedatabase to more and less strictly defined SOAP interfaces. Reuse is pretty limited and won’t be one of the top concerns.
    * Software components are for now written PHP and Python, but I’d hesitate to shut out Java or .NET forever.
    * All exchanged messages need to carry transaction ID’s, logging metadata, authentication info and some other similar metadata.
    * Almost all messages are required to be logged (by law actually). Logging is to be taken care of by the communicating peers themselves.
    * I’d like the ability to automatically validate the exchanged messages for correctnes.
    * Some services, about 1/3 or so, will need to be exported to the outside with SOAP+WSDL using a bit different set of metadata than internal use requires.
    * The data will have XMLSchema definitions probably anyway for other architectual reasons. Atleast base datatypes that are combined into messages in realtively straightforward ways will anyway be defined in XMLSchema.
    * A usable API for RPC is required, together with mapping to native datatypes. Once the messages are defined, I don’t want to know anything about wire encoding, wire protocol or anything. I just want a RPC with all the behind the scenes including logging, routing, etc. taken care for me.
    * I don’t have to fill any buzzword quotas. I don’t care if the architecture is so sky high flexible that it can be defined to do anything or if the architecture has the pinnacle of form and function. I care if it does the things that I need, with me doing as little work as possible.

    So all said and done, even considering all the talk about demise of SOAP, I’d still go with SOAP+WSDL for now. The reasoning from the PHP side (haven’t had the time to thoroughly study relevant Python libraries):

    SOAP has ext/soap on PHP side, that makes RPC braindead simple. I don’t know of anything that elegant for REST.

    SOAP has conventions for transfer of faults, language independent data structures and metadata. Atleast ext/soap sorts all these out for me. I’d expect similar treatment from the Python side of the fence. Using rest I’d have to define my own mapping between PC and wire format and implement them on both platforms. If I had to interface with anybody else, they probably have their own, different, mapping.

    SOAP has atleast some tool support, REST has pretty much none if you don’t count the fact that in simpler cases the format is simple enough to use by hand.

    So I’ll stay with SOAP, for now.

    I do agree on the fact that the WS-* crew should try more to keep their feet on the ground though. If you look at SOAP+WSDL then you get the feeling that they have tried to define all possible solutions to all possible communication problems and then left it to the implementators to sort out which solutions they have time to implement. They should have started by defining something as simple as XML-RPC and welldefined enough that it can actually be consistently implemented and expanded from there. But atleast they have tried to define a mapping from wireformat to RPC, that’s something that the REST crew doesn’t seem to have yet.

    Anyway, if anyone can point out any pragmatic reasons, why should I use REST, I’d be happy to hear them.

  • jayboots

    I still can’t believe how slow YAML uptake has been by the PHP community.

  • http://www.dvdverdict.com/ mjackson42

    I still can’t believe how slow YAML uptake has been by the PHP community.

    Oh dear god, that’s awful. If I didn’t read the yaml.org About page, I’d swear that was invented by a Python programmer. Levels of indentation to denote structure? I can’t imagine working with that in a productive manner.

  • http://www.phppatterns.com HarryF

    Anyway, if anyone can point out any pragmatic reasons, why should I use REST, I’d be happy to hear them.

    Think the main pragmatic reasons are general: stuff like performance / scaling given that HTTP comes with a bunch of stuff to help, easier to debug (“just” HTTP rather than HTTP + extra layers / extra libraries), simple APIs (urls, GET, POST etc.) and simple libraries.

    But none of those are particularily convincing. Also looking at your requirements, seems like there’s experience with SOAP already there which means you’re already able to visualize this project from start to end. REST has a bunch of grey areas where best practice seems to be missing, so it’s entering a “riskier” zone.

    But some general thoughts against your list, which are somewhat like “you don’t want to start from here”.

    * Current system interconnects are anything from NFS transfered xml files to somewhat RESTy HTTP GET access to an imagedatabase to more and less strictly defined SOAP interfaces. Reuse is pretty limited and won’t be one of the top concerns.

    WebDAV might be interesting as an alternative to NFS. If you haven’t tried it, see HTTP_WebDav_Server and try a webdrive evulation as client.

    * Software components are for now written PHP and Python, but I’d hesitate to shut out Java or .NET forever.

    If you’re using XML, in general, Java and .NET have a way in. Of course it depends on exactly how you’re using XML but…

    * All exchanged messages need to carry transaction ID’s, logging metadata, authentication info and some other similar metadata.
    * Almost all messages are required to be logged (by law actually). Logging is to be taken care of by the communicating peers themselves.

    I’m not sure SOAP gives you any particular advantage there – if this information is getting packed into messages, it almost makes more sense to do the work of designing an XML format carefully (vs. say allowing encoded SOAP to take care of it for you). Perhaps there’s some WS-* specs that could help out but seems the more you use of the WS-*, the less chance you have to broad support on many platforms.

    Regarding logging, it sounds like you’re really wanting to store copies of documents (vs. encoded data structures) – there’s SOAP document literal but I think you’ll be headed into trouble there with PHP and Python. Thinking instead of exchange of “simple” XML documents and files makes this a little easier, plus you’ve get access to URLs logged by Apache (or otherwise) which may be nice to have (vs. all access tunnelled through POST to one URL).

    Should also point out that HTTP already comes with fields for some basic meta data (which may or may not be useful), such as date/time stamps, file types. You’d also be able to apply mod_security easily to a REST based service.

    * I’d like the ability to automatically validate the exchanged messages for correctnes.

    That’s doable again, without SOAP. XML Schema or Relax NG if you want to do this against XML. That said, using Python or PHP it’s in some ways easier to do it in code (that’s if validation will only be happening locally). That advantage of XML schema there is (sadly almost), it’s well known so if you publish it to someone else, there’s a better chance they’ll understand it. Validating that a string is between 10 and 15 characters though, effectively nullifies the advantage of encoded SOAP and means you have to make the effort of designing an XML format.

    * Some services, about 1/3 or so, will need to be exported to the outside with SOAP+WSDL using a bit different set of metadata than internal use requires.

    The question there is is SOAP+WSDL actually a requirement in itself? What you say there suggests XSLT but again, the effort of repackaging some data in “raw” Python or PHP is probably minimal.

    * The data will have XMLSchema definitions probably anyway for other architectual reasons. Atleast base datatypes that are combined into messages in realtively straightforward ways will anyway be defined in XMLSchema.

    That’s again pointing at “document / literal” but doesn’t have to mean SOAP. In general my experiences of XML Schema + SOAP is the relationship between them isn’t broadly understood and is subject to interpretation (rightly or wrongly) and varying implementations – publishing an XML schema alongside a “REST interface” strikes me as something that the “other end” is more likely to pay attention to. In PHP that’s an explicit call to schemavalidate.

    * A usable API for RPC is required, together with mapping to native datatypes. Once the messages are defined, I don’t want to know anything about wire encoding, wire protocol or anything. I just want a RPC with all the behind the scenes including logging, routing, etc. taken care for me.

    Here’s the fine detail part – HTTP is itself an RPC protocol with a small collection of methods (GET, POST, PUT etc.) – SOAP (rpc/encoded) and XML-RPC “tunnel” through existing HTTP methods (normally POST). Anyway – can understand what you mean by “not caring” but think this is an elusive goal. With RPC/encoded SOAP you can get to a “hello world” quickly without having to care about what goes over the wire but given earlier validation requirements, the effort required to define schemas for your data structures could be equally spent defining your own XML formats which, once done, means you can stop caring ;)

    Overall I guess the immediacy you (sometimes) get with SOAP or XML-RPC tails off once your API grows or things like performance become a concern.

    Perhaps one argument for HTTP which is strong and easy to grasp is conditional GETs – this get’s very hard / ugly to do with SOAP/RPC or XML-RPC IMO – you have a return value which is either the result you want to something saying “unchanged” and that translates into ugly hacks, to the point where is no one does it. Where I work we used XML-RPC at one point for moving documents to a web server for display – at some point have to re-write it – when an unchanged document takes 5 seconds to load and people are saying “man this is slow”, having an efficient / easy way to report changes, update caches seems like a really good idea.

    So all said and done, even considering all the talk about demise of SOAP, I’d still go with SOAP+WSDL for now. The reasoning from the PHP side (haven’t had the time to thoroughly study relevant Python libraries):

    Actually should point out that the demise is really for online use on the real Internet – for purely “internal” use, where you control the endpoints, SOAP is just as valid (or invalid) as similar older technologies (like CORBA or DCOM).

    SOAP has ext/soap on PHP side, that makes RPC braindead simple. I don’t know of anything that elegant for REST.

    Although it doesn’t match all your requirements, you need to look at that Amazon example up there again – it’s pretty braindead. Also PHP’s serialized format is pretty well supported in other languages – see the list at the end of this post (but note also the risks). JSON or YAML are also well supported in most languages.

    SOAP has atleast some tool support, REST has pretty much none if you don’t count the fact that in simpler cases the format is simple enough to use by hand.

    Well it could well be argued that REST doesn’t need any extra tools because everything is already there. For debugging your browser plus livehttpheaders might often be enough. There’s a ton of HTTP testing / automation tools, beginning with WWW::Mechanize. Of your requirements only the use of XML schema really needs something toolwise (as in XMLSpy).

    Anyway, if anyone can point out any pragmatic reasons, why should I use REST, I’d be happy to hear them.

    Returning hear again, if you’re willing to donate some project hours, why not just try one REST web service.

    I still can’t believe how slow YAML uptake has been by the PHP community.

    Think there are some specific functions stalling interest, namely serialize() and parse_ini_file().

  • http://www.phppatterns.com HarryF

    When a Microsoft engineer publishes this, it just needs quoting;

    The lesson here is that all this complexity being pushed by so-called enterprise architects, software vendors and big 5 consulting companies is bullshit. If you are building distributed applications for your business, you really need to ask yourself what is so complex about the problems that you have to solve that makes it require more complex solutions than those that are working on a global scale on the World Wide Web today.

  • http://www.lopsica.com BerislavLopac

    Harry, if I understood you correctly, you have listed several technologies and standards (REST, custom XML, XSchema, Relax-NG, XSLT…) as a “better” replacement for a single standard (SOAP, with or without WSDL).

    For one, I absolutely agree with you that the principles behid REST are valuable, and I would like to see it well implemented. However, the current state of the affair is that classic Web services have a more stable foothold in the development environments.

    Fow two, REST defines only half of the story, i.e. the way how to transfer your data and RPC. However, there is still no standard for the actual format of the transferred information — only in these comments we have mentioned several, from custom XML to YAML to JSON to PHP serialization. The advantage of SOAP is that it uses a standard, even if not technically ideal, part of the HTTP protocol for information transfer, and has a standard, well-defined format of that information — SOAP.

    And for three, as I argued above, SOAP has been implemented in virtually every development tool and platform out there, making use of Web services as simple as (in PHP, from a recent real-life example I had):


    require_once('lib/nusoap.php');
    $soapclient = new soapclient('http://www.example.com/soapserver.php');
    $events = $soapclient->call('getevents');

  • ajking

    One of those “Says It All Quotes”:

    “URLs and HTTP is why the web is working. HTTP and URLs work so well they’re even capable of being useful in spite of complexities in media-types based on XML.”

    (From lesscode.org)

  • http://www.phppatterns.com HarryF

    Harry, if I understood you correctly, you have listed several technologies and standards (REST, custom XML, XSchema, Relax-NG, XSLT…) as a “better” replacement for a single standard (SOAP, with or without WSDL).

    Well first you’re taking bits of what I said out of context but I understand the point.

    Guess there’s a short view and a long view here;

    The short view… the technologies I’m talking about and generally well known and most people will be working with some or all in a typical web application. SOAP doesn’t mean you can forget XML and certainly not XML schema. In fact there’s a good chance if you’d be using XSLT somewhere as well – many of the web service development platforms encourage you to work with “XML views” on your database and use XSLT between that and your “web service layer”.

    So really you’re often talking about using all these technologies anyway and the decision is then with or without SOAP.

    The long view… generalizing (and probably badly) there’s basically two mindsets to development these days.

    One is the “plug and play” way – putting together components from different vendors, with help from a mega-IDE, and staying well away from the details – if you’re doing J2EE or .NET, this is probably what you’re doing. Perhaps the key point of view here is “one size fits all”.

    The other is the “get intimate” way, where you get deeply involved in the details and regard most development tasks as a special case that needs you personal attention – one size cannot fit all – this is more the dynamic language way and the primary tool is a text editor of some sort.

    I’d argue REST fits the latter view of development and SOAP the former.

    And I think it’s safe to say Yahoo, Google, flickr, del.icio.us and most any of the big online names use the latter approach.

    For one, I absolutely agree with you that the principles behid REST are valuable, and I would like to see it well implemented. However, the current state of the affair is that classic Web services have a more stable foothold in the development environments.

    Yes and no. That REST has a label is almost a problem in itself. Really we’re talking about stuff people already use, in particular HTTP and perhaps XML. You can find HTTP clients and server for most any language and platform you care to name, plus tools for logging, debugging, authentication etc. XML in general is also well done.

    But there are some areas where best practice and strategy is not well defined. For example how do you publish all your URLs in a manner which is easy to discover – perhaps this is the nearest thing to “WSDL for REST” (although you could use WSDL itself to REST).

    But turning the guns of SOAP / WS-*, although there’s plenty of tools and implementations, my experience is it either “just works” and everyone breathes a sigh of relief or it doesn’t work, and you’re in for a debugging nightmare. Because SOAP et al adds extra layers on top of HTTP, it gets very hard to see what’s really happening. Add to that that SOAP is trying to be everything to everyone means that what one person does with SOAP (e.g. document / literal style) may not be well supported by an implementation on another platform.

    Fow two, REST defines only half of the story, i.e. the way how to transfer your data and RPC. However, there is still no standard for the actual format of the transferred information—only in these comments we have mentioned several, from custom XML to YAML to JSON to PHP serialization. The advantage of SOAP is that it uses a standard, even if not technically ideal, part of the HTTP protocol for information transfer, and has a standard, well-defined format of that information—SOAP.

    OK here lies part of SOAPs illusion – there’s basically two styles of encoding with SOAP – encoded and literal and it seems you’re talking about the encoded style (nice summary here BTW).

    In encoded style, you have a one-to-one correspondance between native data types (string, int) and tags in SOAP (xsd:string, xsd:int).

    Now that’s fine, but what, as stepz seems to require up there, do you do if you want a rule to apply to a particular string (e.g. a username) that it’s between, say, 6 and 15 characters and that the characters are alpha-numeric?

    At that point SOAP’s basic type encoding doesn’t help you – you need to add your own XSD to register this rule. At the point you need to start doing this, encoded SOAP loses some it’s value – generally if you want to validate some tags, you want to validate all tags – at that point you way as well have gone with document style SOAP and you’re basically in the same position as you’d be if you defined your XML format from scratch, and you start wondering what the value of SOAP was in all this. And remind me what was the benefit of SOAP vs. XML-RPC again?

    And for three, as I argued above, SOAP has been implemented in virtually every development tool and platform out there, making use of Web services as simple as (in PHP, from a recent real-life example I had):

    I’d still say you can’t beat the example above for simplicity. Similar can be done with “plain old XML” like this.

  • http://www.lopsica.com BerislavLopac

    Harry, this is all completely besides the point. You say that SOAP ads a layer on top of HTTP, but anything adds a layer on top of HTTP — HTTP, RESTful or not, only defines how will the transfer be initiated and performed, but not in which form the data will be transfered. When we use REST, we still have to decide how to wrap that data for HTTP (which is a text-based protocol) to be able to transfer it at all.

    At that point SOAP’s basic type encoding doesn’t help you—you need to add your own XSD to register this rule.

    There is no way that I would ever delegate it to the transfer protocol itself to validate the data it transfers. The data should be validated either at the source or at the receiving end; in between it’s just transferred.

    Web services are really nothing more than glorified RPC’s, and should be viewed as nothing more than remote routines, and treated that way.

  • http://www.lopsica.com BerislavLopac

    And to make something clear: I am not against SOAP, on the contrary. It’s just that I see that Web services have great use in everyday work, and I don’t think they’re going anywhere soon. REST is a wonderful concept, but WS’s are more than adequate for their basic purpose.

  • bonefry

    Just to tell my oppinion … RPC is usefull, and until we invent a simpler sollution to SOAP … SOAP will not be replaced by REST, although we could see REST being adopted more.

    In today’s frameworks, in JAVA/.NET and even Python and PHP … it is a super-simple task to create a web service … you just define some methods and annotate them as being web-services.
    The point is that … the underlying technology (SOAP in this case) is transparent and could be replaced, if needed, with CORBA / XML-RPC / whatever new technology comes along … that’s why abstractions *ARE* usefull.

  • bonefry

    It’s rant time again (apologies—move along unless you’re feeling fiesty). It’s that REST vs. SOAP thing.

    I forgive you this time :) … this subject needs to be debated

  • Anonymous

    posting az comment

    • Guest

      Reply to comment

  • Guest

    Test comment 2

  • Anonymous

    test

Recommended
Sponsors
Because We Like You
Free Ebooks!

Grab SitePoint's top 10 web dev and design ebooks, completely free!

Get the latest in Front-end, once a week, for free.