Introducing XP/Agile processes into a company

[Luke Redpath]

Hi there, I'm trying to get the company I work for to look at agile processes and XP although through my lack of experience (I only know what I've read) and lack of overall understanding, its quite difficult. Its also made difficult by the fact that we have developers working in different parts of the country.

We have a quarterly meeting coming up soon and we'd like to get somebody down to give us a talk/presentation about Agile/XP/Scrum/Other agile processes etc, how they work, how they are implemented etc. Also advice about how to sell to clients, how to quote/estimate job times/costs and how to deal with the notion that clients will only deal with hard, fixed-cost estimates.

Does anybody here do the above, know somebody who does the above or know where we can look for the above. Also, how much it would cost for somebody to come down for an hour or two. Our central office is based in London.

Any advice is much appreciated.

[Roger Ramjet]

Well, first time I've come across this agile/xp thing, so I've had a google and a read: and it's nothing new. It was called Evolutionary Rapid Prototyping back when I was doing my masters in 1992, so this is just re-branding an old idea so far as I can see. Good career move for someone, but hardly original.
viz:
Structured Rapid Prototyping
Rapid Evolutionary Development
Rapid Software Prototyping
Software Prototyping Rapid Applications Development
etc etc
That is why I had not noticed something new called Agile, cos if I saw it I just filed it away under ERD.

Now the most fundamental thing that you must realise about all this is that it's just a mindset. Like TQM (total quality management), ERD is a culture not a method. And like TQM, it does not work without that culture change.

Now I naturally adopted ERD as my development method once I had adopted TQM as my philosophy. TQM means continuous incremental improvement, and therefore continous change. So any system is just a work in progress, not the final solution. That applies equally to a manufacturing process in a factory, or a database in a call centre.

The problem most organisations have with this is that people develop vested interests and personal attachments to the way things are done now: and we all get unsettled by change. Line management especially does not like uncertainty, so the organisation as a whole tends to resist change.

ERD, or Agile, or whatever you want to call it now, leads naturally to modular, loosly-coupled system that are flexible and easily extended on an as-needs basis when the business case is made.

So, how do we do ERD/XP/Agile? By doing TQM: ie. from the bottom up. It is axiomatic in TQM that the real expert in the job is the person that actually does the job every day. So you don't go to the manager to find ways to improve things, you start on the shop floor. Typically with structured design methodologies the development process tries to model the whole process first - from the top down. And this is what leads to those massive development failures that we hear about, like the stock market Taurus system £700million, or the Ambulance booking system £350million and 25 lives. All of these had fundamental flaws built into them from the start because the developers ask management what the system would have to do, and they really did not know - not the details anyway.
With an erd development you start with the people who will use it, find out how they do it now and ask them what they need. You identify small opportunities to improve their lives and rapidly deploy simple solutions. This in turn leads to other opportunities for improvement, and before you know where you are you've got dozens of simple modules that together add up to major change.

If your bosses want concrete examples of such system just point to UNIX and other open source technologies, and compare them to the MS alternatives: Apache v. IIS, UNIX v. NT, etc. All of these were developed in a piecemeal way, as pragmatic solutions to specific problems; and they have continued to be developed that way. Small incremental changes by diverse developers is what characterises open source software and what makes it so powerfull and functional.

PS. Be delighted to come and give a presentation for you in London, just contact me at d.soussan@sovereignls.co.uk

[lastcraft]

Hi...

Well, first time I've come across this agile/xp thing, so I've had a google and a read: and it's nothing new.

Hm, yes and no. Only "not new" in the sense that everything has an evolution. SRD was one of a whole bunch of systems floating around as a reaction to waterfall approaches and the promise of reusable components in the late eighties.

SRD predates XP by only a few years, and was clearly an influence. What XP has done, and other follow-ons, is to turn the theory into practicality. The original "white book" is virtually a "how-to". It includes the mechanics of refactoring, lightweight design tools, the role of testing and the precise relationship between management and the development team.

XP took every good practice; code review, iterations (SRD, RUP), continuous requirements. testing, use cases, evolutionary design (SRD, et al), and turns the knobs up to 10. Code review became pairing, testing became unit testing (nothing gets checked in without tests) and writing acceptance tests before coding, continuous requirements became customer on site, use cases shed some weight to become stories and evolutionary design became refactoring. XP at least is not just SRD, but a blending of a whole bunch of stuff.

Because XP was prescriptive, rather than just a theoretical IEEE paper, it quickly took off amongst developers. Four years ago I used to have to explain what it was, now most developers at least know the basics, even if they haven't tried it.

The management friendly term "agile" would definitely include SRD. Agile has since rediscovered theoretical underpinnings from TQM and lean manufactoring. In fact some systems come from manufactoring, Scrum in particular came from Toyota.

Back to the core problem...

Regarding the management view, Agile has not crossed the chasm to become mainstream. You won't get them to change the way they handle contracts until they have seen the system in action enough to be comfortable with it. It's just too big a leap of faith. Intially you would just be using it internally.

This is a good thing though, as it takes time for all members of the development team to learn not just the practices, but also the reasons for those practices. This can take 6 months and you'll get quite a few arguments.

I would start with the developers only. Examine the systems you currently have in place to ensure delivery and decide where the gaps are. Introduce practices that fill those gaps. Definitely introduce iterations and have developers do "ideal days" estimates. You need this later, so start early. The new practices will lead to some of your current approaches proving unecessary, and you will readjust some more. After a few months of iterations, you should be ready to export the agile approach.

Management and development collide on the business of estimates and scheduling. It's here that you must take your first steps into the broader organisation. The fundamental principle is that develpers estimate, management schedule. Never the other way around. In order to present management with a simple set of predictable estimates, you have to "flatten the cost curve". If your codebase gets messy, then a three day job today may be a ten day job tomorrow. Management cannot plan with such a system. You need to make your development process predictable.

You want your numbers to actually pan out. This is going involve skilled developers that are capable of injecting design and architecture into the code as you go. Are you ready for that?

I would do that first, and be ready to back it up.

Later, when management have used it in-house, they may want to investigate agile contracts. Start with outsourcing deals where your company is paying the money. They can hardly sell the idea to their clients if they haven't experienced it themselves.

yours, Marcus

[hutchic]

XP/Agile isn't for everyone are you sure your company is the right environment?

[Roger Ramjet]

XP/Agile isn't for everyone are you sure your company is the right environment?

Quite agree, hutch, quite agree. It is more about attitudes and culture than tools and methods.

Because XP was prescriptive, ...

Now that's what I have a problem with. You see there is no formula for success - but there are formulas for failure. Indeed, if there are no formulas for success then all formulas must be formulas for failure - eventually. My worry is that people all too easily just adopt the form without adopting the culture that will make it work.

Only "not new" in the sense that everything has an evolution.

Yes, I suppose we all tend to categorise things under the name we first encounter.

Best of luck selling it Luke, best of luck.

[McGruff]

You see there is no formula for success - but there are formulas for failure.

It sounds like you've had a bad experience with XP. It would be interesting to hear how and why it didn't work for you.

[EDIT: scrub that - I just realised you already mentioned that you've only just discovered it]

[dagfinn]

For this kind of question, I've recommended the Yahoo group on XP before, and I'll do it again:

http://groups.yahoo.com/group/extremeprogramming/

[Luke Redpath]

Thanks for the link dagfinn.

Here is one question that I have often wondered about...at the beginning of an iteration you schedule a number of user stories to be implemented based on their estimated ideal days and your teams velocity. I think I understand this, but what happens if you complete those stories before the iteration is over? Is this just a sign that either your estimates are off or your teams velocity is wrong, and do you just stop work on the project until the next iteration, or do you start the next iteration early? What do you do with the leftover days?

Also, being a developer who works mainly on web-based development projects, I've found that a lot of the user stories I've looked at tend to work out as being often no more than half an ideal day of work wherease a lot of books I've read talk about working out stories in terms of ideal days. Is this just a consequence of web-based projects being less complicated (on average) or a case of the user stories being too fine grained?

[chiefmonkey]

Came across this link (via osnews) haven't had a chance to read it yet, but it may be of interest to some.

Perils and Pitfalls of Agile Adoption
http://www.informit.com/articles/art...?p=441115&rl=1

George

[Roger Ramjet]

Came across this link (via osnews) haven't had a chance to read it yet, but it may be of interest to some.

Perils and Pitfalls of Agile Adoption
http://www.informit.com/articles/art...?p=441115&rl=1

George

Yes, well worth a read, sums it up very well. I especially like the bit about the 'roaches'. I stopped working in large organisations because they are crawling with them, and personally, I'm task-oriented - I don't do politics.

[lastcraft]

Hi...

I think I understand this, but what happens if you complete those stories before the iteration is over?

You have a mini-meeting and bring more stories in. Otherwise your velocity could never go up .

Is this just a consequence of web-based projects being less complicated (on average) or a case of the user stories being too fine grained?

Do you have some examples. We go down to half days as our smallest unit.

yours, Marcus

[lastcraft]

Hi...

Now I naturally adopted ERD as my development method once I had adopted TQM as my philosophy. TQM means continuous incremental improvement, and therefore continous change. So any system is just a work in progress, not the final solution. That applies equally to a manufacturing process in a factory, or a database in a call centre.

So what has been your experiences doing SRD? Pros and cons?

yours, Marcus

[Luke Redpath]

Do you have some examples. We go down to half days as our smallest unit.

yours, Marcus

Nothing concrete, but lets take for instance, a really simple blogging system, as a hypothetical project as thats really easy.

Brief overview: a basic blogging/publishing system is required so that the blog authors can post new stories. End users will be able to view the latest entries on the blog homepage, as well as browse the entries by date and category. The end user will also be able to search for blog entries, and leave comments. The blog admin needs to be able to create new categories and moderate comments and manage other blog admins/authors.

Lets say there are three recognisable roles/actors, Blog Admin, Blog Author and Blog Visitor.

Now, using my limited experience of identifying user stories, some of the stories I'd identify are:

Admins can manage blog users
Admins can create new entry categories
Admins can moderate visitor comments
Authors and Admins can post new entries
Authors and Admins can edit existing entries
Authors and Admins can delete existing entries
Visitors can browse entries by category
Visitors can browse entries by date
Visitors can search blog entries
Visitors can leave comments on blog entries
Visitors can see the latest entries on the blog homepage

Looking at the above, I'd say some of those stories are definately no more than an hour or so's work, especially using something like RubyOnRails which lets you roll out such basic functionality quickly. Does this mean my stories are too fine grained? Perhaps the post/edit/delete entry stories could be combined into one "Authors and Admins can manage entries" story? Perhaps I'm just underestimating.

Also are the stories detailed enough? Or should they be short one-liners at this stage that are discussed in more detail including being broken down into individual tasks at the iteration planning meeting?

[Sagewing]

Welcome to my world - I spend many of my days working with organizations to help them adapt formal process. I'm a Rational Consultant, but I also work with XP, Agile, doors, CMM, etc.

Demonstrating the need to adapt process is very tough! I have learned this: convince the decision makers, and push forward. The developers will fight you, but will eventually see the value of the process (if its the right process) . Some of them will never come around, and there's nothing you can do about that.

The important thing is to keep moving. If you spend weeks pitching the process to the developers, all you've really done is bored them to death. People tend not to change their minds until they see it in front of them.

[Roger Ramjet]

I'd say both too fine grained, and not hierachical enough

3 types of users, admins, authors, visitors. Admins are also authors. Everyone is a visitor untill they login.

Authors can add, edit and delete their own entries.
Admins can add, edit and delete anyones entries.
Admins can create, edit and delete authors.
Admins can create, edit and delete new categories.
Visitors can etc,
Admins must moderate visitors comments before they are posted.

If you make it too fine-grained the chance exists that it will be implemented that way. By grouping functionality you can focus on general-purpose components and move from them to the specific.

[Roger Ramjet]

Hi...



So what has been your experiences doing SRD? Pros and cons?

yours, Marcus

Liberating, is my experience. You see I'm 52, been in computing since the 70's. When I was in a masters program in 92-93, I wanted to focus on failure: definition and causes. But that was not what the department wanted. Computing had become sexy and everyone wanted in on the gravy train. The last thing anyone wanted was a prophet of doom going around reminding them that 60% of all systems developments were total failures. Not after 20 years of developing Systems Development Methodologies to tackle that problem. Not when Prof Porter was touting the new paradigm: using IT to leverage a competitive advantage in the market. IT was moving into the boardroom and no-one wanted to rock the boat with tales of failure.

The fundamental problem has always been the gaps between what people think they want and what they say they want, and between what they think they want and what they actually need.

The pros of ERD seemed to address that problem in fundamental ways.
1. clients see working code very early on in the cycle. This reassures management that their money is being effectively spent. This also makes contract renegotiation much easier cos when that happens they will already have lots of delivered functionality
2. the real end-users get to voice their criticisms very early on. These are the people who can make or break your system in the end. Getting them to buy-into the system and take-ownership at an early stage is vital.
3. getting close to them with early prototypes, you get the chance to find out how they really do their jobs, not how they tell management they do their jobs, or management tells them they should be doing their jobs.
4. both you and the end-users are learning. It is delightfull to have them come to you with insights and opportunities for your system that you would have missed. Instead of having that, "If only we had known that earlier on we could have incorporated it into the structure. Oh well, it's too late now" feeling. You get added value for little extra effort.
5. basic, fundamental issues like time-constraints become obvious early on. These are the booby traps that no-one mentions untill it's too late because they are assumed knowledge. When you sit down with an end-user at a protoype, bang, they will point the problem out right away - often in triumph: "See, I told you we don't need to change things around here". Instant buy-in when you fix it cos they really fixed it, and you should tell them so.
6. it gets my creative juices flowing early on. Instead of the classic, analyse and document the current system, .. etc, I get down to exploring solutions from the outset. I especially like it when this leads to the realisation that we were trying to solve the wrong problem in the first place.
7. the satisfaction of delivering software that really does do something usefull, even if it is only something small. And the compliments that follow from end-users. For some of my clients I'm the only IT guy they've ever dealt with who delivers what I say I will, always.

The cons are actually all the same cons that apply to all other methodologies
1. need talent and experience
2. needs self-discipline
3. difficult to produce estimates that are not just phantasies

In the end, it is the nature of the software that all of these rapid development methods can produce that matters for me. Simpler, loosly-coupled modules that can be easily bolted on, modified, or replaced as required. IT is a tool technology, and the simplest tool is the best. It has to be just good-enough for the job in hand, anything more is just a waste.

[OfficeOfTheLaw]

For this kind of question, I've recommended the Yahoo group on XP before, and I'll do it again:

http://groups.yahoo.com/group/extremeprogramming/

Indeed... this is one group I am heavily involved with and has a lot of good messages going back and forth... as well as postings from time to time by the core founders of XP and Agile (I was a bit suprised awhile back when Martin Folwer posted too!).

I'd recommend searching first though... "How can I sell management on XP" and "How can I adopt XP" are frequently asked questions.

[OfficeOfTheLaw]

Also, being a developer who works mainly on web-based development projects, I've found that a lot of the user stories I've looked at tend to work out as being often no more than half an ideal day of work wherease a lot of books I've read talk about working out stories in terms of ideal days. Is this just a consequence of web-based projects being less complicated (on average) or a case of the user stories being too fine grained?

Don't confuse the specified units with the actual concept. When you estimate a story, you estimate how long you think it will take, there is no actual unit that you MUST work it out under. Some stories can be completed within minutes... some on the other hand could take days.

Anyway, for your case, I would suggest this book as good read, then move from there. Also, keep in mind that, although XP is implemented differently by different organizations, you can't do XP without the core tenets (i.e. you can't do XP and leave out pair programming, collective code ownership, TDD, etc).

Also, beware of the haters. A lot of people seem to have a grudge against XP without actually trying it. Truth be told, it's not for everyone, and it is no silver bullet, but so far I've observed that those who have tried it for a 12 week period tend to permanently adopt it.

[Luke Redpath]

Anyway, for your case, I would suggest this book as good read, then move from there. Also, keep in mind that, although XP is implemented differently by different organizations, you can't do XP without the core tenets (i.e. you can't do XP and leave out pair programming, collective code ownership, TDD, etc).

I've read that book and to be honest, I thought it was terrible. Too much focus on pushing their whole XML/XSLT methodology which I just wasn't interested in and I thought it was pretty out of date too, from a web point of view.

And do you really need to pair program to do XP? Its really not something that would fit in with our situation, though there aren't many problems with the other aspects.

[OfficeOfTheLaw]

I've read that book and to be honest, I thought it was terrible. Too much focus on pushing their whole XML/XSLT methodology which I just wasn't interested in and I thought it was pretty out of date too, from a web point of view.

And do you really need to pair program to do XP? Its really not something that would fit in with our situation, though there aren't many problems with the other aspects.

I'll be honest... I never read the book I just suggested. I just thought it may be applicable. Where I come from is more from experience + reading a slew of rather excellent books that Marcus suggested, but suggested the web XP book thinking it had something good since the rest of the series (which I have read) was very good.

Pair programming, although it doesn't sound good in concept ("Why would I want to program with someone looking over my shoulder the whole time?" or "Pair up with someone I hate... yeah, that's helpful!") does quite well than you would expect. The biggest problem at my last job was that I was the most skilled (no ego, just the truth) and worked on several large, complex projects. When it was time for me to go, no one else had any experience at all with any of them, knew nothing, and had large API libraries full of dozens of domain objects to study. To be blunt, the last 2 weeks there was a maddening time spent training coworkers in these projects and why certain design choices was made.

With pair programming, you share experience. People learn from each other, everyone gets a chance to understand the system, to build off of each others experience, and in the end make better design choices. In my case, I always over complicate things, which my peers always point out simpler solutions that I miss.

At my current job, they paired me up with someone during the kinterview to do pair programming sessions with. By the end of the day, I felt I had a complete understanding of the system and even a week after the interview the domain model was fresh in my head. The best thing though, imho, is that you have someone to discuss design decisions with and can argue it out, and in the end come up with a better solution.

Of course, it's 100% XP where I am. Each workstation consists of two keyboards and two mice connected to one workstation, with 2 monitors sharing the same workspace, and once in the center for misc activities (looking stuff up). Sometimes it can get hectic if we disagree and both of us try to take control, but everything always seems to work out fine in the end. Heck, during the interview session, I picked up so much so fast, I was even able to make improvementsto what was being worked after less than 2 hours!

Of course, to get there it's going to take small steps, but I think pair programming has to be the underlying principle if you want to adopt any agile development process because it always boils down to two heads are better than one.

If you really want to get serious about adopting XP, join the yahoo mailing list, ask questions, and see if there is a local XP group in your area. I was only able to find one about two hours away when I first looked up XP, but attending meetings and learning from others (as well as networking and landing my current job through the group) is time well spent.

[OfficeOfTheLaw]

Also, http://www.cs.utah.edu/~lwilliam/Pap...eeSoftware.PDF is well worth the read.

[33degrees]

And do you really need to pair program to do XP? Its really not something that would fit in with our situation, though there aren't many problems with the other aspects.

You do need to pair program to do XP, but that doesn't mean you can't simply take what you like out of XP, and there are other agile methodologies that are worth looking into. I'd suggest checking out Alistair Cockburn's book Crystal Clear, which is considered by some to be more a more realistic methodology than XP. It certainly puts a lot of emphasis on habitability, as he recognizes that any methodology that requires too much effort will get abandonned.

[Luke Redpath]

You do need to pair program to do XP, but that doesn't mean you can't simply take what you like out of XP, and there are other agile methodologies that are worth looking into. I'd suggest checking out Alistair Cockburn's book Crystal Clear, which is considered by some to be more a more realistic methodology than XP. It certainly puts a lot of emphasis on habitability, as he recognizes that any methodology that requires too much effort will get abandonned.

Well the main reason we can't do XP effectively is because we are all in different offices across the country.

I know there are apps out there which will let you edit documents together online but I wouldn't have thought it would be the same.

[OfficeOfTheLaw]

Well the main reason we can't do XP effectively is because we are all in different offices across the country.

I know there are apps out there which will let you edit documents together online but I wouldn't have thought it would be the same.

There are organizations out there that use pair programming with programmers that aren't geograhically in the same location. This usually involves some shared desktop software such as VPN and video conferencing.