Agile Presentation at a Project Management conference

http://blog.crisp.se/2014/03/11/henrikkniberg/keynote-slides-from-my-passion-for-projects-keynote

Work-Flows with Git and Visual Studio 2012-3

Dev work-flow with git.


Git Visual Studio 2012 Tools. Aligns Vs 2012 with the new git support in 2013. It doesn't require TFS to work.
If you don't like the new Team Explorer panel in Vs 2012 and 13, you probably won't enjoy the git integration either.  I personally find the user experience fiddly in such a small panel, and some functionality seems to be buried multiple clicks deep.  However the git integration is cleanly integrated into Vs, but I thought it weird several key features that should be at the forefront missing from the solution explorer panel. File history (log), compare, and revert are only available from Team Explorer.


TortoiseGit.
Basically a port of Svn Tortoise for git.  It feels familiar and stable. There doesn't look like there is any new innovation to set it apart from Svn Tortoise.

Git extensions. This is an open source alternative to Microsoft's offering above.  Also includes explorer integration and Visual Studio Integrations (2008,2010, 2012, 2013).  In my opinion this tool gives a superior user experience both in explorer and in Vs. There is a convenience toolbar in Vs, context menu items in solution explorer, and some thought has been put into surfacing frequently used features with an appropriate level of detail.
I do have to add however, it was problematic to install. First off, it defaults to install in C:\Program Files (x86), however some commandline utils required do not like the '(' and spaces in their path, so they don't work. But, even after changing the install location then 'git-credential-winstore.exe' failed to work at the custom location. It all feels a little difficult and unrefined to be honest.

GitHub for Windows
 This tool has a very nice looking user interface and its obvious considerable effort has been invested in creating a Windows 8 Modern UI.  It makes it very easy to create, view change sets, discover and commit changes.  However it seems to be missing commonly used features I'd expect to be there. For example being able to view the history on a single file; or a blame report; or comparing; or launching a Vs solution on double-click (or running the associated exe by extension), or specifying a custom folder for clone.  A nice UI but missing features that will stop me from using it.

Rather annoyingly it does not allow you to customise where it is installed so it ends up somewhere like this:
C:\Users\Ben\AppData\Local\Apps\2.0\Q7XE9020.PW7\T3LQDOXZ.R2W\gith..tion_317444273a93ac29_0001.0002_5f2f02fff55a2354\GitHub.exe
Nice. Thanks.
Too bad, if you have a SSD C: and want all non-essential apps on another disk.

What have I ended up with?

Rather shockingly no one tool seems to provide a feature complete, low friction, and modern user experience. I ended up using GitHub For Windows to clone a repository, and I have to customise the default path in global settings to get each repository located where I want them. Then using Git Extensions to do the every day stuff.

So why not command line?

Well, there are many command line utils out there, and they all are probably more reliable and complete than these graphical tools.  However, a command line interface is fine for a highly motivated and skilled developer, someone who is interested in command lines.  But, commndline interfaces have a steep learning curve, often involve lengthy command lines easily typed incorrectly, and they make visibility of available options difficult to discover.  Trying to teach a team of people a command line tool, when they are not interested in the tool, they just want to get their work done is very problematic. In my opinion apart from the enthusiast, they are inefficient for anything other than automation. 

The role of an Architect in Agile Part 1

I had an interesting conversation with a highly experienced architect recently:
"Is what you're doing working? Really? Are you sure?"
If you answered yes, then don't change anything.
More than likely however, there are recurring quality problems, defect count is high, bug-fixes are rejected, and developers spend a great deal of time refactoring. You need an architect and better architecture processes.

I pondered about the title of this post, should it be the role of an architect in Agile or just modern architect? Probably the latter, but the question of architecture in agile is something that bugs more than a few people. I certainly don't have all the answers nor proclaim to be an expert, but these are my experiences and observations.

So what's the problem?

Scrum and agile narrations focus on writing code, building code, not the "ideation" and design that is necessary.  Who isn't following Scrum or some self proclaimed "flavour of agile" these days right? Almost never does agile documents mention architecture or design.  They expect the team to design and build as part of a sprint, or maybe spend the first sprint just on design but no help on how this should work. In my experience this is fraught with difficulty. Design by committee doesn't work well, its usually better to have the most experienced person responsible for design and to properly think through a design and proof of concepts takes time. They should, of course, engage other team members for input and review, but one person needs to be responsible. Things go better if the senior(s) have a vision of what the end-goal architecture should look like, building this vision with clarity is the issue.

Stephen Cohen (Chief Architect, Microsoft): "Scrum went through 3-4 years in its original form before admitting that the architect had any role at all to play."

Juval Lowy (Master Architect, IDesign): "The agile priests would like you to believe that following agile will magically produce architecture that is adaptable, resistant to poor coding, and is scalable."
(Apologies for the paraphrasing).

Some practioners claim there is no need for design and it should be simply part of the implementing the story. This works in only the most trivial software, but in my experience most of us veterans would not often descibe what we do as simple. More often than not the code produced is badly structured in retrospect and not properly limiting volatility when business change occurs. Agile addresses this by saying "continuously refactor". How much time could be saved by mapping out an over arching architecture up front?

Wait, aren't you saying that you'd rather do waterfall?

No.  Architecture != Waterfall.

A wise friend once said:
If you put a bunch of extreme programmers in the middle of a city, let’s say Marrakech, and ask them to visit five tourist hot spots without using a “map”, they will wander around for days exploring every little passage way by brute force. If you give another bunch of developers a “map” and put them in the same city, they will use the map to go directly to each of the five tourist hot spots in a matter of minutes or hours.

Architecture is the map. You'd be crazy not to have a map (or get started creating one).

Simply sitting down with a one liner on a piece of card does not mean any developer from graduate to senior will be able to first estimate, then write tests, then magically produce well designed code that properly encapsulates volatility. This is a naive and utopian ideal that never happens.  If the architecture is already mapped out and clearly articulated then it absolutely is.


Once you have a map, in enough detail to be understood by your team, Scrum works okay. It should be done before the team is assembled to begin building.  

You should strive to create a shadow board architecture that allows making design decision for new business features easy, consistent and fast.

Ok, so architecture is important, but do we need an Architect?

What skills does agile require of its team members?

• Cross-skilled (Dev, QA, SOA, UI, UX, DBA, Design, and anything else required);
• Able to work on and explain any part of the system;
• TDD & Unit testing;
• QA practices;
• Proven architecture and design skills;
• Industry context knowledge and patterns;
• Ability to write SDK documentation;
• Ability to consistently review code;
• Mentoring skills;
• Able to create processes and procedures to ensure predictable outomes (ie adapt);
• Good communication skills;
• Able to effectively peer program;

Sounds very much like a very senior experienced veteran developer, if not an architect.  Every company I've been involved with would like all their developers to be at this level, but all never are. You'd be lucky if 20% of developers are at this level, usually not enough to have one per team. The people with these skills are in high demand, but the company will want to make do with what they have. Which means these people will likely be shared around out of necessity. Agile more than frowns on team members changing teams all the time, and there is no room for negotiation on that, believe me!  Even with no management direction senior developers will attempt to fill the design void. So, these highly skilled and experienced people are your architects, if not in name, then they are performing the role de-facto.

No one would argue that business needs are changing faster than ever. This necessitates good design that encapsulates volatility as best it can. The role of an Architect is most definitely STILL REQUIRED and because of faster pace of change, more relevant than ever.  

How should architecture fit into an agile process?

It is common to begin (and sometimes complete) the UX work before the Scrum team starts work. Architecture is no different, it should be running in parallel to UX.


Architecture should not be a heavy process, it should be mapped out once at the beginning for the whole new system, and sometimes before each major feature addition to an existing product.  According to  IDesign's "The Method" architecture can be completed in 3 to 5 weeks.  It is not something that happens for each user-story.


Some highly respected architects I have spoken to simply state a proper architecture process does not fit with Agile, period. I won't join this debate, however, agile isn't going to step aside any time soon. To date, I haven't ever won the debate on Agile versus "The Method". It will take time. For now, I believe we should work with agile. 


Scott Ambler posted his take on fitting architecture into agile processes. The idea is to envision the architecture during the requirements and analysis phase. (Before the development sprints begin).  The key is just-enough, then during development keep reiterating over it during the implementation phase.  Just-enough for your target team to comprehend.  Amend and refine when necessary.

Architecture is not something that is needed for every story.  It is the end-goal vision of your system.  This may happen once per release or even less frequently. You must have a clear idea of where you want your system to be 3 or 4 releases from now.  Its ok to change the vision.  The vision doesn't have to consider business features in detail, its the more like an overarching strategy for implementing them, but you should consider likely business extensions and how your architecture will support the changes.

Are you building a pluggable SOA system? If so, are there standard patterns all communications should use?  Is it worthwhile applying aspects to all components? What would happen if services organically grew where developers thought it was quickest and easiest to slap them in? What about realistic and likely business changes? Be cynical, its healthy. These are the questions important to consider when designing an architecture.

It takes effort to make something appear easy, and its easy to make something complex. - Unknown.

The role of an Architect in Agile Part 2

What should an architect be responsible for?

An architect or team of architects should be responsible for: 

• Assisting in gathering and analysing requirements.
• Sell and negotiate technical constraints and aspects to stakeholders.
• Articulate the end goal architectural vision designed to encapsulate change.
• Communicate with diagrams the intended architecture and implementation plan.
• Prove (or disprove) technologies and techniques (prototyping).
• Own the definition of integration points and service interfaces.
• Establish processes to ensure predictable outcomes and quality in the SDLC.
• Oversee delivery of software in accordance with the intended architecture (reviews).
• Acceptable quality levels (coding standards, code metrics and performance parameters).

The architect is primarily responsible for creating a big picture of how a system should be assembled, integrated and where extension points are for probable business additions. This should include mostly diagrams, lists and pseudo-code, because developers don't read. The key is to keep it simple, just enough for the targeted team.

Controversially IDesign's "The Method" argues that the architect should also project manage. The explanation was simple, who else knows best how things integrate and their dependencies?

During implementation the job isn't done. Architecture is also governance of ensuring all these things have happened. An architect should oversee a feature from inception to implementation to final load and concurrency testing.  A Scrum team should only be working on one feature at a time (this is a founding principle of agile - focus and don't context switch).  This allows an architect to potentially oversee two features across two teams at the same time. Its also worth considering the architect joining the team as a development resource, if resources permit.

Any respectable Scrum coach will tell you that Scrum does not mandate no design. It simply states do what is required. If that means design can be done inside the same sprint as implementation, do it, although that is highly unlikely and I wouldn't advise it.  The best plan is to complete the necessary design work prior to initiating a stream of work with the Scrum team. The trick is don't do too much. Too much will vary depending on strength of experience in the team, and how effective they are at communicating. The more inexperienced the team the more design work needed.

Having architecture artifacts up front before starting a piece of work, and for grooming sessions, means it is highly visible and scrutised. This inevitably improves the design, and reduces unknowns.

Who's the boss?

Answer: The client. The Product Owner (PO) represents and speaks for the client. That is not to say the architect shouldn't have direct contact with the client, the PO is there to help you with this. Embracing change is the key to a successful software product. Often more valuable information comes to light during the project. Expect change. However, an inexperienced PO will change their mind more often than necessary. This can be addressed with a little more architecture work, this will keep the PO's reasoning honest. Any whiff of analysis paralysis means you either dump the feature/use-case or dumb it down and start the Scrum process with basic a implementation only. Unfortunately a PO that changes their mind too often results in poor qualilty software that costs more than it should to maintain or extend.

A good PO will work with their architect during the sprint review to ensure the quality metrics are adaquate and all agreed standards have been met.  Although the architect is technical, they are definitely in the Product Owner camp. The PO should lean on their architect for technical advice and verification of the product delivered.  The PO should fail the sprint if the architect can prove that the implementation doesn't comply with the design, or standards have not been met.

IDesign's "The Method" advocates the architect head up the project team and be the one to call the shots and sign-off the final product. This is a very tough sell to the establishment today, but it could be the future.

Some tips on how an architect can go about their job?

(Thanks to IDesign's Michael Montgomery for the inspiration for this list).

• Formalise and document your requirements gathering approach.
• Formalise document templates to store the gathered information. 
• Don't gather information using sticky notes! Save it digitally using defined consistent templates.
• No gold-plating. Everything is “just enough” based on known and likely use-cases, team composition, and hand-off-point.
• Iterate on everything, including requirements, getting “just enough” before the Design Phase begins. Gather - Refine - Review/Assess, Gather - Refine - Review/Assess, until your ready to start implementation.
• Continually reinforce to the BA's and PO's that they bring the ‘when, what, why, who’, but never the how.
• Developers don’t read. Format use cases as ‘pseudo-code’ (i.e. numbered/bulleted ‘lists’) and diagrams.
• Never use the term ‘spec’. Call it the ‘requirements lists’ or 'acceptance criteria'.
• With the advent of UX, separate functional (UX/UI) from back-end service operational (SOA) requirements lists. Treat back end services as a different product to a client UI. They will have their own lifetimes driven by the same use cases.
• UX is different from UI is different from SOA is different from Framework. Each may need its own requirements treatment, depending on the size and complexity of the system.
• Developers don’t do UX (although they may think they can - don't believe them).
• The UX guys must involve stakeholders early and often. This will need to be quite mature before implementation can begin.
• For user driven applications (which is most), it’s well-crafted UX workflows that produce clearly defined use cases that produce succinct SOA. If you can get the UX guys ‘out in front of the ball’, you put yourself in a sweet spot. This particularly holds true in composite UX.

Conclusion

Given a choice, I would prefer to use IDesign's "The Method". My problem with it has been convincing employers and stakeholders, even though it's a new approach, it's a proven one. Others with superhuman sales and negotiation skills may have more success than I. Working in with Scrum, its roles, and other established roles has been a given for me; and I'm betting you too. However, I believe this is completely realistic in a hybrid model. The role of an architect is absolutely essential, but the role and its skills are far more than a senior developer's.  An architect's number one skill, is their communication skills.  In my opinion as an industry we need to formalise roles, and the architect role is paramount and akin to the building industry's architect in responsibility but more similar to a head-chef in activities.

References and more information

Scott Ambler on User Stories
Scott Ambler on Architecture in Agile
IDesign.Net
http://www.idesign.net/articles/agile_and_the_architect.htm
Able Architecture Part 1: 
Able Architecture Part 2: 

Fitting Security into the Agile SDLC

Microsoft have some good guidance on this

http://www.microsoft.com/security/sdl/discover/sdlagile.aspx

My personal opinion on this, is that it does look good, however teams starting out with Agile processes should get the basics right first.  I've seen companies rush through Scrum implementations too many times with little or no training and change management, only to fail.  Once you have the basic Scrum process running smoothly concentrate on implementing good up front design practises, including security.

12 Principles of Agile Software Development

I came across this great presentation on some founding goals for Agile Software Development. Its great to use these principles to base all decisions and processes.
12 principles for Agile Development

View more presentations from Julien Henzelin

Why agile software development is like teenage sex

Alexandre de Pellegrin writes: 

Today, Romain Gauthier (OCTO Technology, for the moment) sent me a funny post on Agile development. I really like Agile methods but it's so funny that I had to paste it here :

Why is agile software development like teenage sex?

• It's on everyone's mind all the time.
• Everyone is talking about it all the time.
• Everyone thinks everyone else is doing it.
• Almost no one is really doing it.

The few who are doing it are:

• doing it poorly
• hopeful it will be better next time
• not practicing it safely

Source:

http://javacolors.blogspot.com/2009/02/why-is-agile-software-development-like.html

The need for transparency in software development

David Cooksey talks about the Agile principle of transparency specifically in software development.

http://blog.thycoticsolutions.com/2011/04/14/the-agile-virtue-of-transparency/

Agile Development Principles To Live By

I'm currently reading an excellent book by Robert Martin (aka Uncle Bob) Agile Principles Patterns and Practises in C# (Prentice Hall).  I highly recommend it.
There is a fantastic list of agile principles in chapter 1, here's my synopsis of Robert's list.

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
   The smaller the deliverable of software the higher the quality and the less risk of non-delivery.  The more often you can deliver the higher the overall quality. All deliverables are production quality code.
   
2. Welcome changing requirements, even late in the development. Agile processes harness change for the customer's competitive advantage. Embrace change, change is good, it demonstrates we have learnt more about the customer's needs. An agile development team should focus on software the is easy to change and maintain, simplifying the adaptive process. In my opinion I would also add to this, communicate the cost of change effectively to stakeholders as well; change does cost and whimsical change should be transparent and visible.
   
3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale. Each delivery should satisfy some customer need.
   
4. Business people and developers must work together daily throughout the project. In order to be agile there must be frequent interaction and assessment. A software project is not a fire and forget weapon.
   
5. Build projects around motivated individuals. Give them the environment and support they need and trust them to get the job done. People are always the most important factor in any software project, everything else is secondary and will not compensate for the wrong people on a project.
   
6. The most efficient and effective form of conveying information to and within a team is face-to-face conversation. Speech is a far richer and more concise form of communication over written forms, more information is imparted more quickly and with more detail. Documentation should be created incrementally, but only when NEEDED.
   
7. Working software is the primary measure of progress. The project is 30% done when 30% of all features needed are finished and delivered.
   
8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.  It is not a 100metre sprint, its more like a marathon. Care should be taken not to over commit.
   
9. Continuous attention to technical excellence and good design enhances agility. High quality is the key to speed, badly written code generally leads to a do-over.  All team members must believe in and be committed to high quality and excellence. They do not create messes that they promise to fix next month.
   
10. Simplicity - the art of maximising the amount of work not done is essential.  Take the simplest path that is consistent with the team and project goals. They do not put a lot of importance on solving tomorrow's problems, nor do they try and defend against them today. Rather focus on writing clear quality code that is easy to change.
    
11. The best architecture, requirements, and design emerges from self organising teams. An agile team is a self organising team. Responsibilities should not be handed out, rather let the team decide and come to a consensus on decisions or let them choose who is best able to make these decisions if they feel they cannot contribute.  All team members should feel they have influence over decisions made.
    
12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly. The agile team knows that its environment is constantly changes and the best processes from yesterday may not still be best.
    

This post is meant to be a quick reference of valuable Agile information.

References:

• Agile Principles Patterns and Practises in C# (Prentice Hall)
• www.controlchaos.com
• Clarus Consulting (Thanks to Ed and Bruce for all their valuable advice and help over the last few months).