Back in the initial post in this series, I described a core problem in Agile As She Is Practiced: we make time for certain things, but not for design. A standard sprint has time for grooming, planning, development, and retrospective. Some of these things overlap (for example, you’re probably grooming stories for the next sprint during your current sprint) and your sprint process may be a little different, but I think this is a fair summary. During grooming, the developers and product owner tease out business rules and think about what they need to do. Then, in planning, developers commit to a specific workload for the upcoming sprint. They follow through during the development phase, and then everybody gets back together for a retrospective in which developers and product owner describe what went well, what did not go well, and what the team may wish to change in the upcoming sprint. Note that during this time, we never set any time aside for design. In fact, looking at a scrum event listing or basics of a scrum sprint, there’s no explicit room for design.
As far as I can tell, the lack of explicit design time is on purpose, and to the extent that this is true, it weakens the power of Agile development. I understand that a core Agile principle is “don’t over-design things,” based on the YAGNI principle , but this principle has its limitations. If the cost of change is low, YAGNI is absolutely the right policy; as the cost of change increases, foresight becomes more important. There are things that Agile aficionados propose to reduce cost of change—such as test libraries, continuous integration, high levels of code coverage, programming against interfaces, etc.—but even at the limit, I will argue that there needs to be time for research and design, for prototyping, and for long-term design work outside of a single sprint. It isn’t enough simply to wait until a particular feature is on a sprint before doing any design work, and it’s not extremely common that you get an opportunity to make sweeping design changes within the guise of a single story; instead, we tend to accumulate technical debt and pay it off in a hap-hazard fashion, hopefully improving our product at the margin but rarely getting beyond that. A big part of the problem is that the way we defined sprints above prevents us from thinking any deeper than “the current sprint” or maybe “the current and next sprint or two,” preventing us from thinking about the meta-program layer—that is, the layer(s) of code which developers use to implement features. This meta-program layer needs maintenance and planning just like any other code, but changes at that level are much more sweeping than changing a function in a module to satisfy a product story. Instead, we’re looking at architectural decisions which need foresight, and trying to lump those into a single sprint can lead to catastrophe.
This does not mean that we need to throw away the idea of sprints; rather, we need to be more explicit about the tools within a sprint context. By this, I mean that we need to make some time for long-term design, just like we do for requirements gathering (grooming), QA, and review. Design is not development, nor should it be considered something which reduces a developer’s focus factor; rather, design is a process which runs concurrently with development. We can iterate on our designs while working on development tasks, and good Agile practices include the use of research spikes, so practitioners already understand these concepts. If we model design as a series of timeboxed research spikes, we can fit design into the Agile sprint model in the Development section.
If you’re an architect, lead developer, project manager, or somebody with serious clout on your Agile team, getting to a point in which design is a first-class citizen is viable. But what happens if you’re a more junior developer who understands the need for longer-term design but can’t convince the higher-ups of this? In that case, you might need to do design on the sneak. The easiest way to sneak in design is to tie it to product stories. As you work on Feature X, add a half hour’s extra time to find the pain points in feature development and write them down and try to be as specific as you can. As you start collecting a journal of problems, you will start seeing common issues. Once you have details on a couple of these issues, take some time to dive into the problem. Once you figure out a better design, try to iterate on it locally and work through that version 0.5 iteration on your own. After a while, you’ll end up with a good 1.0 design and that’s the time to make a proposal up the chain of command. You might need to think more marginal than I’ve recommended here; unless you can get serious buy-in on a radical design change, take advantage of the ratchet effect: you already have a final model in mind, so look at ways of getting to that final model piecemeal. That way, you can potentially persuade team leaders which may not be willing to spend the time for design and implementation as a block but might be willing to spend time for design and implementation in smaller segments. Although piecemeal design changes are not always ideal, they are a good gateway for more serious design. Once you have a few successful design changes under your belt, team members and higher-ups are more likely to take your design changes seriously and expect them to be high-quality.
In the end, getting fundamental design to be a first-class citizen in Agile development has its difficulties. For the most part, it seems like the belief is that design is something you amortize in the focus factor or work outside the confines of a sprint. Sometimes, teams plan ahead using research spikes, but even that tends to be focused on new development rather than cleaning up technical debt. Given these systemic difficulties, the best way to keep an eye on long-term design is to incorporate it in as many development processes as you can, iterating on the design until it is strong enough to groom as a story.