DevOps as an Aspiration

Yesterday I gave a tech talk to my colleagues about how we are using Bitbucket and Jenkins for continuous integration. The talk focused on the things way Bitbucket could be used to integrate with Jira, and how we've set up Jenkins to build and deploy the various branches. It had the title "Towards DevOps".

Towards is the right word to describe what we are doing. The ideal of Dev to Production in 60 minutes is unachievable for many reasons, but the kinds of things that enable such operations are still useful.

So for us, we wanted to use the setup to improve quality control. Successful compilation and running of unit tests is a necessary step before releasing code, and linking Bitbucket to Jenkins gives us that seamlessly.

We also wanted to automate deployments, so that we can go into the Test environment at any time. We could use Jenkins for this and have the operation launched with the click of a button (actually 2 clicks because we have a customisable deployment script for choosing branch, build, and environment.)

This setup is limited to the lower environments and is far from complete. For one, DB scripts are completely manual and reliant on one person. But we are looking to fix that, just as we are looking to include Change Management for automated deployments into the higher environments.

Whatever we end up with will not overturn the enterprise setup. But it has already put us in a better position than we were before, and given us a target on how to further improve things. The corporate culture is not an excuse for making things better.

The Limitations of the Scrum Master

grant me the serenity to accept the things I cannot change, courage to change the things I can, and wisdom to know the difference

The role of scrum master exists to ensure the scrum process can flow smoothly. The role handles the administrative burden of the process, and provides a dedicated resource to ensure the process runs smoothly.

It goes without saying the wider environment the development takes place in is beyond the control of the team itself. It may affect the team's efficacy, but it will do no good to try to fight against it.

Similarly, a scrum master is not HR. Interpersonal issues may be getting in the way of the process, but a scrum master isn't the person to deal with them.

And the scrum master is not the product owner. The responsibility of what the product will be doesn't rest with the scrum master, even though that vision directs the process.

Injecting oneself into the totality of the process is a surefire path to stress and futility in the wider process. There are plenty of things the scrum master can influence, and knowing that is vital for successful delivery.

Listen to and Address the Concerns

If there is resistance to change, there may be good reasons for it. The way things are currently done have reasons embedded in them, even if those reasons are not immediately discernible.

When proposing a new way forward, the best thing I've found to do is to ask what the concerns will be with the new approach. At that point, a discussion can be had.

In the case of our proposed automation of testing deployments, the DBA's concern was around the state of the databases. Presently, changes in the Development environment can come from anyone. But by deploying to test manually, the DBA can see what's happening in that space. That way there are no surprises in Acceptance.

So if we are going to automate, that's the concern we need to address. If the DBA would be happy to get involved during the development - either writing the scripts or code reviewing the written scripts - then the concern is assuaged.

It would have been problematic to go ahead without getting the relevant stakeholders on board. It's far better to have them as part of the decision and the process. So to do that, listen to them.

Faster Horses

"If I had asked people what they wanted, they would have said faster horses." - Henry Ford

Collaboration is a tricky business. On the one hand, we need to deliver what the customer wants. On the other, the customer may not know what they want until they see it.

As agile developers, we are meant to deliver value to the customer. Sometimes that's going to be by doing what they want. At other times, it's going to be anticipating it.

The lean value of seeing the whole is vital to adopt. If the development team understand the domain, it gives them the opportunity to see things the customer doesn't. That it turn leads to a higher value product.

This is why collaboration is so important. It allows early and frequent feedback on what features are being delivered. This takes away the surprise element, as well as giving the opportunity for collaboration.

Development over time involves a deep immersion in a problem space. Ideas will spring up as the problem space is explored, and a good agile team should be ready to respond when this happens.

There will Always be Resistance to Change

The complaint I have heard the most often any time myself or others in my team have pushed forward with a new idea is "but that's not the way we do things". The appeal, when divorced from the obvious truth of such a statement, is that what we currently do works. So what advantage would we get from changing?

This is a question we always need to ask ourselves, because being agile is not change for the sake of change. There is usually an identification of a deficiency, inefficiency, or bottleneck that the suggestion tries to address.

We have a deficiency now in that the database changes are wholly separate to the code changes they align with. When we deploy, we have to coordinate with the DBAs to make sure they run their scripts at the right time. This is fine when they are available, but a problem when they aren't.

But even if they were always available, it's a coordination problem that can be easily solved by scripting. As it stands, anyone can deploy to Test whenever there is a need to do so. And there's currently no obvious way beyond the developers knowing what changes are going in as to whether the DBA is required.

Evidently this is a fairly common problem because there are multiple products that solve it by introducing DB versioning. The main resistance is getting the DBAs themselves on board.

They are correct in that it's a cultural change, moving away from a system that works. But it's a necessary change because the system as it works has obvious (and sometimes painfully pressing) limitations. We want this change because we can and need to do better than what we have.

It's good to have a reminder that development is largely people, so we should be mindful of how people can react to being asked to change.

Pilot Technical Ideas in Sprints

Every fortnight, all the technical people in our section meet for a forum - our chance to bring up any technical issues or things we've learnt that might be of interest to others. This has mixed results, but occasionally it throws up a good idea that the team can pursue.

The general problem with ideas is how easily they can be argued for or against. Of they were implemented, on the other hand, their merit is demonstrable.

Therein lies the difficulty for a scrum team. There's always things that could be potentially of value, but they have to go up against that time being used for something else. The forum becomes an exercise in implicit permission - we put the idea forward for quality control.

If the idea has tacit merit and implicit consent, then it's something that can be put forward as part of the development. Since sprints are about providing business value, the idea should in some way contribute to business value.

One example floating around at the moment is moving to a microservice architecture. We have our reasons for doing this, but it will still take time to set up and deliver. But doing it as part of a particular user story means we can realise that change and assess its merits. It serves the business, as well as allowing for the developers to try out better ways of delivery.

Having One Big Target

I've been in sprints where there are lots of little things to be done, as well as sprints where there's just one thing that has to happen. My experience is having one big target is much better than lots of little targets.

The reason, I think, is that one big target lacks the context switching that going from target to target. It allows all people to focus on one problem and gradually work to that.

A big target is not an ill-defined target. The same rules apply as apply in all scrum - it should be coherent enough that it can be worked on. Anything less and a big target has a chance of failure.

A big target should still be broken down to small targets. But the package of targets together provides a context to the work. A package of disparate targets has their own context, with the developers needing to adopt that mindset to begin work. Smaller tasks for a greater whole do not have that problem.

The Utility of a WIP Limit

One of the advantages of using a Kanban board for tracking scrum is that it identify bottlenecks in the process. The Kanban board has an interesting way to stop this - putting a limit on the amount of tickets that can be in a column at any given time.

Putting a WIP limit prevents too much work piling up. The team ought to be doing what it can to keep the flow going smoothly. It helps eliminate bottlenecks by preventing issues from piling up.

It also serves as a reminder to the team of collective commitment. That we are all in it together, and blockers affect everyone. Tickets stuck in analysis, development, or testing all have the same outcome - that feature doesn't get out. And given business value determines ranking order, the tickets in WIP are more valuable than tickets that are on the backlog.

The Nature of Agile Testing

"The focus of agile development is producing high-quality software in a time frame that maximizes its value to the business. This is the job of the whole team, not just testers or designated quality assurance professionals. Everyone on an agile team gets “test-infected.” Tests, from the unit level on up, drive the coding, help the team learn how the application should work, and let us know when we’re “done” with a task or story. An agile team must possess all the skills needed to produce quality code that delivers the features required by the organization. While this might mean including specialists on the team, such as expert testers, it doesn’t limit particular tasks to particular team members. Any task might be completed by any team member, or a pair of team members. This means that the team takes responsibility for all kinds of testing tasks, such as automating tests and manual exploratory testing. It also means that the whole team thinks constantly about designing code for testability." [Crispin, L. and Gregory, J., Agile Testing, p.15]

Our JIRA Scrum Board Columns

• TODO
  It holds the list of all unassigned tickets for the sprint. The expectation is that for the tickets to get into this column, they've had enough refinement done before the sprint that they can be worked on with minimal further clarification. The tickets are ordered vertically by business rank.


• Development
  For when the ticket is being actively developed. The ticket is assigned to the person doing the work, and will remain in this column until the work is complete. If the work being done is Java, then branches are made off the ticket. Tickets with branches that have a pending pull request get coloured orange, while tickets with merged pull requests are green.


• Testing
  When the development work is ready to test, tickets are moved into Test. It will be up to whoever in the testing team is available to assign themselves the work. If a severe defect is found, then a sub-task is created from the ticket. If the defect is not a problem for the functionality, then a new ticket is created and put on the backlog.


• Product Owner Review
  The product owner gets final say over whether a unit of work is done. This column is the for the final verification by the product owner that the ticket meets the business expectation.


• Done
  The product owner moves the cards to this column when they are satisfied the story is complete. When a sprint ends, any tickets not in this sprint will be moved either to the next available sprint, or into the backlog.

Dude's Law: Value = Why / How

The one of the consequences of agile development is the focus on a minimum viable product. The product owner sets the work by business value, so the user stories with the highest value are always worked on first.

The business need for a user story (or the amount of work done on a user story) does depend on the ability for the development team to deliver that story. A really complex piece of work might be the best thing from the business perspective, but is it more important than multiple smaller tickets? This is where Dude's Law comes in.

Dude's Law: Value = Why / How

If something takes a lot of effort, that's effort that could be expended on something else. So if a user story is going to take a lot of effort, it's value ought to be weighed up against what alternatives could be done instead.

The inverse holds for where the how is minimal. Changes that are trivial to implement can high value even if they are not priorities from a business perspective. But a user story that's difficult to implement and of low business importance holds very little value.

Applying Dude's Law gives a way to quantify what tasks ought to be achieved in a sprint. Like everything, the code should be written such that if the project were to be shut down tomorrow, as much business value as possible should have been added until that point.

JIRA Tip - Use Columns, Not Workflows

Before we started using JIRA, we had Kanban boards made of electrical tape on the walls. They had 4 columns: Backlog, In Progress, Done, and Done & Done. Three of those 4 columns were self-documenting, but the In Progress column had cards covered in post-it notes for what work was being done on it and who was doing that work.

When we moved to JIRA, the first thing we did was customise the board. By default, the standard scrum board comes with three columns (and corresponding statuses) - TODO, In Progress, and Done. We've expanded that to 5. We tried including extra statuses for each column, but JIRA couldn't visually transition between statuses on the same column.

As we set up statuses, we didn't put a workflow on them. Any card can be changed to any other status at any time.

One reason for this is that the kinds of tasks don't always have the same workflow. We need the flexibility to go from development to done, or go back from product owner review. The default catch-all In Progress does this well but lacks the explicit documenting of where the process is up to that we ate trying to achieve with the extra columns.

Another reason is that the columns are visually documenting of the general process. Moving the flow from left to right is natural, so it's done by convention and doesn't need to be forced.

The final reason is that all the people in the project are professionals, such that there's no need to constrain their use of the tool. JIRA is for the team to use as a guide to development, so it should be up to them how they make the best of it. There's just no need to constrain the team's agility.

The Value of Values

The most attractive thing about any agile methodology is that the processes are am embodiment of values. What makes agile agile is the ability to change those processes if and when doing is what's best for the project.

Scrum, XP, and Lean all have their own ways of doing things - but the adoption of those practices doesn't embody agility.

For example, we are using the scrum methodology. This means designated sprints, daily stand-ups, planning, and retrospectives. But these aren't what makes us agile. It's the Scrum values - commitment, openness, courage, respect, and focus - that makes us agile.

The values that the different agile methodologies hold are also able to be utilised without adoption of the practices. The idea of seeing the whole is a value of lean, but it's just as useful a value to hold to while doing scrum.

The advantage of values is being able to respond and change to difficult situations without falling back on the process. Values are our metric by which change is guided and assessed. They are the why of the processes that allow us to know what those processes are for.

Processes without the values can be followed and confer the same advantages as a team who have internalised the values. But knowing the processes without the values makes it harder to realise those advantages when the processes become burdensome or sub-optimal. An agile team has the ability to better utilise the processes than one who follow the processes without the values guiding them.

Identifying and Eliminating Bottlenecks

"Don't be irreplaceable, if you can't be replaced, you can't be promoted." (from Dilbert's Laws of Work)

The easiest way to identify a bottleneck is to see what happens when time is a factor. If someone (or something) is critical to an operation, then things don't happen. That is obviously a less-than-ideal way to find where a bottleneck is.

Bottlenecks usually present themselves long before a crisis, but aren't seen as critical at the time. They are usually things that only occasionally matter, so general availability is usually enough.

Unusual situations aren't unavoidable situations. Taking steps to eliminate the bottleneck is sensible risk management. It is much better than the alternatives, such as cancelling leave or letting deadlines slip.

For those bottlenecks that are part of everyday development, they should be mitigated and eliminated as quickly as possible. In our team, only a few people had the ability to do deploys. This was liveable until those people went on leave. So we implemented automated deploys from Jenkins so that would never be a problem again.

Chickens and Pigs

A Pig and a Chicken are walking down the road.
The Chicken says: "Hey Pig, I was thinking we should open a restaurant!"
Pig replies: "Hm, maybe, what would we call it?"
The Chicken responds: "How about 'ham-n-eggs'?"
The Pig thinks for a moment and says: "No thanks. I'd be committed, but you'd only be involved."

Most people who are involved in a project are not committed. They are chickens. And while they make decisions that affect the project, they are not committed in the same way as the development team are.

The development team are the ones who are responsible for what is delivered. They are pigs. Their actions directly affect the project.

A chicken making decisions on behalf of the pigs can adversely affect the team's ability to do their job. As the chicken has no skin in the game, their decisions are divorced from the consequences.

A pig who acts like a chicken is also problematic. If they are not committed to delivery, their involvement detracts from the team's collective ability to deliver.

When Algorithms Test

Code needs to be tested. Sometimes that testing is best done by an algorithm.

We recently had a user story that involved removing particular characters from the application number string. The purpose was to address a readability issues - at a glance certain characters and numbers are indistinguishable, so removing the offending characters from the string they were generated from. A simple enough coding exercise.

This card, however, sat in the testing column for a long while. It didn't need to, as we had built a unit test that did what a regular tester would do - throw lots of data at the generation and see whether those offending characters were present.

An automated test, in this case, could do what a human could not: quickly generate and accurately assess the functionality. It's checked 1000 times every time the code is compiled - overkill as far as what's needed to be tested, but that overkill is computationally negligible.

Writing the unit test first (TDD) gives the developer the target to aim for, as well as creating the before and after test artefacts. At the very least, the unit tests results could supplement the functional spot-checking that would constitute normal testing for such a story.

Writing code that verifies functionality is not simply for helping developers to write better code, but can be the best tool at the disposal of the testers for doing testing. It's leveraging technology at tasks it excels, which is what we ought to be using technology for.

Book Recommendation: The Nature of Software Development

Development has more than enough to keep anyone occupied on immediate concerns. But it's worth standing back in order to reflect on what it is we are doing.

The Nature of Software Development by Ron Jeffries can best be seen as an expansion on the agile manifesto. Like the manifesto, it's high level but gets right to the point. Like the manifesto, it's concise but profound. And like the manifesto, the aim is to identify what makes for successlooks like.

What we are doing when building software is trying to add value. The word can mean different things in different contexts, but in this case it is a strength rather than a weakness.

We build software because the software will serve a purpose. We should focus on delivering that which serves that purpose, and not waste time on features that are superfluous to that.

We should develop such that there will be value if the project winds up tomorrow. The code should always be ready to deploy. We should fix problems as we go along because that makes development quicker in the long run.

There's much more to the book than I'll cover here. It's a short book, and conversational in tone, and that fits with the message the cook conveys. Deliver what adds value, and don't waste time with filler that doesn't. This book is very valuable.

The Heisenberg Product Owner

"We're done when I say we're done" - Heisenberg

From the development's team perspective, product owner sign off is the one stage of the development process they can't control. Ideally, this should a minimal step - all the work has been done before this point. But the product owner can always say no.

As scrum masters, our job is to try to sort this out, as this is indicative of failure somewhere along the way. The product owner is a part of the team, and thus involved in the day-to-day delivery of the project. So the question is where the failure is taking place.

There are many reasons within the team this could happen, and many ways a team could deal with it. One thing to watch out for, however, is a product owner who simply cannot make up their mind.

If this is the case, it's up to the scrum master to intervene. Simply acquiescing to the product owner is not a good idea as it breaks the flow of development. It also erases the trust and enthusiasm of the development, as their work is undermined.

If new aspects of a user story are required, they are to be turned into new user stories, prioritised, and put on the backlog. This gives a path to improvement that ensures those features are added, while not holding up the flow of development. The development team then doesn't have to worry about moving targets or feature creep.

Similarly, if user stories aren't being fleshed out enough, then there's grounds for spending more time in backlog refinement. Or, perhaps, the developers can swarm with the product owner during development.

As scrum masters, it's our job to ensure the process is running smoothly, and that includes how the product owner works with the team.

Following the Boy Scout Rule

"Always leave the campground cleaner than you found it."

In all my years as a software developer, the only times I've ever gotten to start from scratch is when I've worked on personal projects. And even then, starting from scratch is a last resort.

My experience of being a programmer is working in someone else's architecture. I get to expand it out, fix it up, and sometimes refactor it away. And each time I end up having the same question - why is it built that way? And each time there's the same answer - it's because the code needed to be delivered quickly.

It feels at times that all we can do as programmers is try not to make things worse. One thing I've done is tried to fit in with the existing way of doing things. That way the code is consistent for those who maintain it in the future. But it doesn't solve the problem, merely prevents it from getting any worse.

The answer is meant to be refactoring, though it's really not an answer for fragile code. It's a sad irony that the code in most need of refactoring is the code least able to be refactored...

Any complex system is going to seem impenetrable at first glance, but there are ways to improve it. A system doesn't need to be overhauled to be fixed, and something as simple as tidying up the code that's being worked on is enough to make things better.

For this project, we have pretty abysmal unit testing. So one rule we've set for ourselves is to put unit tests in wherever we find ourselves enhancing the code. Coverage is still woeful, but it is less woeful than it was a month ago. Another thing is to make sure no commit has commented-out code.

Code doesn't get to a messy state by a single act, nor can it be fixed in one. The complex mess that is the average codebase (at least average from my experience) is the way it is because of a long path to that state. Making things better is a similarly long path, but thankfully one that be addressed as part of enhancing a system. Fragility is quantitative, not qualitative.

Keeping Track of Tasks with SplenDO

A few jobs back, my monitor would be covered in post-it notes. It was my way to keep track of my tasks, or my progress in particular tasks. It's a practice I wish I kept up, and I'm disappointed I let the lack of office-supplied stationary get in the way of doing it. For my development, it was the time I felt the most productive.

Between Notepad, Sticky Notes, and actual notepads, I've never managed to find a sustainable suitable replacement. Memory only goes so far.

For the last few weeks, I've been trialling out the SplenDO on my mobile. This is what I found so far.

The useful:

• Putting rooms on meetings - Knowing when meetings are is useful enough (though mostly taken care of by Outlook), but having the place of the meeting at your fingertips is invaluable.
• Tasks I'd normally delay - Some things I know I tend to procrastinate on, such as anything involving chasing up people. Having it as a task (and one that becomes overdue) compels me to get it out of the way sooner rather than later.
• Temporal tasks - For the occasional task that has a deadline, it's very useful to have a reminder close to when it's needed.
• Tasks that come up in conversation - So much of what needs to happen at work is stated during the course of conversations. This alleviates the need to carry a notepad and pen everywhere, or trying to remember every action item after the conversation is over.

The not-so-useful:

• Habitual tasks - The daily stand-up happens at 9:45am each weekday. While I have the tasklist to remind me of this, it doesn't do much to have it as a task. I find I do it just to tick it off that it is done.
• Aspirational tasks - If the task isn't necessary, then putting it on the list becomes a source of frustration as the deadline passes.
• Setting times on non-temporal tasks - This is a case of over-planning. If a task is not time-dependent, it makes no sense to try to give a deadline to it.

The verdict:
As far as keeping organised goes, this is far superior to my old post-it note system. It's simple to use, and it's notifications keep it relevant. It's great for someone who always has a smartphone around.