Yesterday I gave a tech talk to my colleagues about how we are using Bitbucket and Jenkins for continuous integration. The talk focused on the things way Bitbucket could be used to integrate with Jira, and how we've set up Jenkins to build and deploy the various branches. It had the title "Towards DevOps".
Towards is the right word to describe what we are doing. The ideal of Dev to Production in 60 minutes is unachievable for many reasons, but the kinds of things that enable such operations are still useful.
So for us, we wanted to use the setup to improve quality control. Successful compilation and running of unit tests is a necessary step before releasing code, and linking Bitbucket to Jenkins gives us that seamlessly.
We also wanted to automate deployments, so that we can go into the Test environment at any time. We could use Jenkins for this and have the operation launched with the click of a button (actually 2 clicks because we have a customisable deployment script for choosing branch, build, and environment.)
This setup is limited to the lower environments and is far from complete. For one, DB scripts are completely manual and reliant on one person. But we are looking to fix that, just as we are looking to include Change Management for automated deployments into the higher environments.
Whatever we end up with will not overturn the enterprise setup. But it has already put us in a better position than we were before, and given us a target on how to further improve things. The corporate culture is not an excuse for making things better.
Wednesday, November 9, 2016
Tuesday, November 8, 2016
The Limitations of the Scrum Master
grant me the serenity to accept the things I cannot change, courage to change the things I can, and wisdom to know the difference
The role of scrum master exists to ensure the scrum process can flow smoothly. The role handles the administrative burden of the process, and provides a dedicated resource to ensure the process runs smoothly.
It goes without saying the wider environment the development takes place in is beyond the control of the team itself. It may affect the team's efficacy, but it will do no good to try to fight against it.
Similarly, a scrum master is not HR. Interpersonal issues may be getting in the way of the process, but a scrum master isn't the person to deal with them.
And the scrum master is not the product owner. The responsibility of what the product will be doesn't rest with the scrum master, even though that vision directs the process.
Injecting oneself into the totality of the process is a surefire path to stress and futility in the wider process. There are plenty of things the scrum master can influence, and knowing that is vital for successful delivery.
The role of scrum master exists to ensure the scrum process can flow smoothly. The role handles the administrative burden of the process, and provides a dedicated resource to ensure the process runs smoothly.
It goes without saying the wider environment the development takes place in is beyond the control of the team itself. It may affect the team's efficacy, but it will do no good to try to fight against it.
Similarly, a scrum master is not HR. Interpersonal issues may be getting in the way of the process, but a scrum master isn't the person to deal with them.
And the scrum master is not the product owner. The responsibility of what the product will be doesn't rest with the scrum master, even though that vision directs the process.
Injecting oneself into the totality of the process is a surefire path to stress and futility in the wider process. There are plenty of things the scrum master can influence, and knowing that is vital for successful delivery.
Monday, November 7, 2016
Listen to and Address the Concerns
If there is resistance to change, there may be good reasons for it. The way things are currently done have reasons embedded in them, even if those reasons are not immediately discernible.
When proposing a new way forward, the best thing I've found to do is to ask what the concerns will be with the new approach. At that point, a discussion can be had.
In the case of our proposed automation of testing deployments, the DBA's concern was around the state of the databases. Presently, changes in the Development environment can come from anyone. But by deploying to test manually, the DBA can see what's happening in that space. That way there are no surprises in Acceptance.
So if we are going to automate, that's the concern we need to address. If the DBA would be happy to get involved during the development - either writing the scripts or code reviewing the written scripts - then the concern is assuaged.
It would have been problematic to go ahead without getting the relevant stakeholders on board. It's far better to have them as part of the decision and the process. So to do that, listen to them.
When proposing a new way forward, the best thing I've found to do is to ask what the concerns will be with the new approach. At that point, a discussion can be had.
In the case of our proposed automation of testing deployments, the DBA's concern was around the state of the databases. Presently, changes in the Development environment can come from anyone. But by deploying to test manually, the DBA can see what's happening in that space. That way there are no surprises in Acceptance.
So if we are going to automate, that's the concern we need to address. If the DBA would be happy to get involved during the development - either writing the scripts or code reviewing the written scripts - then the concern is assuaged.
It would have been problematic to go ahead without getting the relevant stakeholders on board. It's far better to have them as part of the decision and the process. So to do that, listen to them.
Friday, November 4, 2016
Faster Horses
"If I had asked people what they wanted, they would have said faster horses." - Henry Ford
Collaboration is a tricky business. On the one hand, we need to deliver what the customer wants. On the other, the customer may not know what they want until they see it.
As agile developers, we are meant to deliver value to the customer. Sometimes that's going to be by doing what they want. At other times, it's going to be anticipating it.
The lean value of seeing the whole is vital to adopt. If the development team understand the domain, it gives them the opportunity to see things the customer doesn't. That it turn leads to a higher value product.
This is why collaboration is so important. It allows early and frequent feedback on what features are being delivered. This takes away the surprise element, as well as giving the opportunity for collaboration.
Development over time involves a deep immersion in a problem space. Ideas will spring up as the problem space is explored, and a good agile team should be ready to respond when this happens.
Thursday, November 3, 2016
There will Always be Resistance to Change
The complaint I have heard the most often any time myself or others in my team have pushed forward with a new idea is "but that's not the way we do things". The appeal, when divorced from the obvious truth of such a statement, is that what we currently do works. So what advantage would we get from changing?
This is a question we always need to ask ourselves, because being agile is not change for the sake of change. There is usually an identification of a deficiency, inefficiency, or bottleneck that the suggestion tries to address.
We have a deficiency now in that the database changes are wholly separate to the code changes they align with. When we deploy, we have to coordinate with the DBAs to make sure they run their scripts at the right time. This is fine when they are available, but a problem when they aren't.
But even if they were always available, it's a coordination problem that can be easily solved by scripting. As it stands, anyone can deploy to Test whenever there is a need to do so. And there's currently no obvious way beyond the developers knowing what changes are going in as to whether the DBA is required.
Evidently this is a fairly common problem because there are multiple products that solve it by introducing DB versioning. The main resistance is getting the DBAs themselves on board.
They are correct in that it's a cultural change, moving away from a system that works. But it's a necessary change because the system as it works has obvious (and sometimes painfully pressing) limitations. We want this change because we can and need to do better than what we have.
It's good to have a reminder that development is largely people, so we should be mindful of how people can react to being asked to change.
This is a question we always need to ask ourselves, because being agile is not change for the sake of change. There is usually an identification of a deficiency, inefficiency, or bottleneck that the suggestion tries to address.
We have a deficiency now in that the database changes are wholly separate to the code changes they align with. When we deploy, we have to coordinate with the DBAs to make sure they run their scripts at the right time. This is fine when they are available, but a problem when they aren't.
But even if they were always available, it's a coordination problem that can be easily solved by scripting. As it stands, anyone can deploy to Test whenever there is a need to do so. And there's currently no obvious way beyond the developers knowing what changes are going in as to whether the DBA is required.
Evidently this is a fairly common problem because there are multiple products that solve it by introducing DB versioning. The main resistance is getting the DBAs themselves on board.
They are correct in that it's a cultural change, moving away from a system that works. But it's a necessary change because the system as it works has obvious (and sometimes painfully pressing) limitations. We want this change because we can and need to do better than what we have.
It's good to have a reminder that development is largely people, so we should be mindful of how people can react to being asked to change.
Wednesday, November 2, 2016
Bad Unit Tests are Worse than no Unit Tests
Unit tests represent our best intentions as developers. They are non-functional code that take time and effort to write and maintain. But what they give us is confidence in the quality of the code, and confidence that modifying code won't break existing functionality.
Poorly written unit tests can make developers forget their best intentions. If unit tests are poorly written, involve heavy mocking, or are tightly coupled with the implementation, unit tests become an impediment.
Because effort has to go into understanding those tests, and the tests are divorced from their original context, it's understandable developers will choose to sacrifice the test rather than the time for maintaining it.
Because of the role unit tests play, it's 5 best to avoid this situation. This means spending time writing good unit tests and testable code.
Good unit tests are decoupled from the solution. They are functional without being prescriptive as to the details. It's not about making sure every line of code is covered, but making sure the code does what is expected.
To achieve good unit tests, it requires writing testable code. Unit tests shouldn't be an afterthought, but part of the development. If heavy mocking is required, that's a sign the code needs refactoring. If it's hard to see how unit tests could be made to test the code, that's another sign.
Unit tests are like any other aspect of a codebase. At all times, we are writing to be understood long after we have left the project. This is just as true for code where other developers come in cold, or even ourselves once we have context switched away from the problem at hand.
Poorly written unit tests can make developers forget their best intentions. If unit tests are poorly written, involve heavy mocking, or are tightly coupled with the implementation, unit tests become an impediment.
Because effort has to go into understanding those tests, and the tests are divorced from their original context, it's understandable developers will choose to sacrifice the test rather than the time for maintaining it.
Because of the role unit tests play, it's 5 best to avoid this situation. This means spending time writing good unit tests and testable code.
Good unit tests are decoupled from the solution. They are functional without being prescriptive as to the details. It's not about making sure every line of code is covered, but making sure the code does what is expected.
To achieve good unit tests, it requires writing testable code. Unit tests shouldn't be an afterthought, but part of the development. If heavy mocking is required, that's a sign the code needs refactoring. If it's hard to see how unit tests could be made to test the code, that's another sign.
Unit tests are like any other aspect of a codebase. At all times, we are writing to be understood long after we have left the project. This is just as true for code where other developers come in cold, or even ourselves once we have context switched away from the problem at hand.
Tuesday, November 1, 2016
Pilot Technical Ideas in Sprints
Every fortnight, all the technical people in our section meet for a forum - our chance to bring up any technical issues or things we've learnt that might be of interest to others. This has mixed results, but occasionally it throws up a good idea that the team can pursue.
The general problem with ideas is how easily they can be argued for or against. Of they were implemented, on the other hand, their merit is demonstrable.
Therein lies the difficulty for a scrum team. There's always things that could be potentially of value, but they have to go up against that time being used for something else. The forum becomes an exercise in implicit permission - we put the idea forward for quality control.
If the idea has tacit merit and implicit consent, then it's something that can be put forward as part of the development. Since sprints are about providing business value, the idea should in some way contribute to business value.
One example floating around at the moment is moving to a microservice architecture. We have our reasons for doing this, but it will still take time to set up and deliver. But doing it as part of a particular user story means we can realise that change and assess its merits. It serves the business, as well as allowing for the developers to try out better ways of delivery.
The general problem with ideas is how easily they can be argued for or against. Of they were implemented, on the other hand, their merit is demonstrable.
Therein lies the difficulty for a scrum team. There's always things that could be potentially of value, but they have to go up against that time being used for something else. The forum becomes an exercise in implicit permission - we put the idea forward for quality control.
If the idea has tacit merit and implicit consent, then it's something that can be put forward as part of the development. Since sprints are about providing business value, the idea should in some way contribute to business value.
One example floating around at the moment is moving to a microservice architecture. We have our reasons for doing this, but it will still take time to set up and deliver. But doing it as part of a particular user story means we can realise that change and assess its merits. It serves the business, as well as allowing for the developers to try out better ways of delivery.
Monday, October 31, 2016
Having One Big Target
I've been in sprints where there are lots of little things to be done, as well as sprints where there's just one thing that has to happen. My experience is having one big target is much better than lots of little targets.
The reason, I think, is that one big target lacks the context switching that going from target to target. It allows all people to focus on one problem and gradually work to that.
A big target is not an ill-defined target. The same rules apply as apply in all scrum - it should be coherent enough that it can be worked on. Anything less and a big target has a chance of failure.
A big target should still be broken down to small targets. But the package of targets together provides a context to the work. A package of disparate targets has their own context, with the developers needing to adopt that mindset to begin work. Smaller tasks for a greater whole do not have that problem.
The reason, I think, is that one big target lacks the context switching that going from target to target. It allows all people to focus on one problem and gradually work to that.
A big target is not an ill-defined target. The same rules apply as apply in all scrum - it should be coherent enough that it can be worked on. Anything less and a big target has a chance of failure.
A big target should still be broken down to small targets. But the package of targets together provides a context to the work. A package of disparate targets has their own context, with the developers needing to adopt that mindset to begin work. Smaller tasks for a greater whole do not have that problem.
Friday, October 28, 2016
Burnout
I describe the agile process as sustainable development. The idea is that development should be able to be continued indefinitely. This is a big change from the usual development approach of big pushes and lulls.
It's a old habit, and old habits die hard. We know we can push harder at times to get over a line. And as individuals, we suffer for doing that. We run the risk of burnout.
We recently jumped deep into a problem that was complex and time sensitive. It required a big effort to get it over the line. At the end of it we made it, but it came at a cost. The following sprint has been deflated, with tired developers, a lack of enthusiasm and focus, and inevitably more mistakes.
The scrum process is set up to avoid this. That we agree to take in what we can handle, and base what we take in future sprints from past performance. We learn what we can achieve from what we have achieved. So those moments we go above and beyond throws that out.
It's an achievement to deliver 4 weeks of work in 2. But if the cost is tired unmotivated developers, there had better have been a very good reason for doing so.
It's a old habit, and old habits die hard. We know we can push harder at times to get over a line. And as individuals, we suffer for doing that. We run the risk of burnout.
We recently jumped deep into a problem that was complex and time sensitive. It required a big effort to get it over the line. At the end of it we made it, but it came at a cost. The following sprint has been deflated, with tired developers, a lack of enthusiasm and focus, and inevitably more mistakes.
The scrum process is set up to avoid this. That we agree to take in what we can handle, and base what we take in future sprints from past performance. We learn what we can achieve from what we have achieved. So those moments we go above and beyond throws that out.
It's an achievement to deliver 4 weeks of work in 2. But if the cost is tired unmotivated developers, there had better have been a very good reason for doing so.
Thursday, October 27, 2016
The Utility of a WIP Limit
One of the advantages of using a Kanban board for tracking scrum is that it identify bottlenecks in the process. The Kanban board has an interesting way to stop this - putting a limit on the amount of tickets that can be in a column at any given time.
Putting a WIP limit prevents too much work piling up. The team ought to be doing what it can to keep the flow going smoothly. It helps eliminate bottlenecks by preventing issues from piling up.
It also serves as a reminder to the team of collective commitment. That we are all in it together, and blockers affect everyone. Tickets stuck in analysis, development, or testing all have the same outcome - that feature doesn't get out. And given business value determines ranking order, the tickets in WIP are more valuable than tickets that are on the backlog.
Putting a WIP limit prevents too much work piling up. The team ought to be doing what it can to keep the flow going smoothly. It helps eliminate bottlenecks by preventing issues from piling up.
It also serves as a reminder to the team of collective commitment. That we are all in it together, and blockers affect everyone. Tickets stuck in analysis, development, or testing all have the same outcome - that feature doesn't get out. And given business value determines ranking order, the tickets in WIP are more valuable than tickets that are on the backlog.
Wednesday, October 26, 2016
Whatever Tools for the Job
When I was a child, I was amazed at just how many tools were in my grandfather's workshop. I didn't even know what most of the tools did, let one why there would be so many.
My toolkit is virtual rather than physical. Like the craftsman, knowing which tool is right for the job and how to use it effectively is vital.
One recent task I had to do was to query our database and get some intelligible information from it. To do this, I used a combination of SQL Developer and Excel, taking advantage of features in both to perform the task. A DBA may have been able to solve it purely in SQL (just as I'd easily be able to solve it with a targeted Java function), but the combination of tools worked for me. I used SQL Developer and Excel for what each was good at doing.
Knowing multiple ways to solve a problem gives flexibility when issues arise. Sometimes a spreadsheet or even Notepad are the right tools for the job in the right circumstances.
My toolkit is virtual rather than physical. Like the craftsman, knowing which tool is right for the job and how to use it effectively is vital.
One recent task I had to do was to query our database and get some intelligible information from it. To do this, I used a combination of SQL Developer and Excel, taking advantage of features in both to perform the task. A DBA may have been able to solve it purely in SQL (just as I'd easily be able to solve it with a targeted Java function), but the combination of tools worked for me. I used SQL Developer and Excel for what each was good at doing.
Knowing multiple ways to solve a problem gives flexibility when issues arise. Sometimes a spreadsheet or even Notepad are the right tools for the job in the right circumstances.
Tuesday, October 25, 2016
The Nature of Agile Testing
"The focus of agile development is producing high-quality software in a time frame that maximizes its value to the business. This is the job of the whole team, not just testers or designated quality assurance professionals. Everyone on an agile team gets “test-infected.” Tests, from the unit level on up, drive the coding, help the team learn how the application should work, and let us know when we’re “done” with a task or story.
An agile team must possess all the skills needed to produce quality code that delivers the features required by the organization. While this might mean including specialists on the team, such as expert testers, it doesn’t limit particular tasks to particular team members. Any task might be completed by any team member, or a pair of team members. This means that the team takes responsibility for all kinds of testing tasks, such as automating tests and manual exploratory testing. It also means that the whole team thinks constantly about designing code for testability." [Crispin, L. and Gregory, J., Agile Testing, p.15]
Monday, October 24, 2016
Our JIRA Scrum Board Columns
- TODO
It holds the list of all unassigned tickets for the sprint. The expectation is that for the tickets to get into this column, they've had enough refinement done before the sprint that they can be worked on with minimal further clarification. The tickets are ordered vertically by business rank. - Development
For when the ticket is being actively developed. The ticket is assigned to the person doing the work, and will remain in this column until the work is complete. If the work being done is Java, then branches are made off the ticket. Tickets with branches that have a pending pull request get coloured orange, while tickets with merged pull requests are green. - Testing
When the development work is ready to test, tickets are moved into Test. It will be up to whoever in the testing team is available to assign themselves the work. If a severe defect is found, then a sub-task is created from the ticket. If the defect is not a problem for the functionality, then a new ticket is created and put on the backlog. - Product Owner Review
The product owner gets final say over whether a unit of work is done. This column is the for the final verification by the product owner that the ticket meets the business expectation. - Done
The product owner moves the cards to this column when they are satisfied the story is complete. When a sprint ends, any tickets not in this sprint will be moved either to the next available sprint, or into the backlog.
Friday, October 21, 2016
Dude's Law: Value = Why / How
The one of the consequences of agile development is the focus on a minimum viable product. The product owner sets the work by business value, so the user stories with the highest value are always worked on first.
The business need for a user story (or the amount of work done on a user story) does depend on the ability for the development team to deliver that story. A really complex piece of work might be the best thing from the business perspective, but is it more important than multiple smaller tickets? This is where Dude's Law comes in.
If something takes a lot of effort, that's effort that could be expended on something else. So if a user story is going to take a lot of effort, it's value ought to be weighed up against what alternatives could be done instead.
The inverse holds for where the how is minimal. Changes that are trivial to implement can high value even if they are not priorities from a business perspective. But a user story that's difficult to implement and of low business importance holds very little value.
Applying Dude's Law gives a way to quantify what tasks ought to be achieved in a sprint. Like everything, the code should be written such that if the project were to be shut down tomorrow, as much business value as possible should have been added until that point.
The business need for a user story (or the amount of work done on a user story) does depend on the ability for the development team to deliver that story. A really complex piece of work might be the best thing from the business perspective, but is it more important than multiple smaller tickets? This is where Dude's Law comes in.
Dude's Law: Value = Why / How
If something takes a lot of effort, that's effort that could be expended on something else. So if a user story is going to take a lot of effort, it's value ought to be weighed up against what alternatives could be done instead.
The inverse holds for where the how is minimal. Changes that are trivial to implement can high value even if they are not priorities from a business perspective. But a user story that's difficult to implement and of low business importance holds very little value.
Applying Dude's Law gives a way to quantify what tasks ought to be achieved in a sprint. Like everything, the code should be written such that if the project were to be shut down tomorrow, as much business value as possible should have been added until that point.
Thursday, October 20, 2016
JIRA Tip - Use Columns, Not Workflows
Before we started using JIRA, we had Kanban boards made of electrical tape on the walls. They had 4 columns: Backlog, In Progress, Done, and Done & Done. Three of those 4 columns were self-documenting, but the In Progress column had cards covered in post-it notes for what work was being done on it and who was doing that work.
When we moved to JIRA, the first thing we did was customise the board. By default, the standard scrum board comes with three columns (and corresponding statuses) - TODO, In Progress, and Done. We've expanded that to 5. We tried including extra statuses for each column, but JIRA couldn't visually transition between statuses on the same column.
As we set up statuses, we didn't put a workflow on them. Any card can be changed to any other status at any time.
One reason for this is that the kinds of tasks don't always have the same workflow. We need the flexibility to go from development to done, or go back from product owner review. The default catch-all In Progress does this well but lacks the explicit documenting of where the process is up to that we ate trying to achieve with the extra columns.
Another reason is that the columns are visually documenting of the general process. Moving the flow from left to right is natural, so it's done by convention and doesn't need to be forced.
The final reason is that all the people in the project are professionals, such that there's no need to constrain their use of the tool. JIRA is for the team to use as a guide to development, so it should be up to them how they make the best of it. There's just no need to constrain the team's agility.
When we moved to JIRA, the first thing we did was customise the board. By default, the standard scrum board comes with three columns (and corresponding statuses) - TODO, In Progress, and Done. We've expanded that to 5. We tried including extra statuses for each column, but JIRA couldn't visually transition between statuses on the same column.
As we set up statuses, we didn't put a workflow on them. Any card can be changed to any other status at any time.
One reason for this is that the kinds of tasks don't always have the same workflow. We need the flexibility to go from development to done, or go back from product owner review. The default catch-all In Progress does this well but lacks the explicit documenting of where the process is up to that we ate trying to achieve with the extra columns.
Another reason is that the columns are visually documenting of the general process. Moving the flow from left to right is natural, so it's done by convention and doesn't need to be forced.
The final reason is that all the people in the project are professionals, such that there's no need to constrain their use of the tool. JIRA is for the team to use as a guide to development, so it should be up to them how they make the best of it. There's just no need to constrain the team's agility.
Wednesday, October 19, 2016
The Value of Values
The most attractive thing about any agile methodology is that the processes are am embodiment of values. What makes agile agile is the ability to change those processes if and when doing is what's best for the project.
Scrum, XP, and Lean all have their own ways of doing things - but the adoption of those practices doesn't embody agility.
For example, we are using the scrum methodology. This means designated sprints, daily stand-ups, planning, and retrospectives. But these aren't what makes us agile. It's the Scrum values - commitment, openness, courage, respect, and focus - that makes us agile.
The values that the different agile methodologies hold are also able to be utilised without adoption of the practices. The idea of seeing the whole is a value of lean, but it's just as useful a value to hold to while doing scrum.
The advantage of values is being able to respond and change to difficult situations without falling back on the process. Values are our metric by which change is guided and assessed. They are the why of the processes that allow us to know what those processes are for.
Processes without the values can be followed and confer the same advantages as a team who have internalised the values. But knowing the processes without the values makes it harder to realise those advantages when the processes become burdensome or sub-optimal. An agile team has the ability to better utilise the processes than one who follow the processes without the values guiding them.
Scrum, XP, and Lean all have their own ways of doing things - but the adoption of those practices doesn't embody agility.
For example, we are using the scrum methodology. This means designated sprints, daily stand-ups, planning, and retrospectives. But these aren't what makes us agile. It's the Scrum values - commitment, openness, courage, respect, and focus - that makes us agile.
The values that the different agile methodologies hold are also able to be utilised without adoption of the practices. The idea of seeing the whole is a value of lean, but it's just as useful a value to hold to while doing scrum.
The advantage of values is being able to respond and change to difficult situations without falling back on the process. Values are our metric by which change is guided and assessed. They are the why of the processes that allow us to know what those processes are for.
Processes without the values can be followed and confer the same advantages as a team who have internalised the values. But knowing the processes without the values makes it harder to realise those advantages when the processes become burdensome or sub-optimal. An agile team has the ability to better utilise the processes than one who follow the processes without the values guiding them.
Tuesday, October 18, 2016
Bitbucket for History
One really useful Bitbucket feature is being able to view the history of a file in a browser. This is useful for tracking changes in any given commit.
When combined with Blame, it's a great diagnostic tool for code history. It's easy to go back through code to the right commit, even when subsequent commits have occurred. It's just a matter of switching to the right commit and checking Blame again.
When combined with Blame, it's a great diagnostic tool for code history. It's easy to go back through code to the right commit, even when subsequent commits have occurred. It's just a matter of switching to the right commit and checking Blame again.
Monday, October 17, 2016
Programmatic vs Functional Testing
From a testing perspective, it makes no difference how a programmatic solution is implemented if the functionality remains the same. What matters is how the system behaves, and not what goes into that behaviour.
Functional testing ought to treat the system as a black box. This decouples the testing suite from the solution, allowing testers to focus on the quality of the software rather than getting bogged down in the details of how the software works.
This also gives the programmers the freedom to refactor the solution without causing extra rework on the testing suite.
The question, always, is what value does it provide? The cost of testing the programmatic details not only has the time expense of the testing, but the additional cost of not focusing that effort elsewhere.
Functional testing ought to treat the system as a black box. This decouples the testing suite from the solution, allowing testers to focus on the quality of the software rather than getting bogged down in the details of how the software works.
This also gives the programmers the freedom to refactor the solution without causing extra rework on the testing suite.
The question, always, is what value does it provide? The cost of testing the programmatic details not only has the time expense of the testing, but the additional cost of not focusing that effort elsewhere.
Friday, October 14, 2016
Git Tricks: Working in the Wrong Branch
One of the tricky things about Git is getting into the mindset that a branch is a snapshot of files rather than a diff of a file system.
Another tricky thing is remembering to switch before starting to code.
One thing that's tripped up a few of my team members (and myself too!) is what to do when coding in the wrong branch. If they haven't committed, this is easily solved:
Another tricky thing is remembering to switch before starting to code.
One thing that's tripped up a few of my team members (and myself too!) is what to do when coding in the wrong branch. If they haven't committed, this is easily solved:
Simply switch between branches. The uncommitted files will be unaffected.The main thing to watch out for is when a changed version of that file is on the branch being switched to. At that point, manual merging needs to happen. But for the general case, it's as easy as pointing your local repository at the desired branch.
Thursday, October 13, 2016
Identifying and Eliminating Bottlenecks
"Don't be irreplaceable, if you can't be replaced, you can't be promoted." (from Dilbert's Laws of Work)
The easiest way to identify a bottleneck is to see what happens when time is a factor. If someone (or something) is critical to an operation, then things don't happen. That is obviously a less-than-ideal way to find where a bottleneck is.
Bottlenecks usually present themselves long before a crisis, but aren't seen as critical at the time. They are usually things that only occasionally matter, so general availability is usually enough.
Unusual situations aren't unavoidable situations. Taking steps to eliminate the bottleneck is sensible risk management. It is much better than the alternatives, such as cancelling leave or letting deadlines slip.
For those bottlenecks that are part of everyday development, they should be mitigated and eliminated as quickly as possible. In our team, only a few people had the ability to do deploys. This was liveable until those people went on leave. So we implemented automated deploys from Jenkins so that would never be a problem again.
The easiest way to identify a bottleneck is to see what happens when time is a factor. If someone (or something) is critical to an operation, then things don't happen. That is obviously a less-than-ideal way to find where a bottleneck is.
Bottlenecks usually present themselves long before a crisis, but aren't seen as critical at the time. They are usually things that only occasionally matter, so general availability is usually enough.
Unusual situations aren't unavoidable situations. Taking steps to eliminate the bottleneck is sensible risk management. It is much better than the alternatives, such as cancelling leave or letting deadlines slip.
For those bottlenecks that are part of everyday development, they should be mitigated and eliminated as quickly as possible. In our team, only a few people had the ability to do deploys. This was liveable until those people went on leave. So we implemented automated deploys from Jenkins so that would never be a problem again.
Wednesday, October 12, 2016
Purposes of Unit Testing
- As a development process
Test-driven development is a useful skill to learn. It encourages a clean design, as writing unit tests for messy design is both painful to write and painful to maintain. It also gives a target to code against, by setting what what the function ought to achieve before the function is even written. - Proving the code behaves correctly
Writing unit tests is the simplest way to show that code does what it is meant to do. Unit tests allow easy checking of boundary conditions, to ensure that the function behaves as expected. - Demonstrating robustness
In complex systems, well-written unit tests demonstrate that functions behave as expected. As more features are added, or as code is reused, unit tests demonstrate that these changes do not affect the existing functionality. Conversely, if unit tests easily brake when code changes, it's a good indicator of fragility. - Testing
Some tests algorithms are better for than humans. When testing a function involves a large amount of data, unit tests are best equipped for this job. This is also true for time-dependent functions, or testing algorithms. - Refactoring
Unit tests provide the assurance needed that refactoring the codebase hasn't accidentally changed functionality. Without unit tests, refactoring is fraught with risk.
Tuesday, October 11, 2016
Chickens and Pigs
A Pig and a Chicken are walking down the road.Most people who are involved in a project are not committed. They are chickens. And while they make decisions that affect the project, they are not committed in the same way as the development team are.
The Chicken says: "Hey Pig, I was thinking we should open a restaurant!"
Pig replies: "Hm, maybe, what would we call it?"
The Chicken responds: "How about 'ham-n-eggs'?"
The Pig thinks for a moment and says: "No thanks. I'd be committed, but you'd only be involved."
The development team are the ones who are responsible for what is delivered. They are pigs. Their actions directly affect the project.
A chicken making decisions on behalf of the pigs can adversely affect the team's ability to do their job. As the chicken has no skin in the game, their decisions are divorced from the consequences.
A pig who acts like a chicken is also problematic. If they are not committed to delivery, their involvement detracts from the team's collective ability to deliver.
Monday, October 10, 2016
Branching from a Branch
In Git, everything is a branch. We branch off master for features and bugfixes. And with this, a curious case came up in our team. A feature was dependent on another feature, so he branched from the feature branch.
This strategy had one drawback. He finished before the first feature branch was finished, so merging back into master brought a lot of unnecessary code with it.
Reviewing the situation, of course this had to happen. The dependency between the two user stories required that parts of feature A were needed for feature B. And there was no way to merge feature B in without merging those parts of feature A.
In other words, this shouldn't have been an issue. Except that he wasn't prepared to take ownership of someone else's user story. That would be fair, except he tried to merge his changes that inevitably depended on the other user story.
What should have happened was that feature B should have been put on hold until feature A had been merged back to master. Or, ideally, that feature B should not have begun until its dependency (feature A) had been completed. At the very least, those parts of feature A needed for feature B should have been committed and code reviewed before feature B was merged.
What ended up happening was parts of feature A were merged into master as part of feature B, though they were ignored from the code review perspective. This is far from the ideal, and caused plenty of confusion.
This strategy had one drawback. He finished before the first feature branch was finished, so merging back into master brought a lot of unnecessary code with it.
Reviewing the situation, of course this had to happen. The dependency between the two user stories required that parts of feature A were needed for feature B. And there was no way to merge feature B in without merging those parts of feature A.
In other words, this shouldn't have been an issue. Except that he wasn't prepared to take ownership of someone else's user story. That would be fair, except he tried to merge his changes that inevitably depended on the other user story.
What should have happened was that feature B should have been put on hold until feature A had been merged back to master. Or, ideally, that feature B should not have begun until its dependency (feature A) had been completed. At the very least, those parts of feature A needed for feature B should have been committed and code reviewed before feature B was merged.
What ended up happening was parts of feature A were merged into master as part of feature B, though they were ignored from the code review perspective. This is far from the ideal, and caused plenty of confusion.
Friday, October 7, 2016
An IDE is a Tool, so Learn to Wield One
In my dabbling in my youth with Linux, I learnt a few of the basic command-line commands. It was enough to get by, but I wasn't anywhere near as effective with it as my mate who had converted me to Linux was. He could do all sorts of things with ease from the command-line, while I was constrained by what the GUI would allow me to do.
I learnt the lesson that I should be just as comfortable writing code in Notepad (or on paper) as I am in an IDE. The IDE was dismissed as a crutch that poor developers leaned on.
Over the years, I've come to see that it's both wrong and short-sighted to dismiss IDEs that way.
If for no other reason, being able to code in an IDE takes away some of the more tedious parts of programming. It's a waste of energy to do things like match up every parenthesis, or to scour the code for spelling mistakes. Having syntax highlighted makes reading code so much easier. Having linked code makes drilling down through classes effortless.
There are direct advantages too. Auto-generation of boilerplate code. Autocomplete. Real time code analysis. Instant accompanying documentation. Easy refactoring. Compilation, unit test, and debugging all at the ready.
Learning the shortcuts for an IDE can make it even more powerful. With Eclipse, I take full advantage of some of the text editing shortcuts to make code entry quicker. There's just no way I'd be able to be that effective on a text editor. For most common actions, I don't even have to leave my keyboard.
That's not to say I do everything in my IDE. I find having command line batch scripts more useful for compilation. I prefer TortoiseGit over EGit for source control. But coding-wise, learning how to use Eclipse effectively makes me far more productive.
I learnt the lesson that I should be just as comfortable writing code in Notepad (or on paper) as I am in an IDE. The IDE was dismissed as a crutch that poor developers leaned on.
Over the years, I've come to see that it's both wrong and short-sighted to dismiss IDEs that way.
If for no other reason, being able to code in an IDE takes away some of the more tedious parts of programming. It's a waste of energy to do things like match up every parenthesis, or to scour the code for spelling mistakes. Having syntax highlighted makes reading code so much easier. Having linked code makes drilling down through classes effortless.
There are direct advantages too. Auto-generation of boilerplate code. Autocomplete. Real time code analysis. Instant accompanying documentation. Easy refactoring. Compilation, unit test, and debugging all at the ready.
Learning the shortcuts for an IDE can make it even more powerful. With Eclipse, I take full advantage of some of the text editing shortcuts to make code entry quicker. There's just no way I'd be able to be that effective on a text editor. For most common actions, I don't even have to leave my keyboard.
That's not to say I do everything in my IDE. I find having command line batch scripts more useful for compilation. I prefer TortoiseGit over EGit for source control. But coding-wise, learning how to use Eclipse effectively makes me far more productive.
Thursday, October 6, 2016
When Algorithms Test
Code needs to be tested. Sometimes that testing is best done by an algorithm.
We recently had a user story that involved removing particular characters from the application number string. The purpose was to address a readability issues - at a glance certain characters and numbers are indistinguishable, so removing the offending characters from the string they were generated from. A simple enough coding exercise.
This card, however, sat in the testing column for a long while. It didn't need to, as we had built a unit test that did what a regular tester would do - throw lots of data at the generation and see whether those offending characters were present.
An automated test, in this case, could do what a human could not: quickly generate and accurately assess the functionality. It's checked 1000 times every time the code is compiled - overkill as far as what's needed to be tested, but that overkill is computationally negligible.
Writing the unit test first (TDD) gives the developer the target to aim for, as well as creating the before and after test artefacts. At the very least, the unit tests results could supplement the functional spot-checking that would constitute normal testing for such a story.
Writing code that verifies functionality is not simply for helping developers to write better code, but can be the best tool at the disposal of the testers for doing testing. It's leveraging technology at tasks it excels, which is what we ought to be using technology for.
We recently had a user story that involved removing particular characters from the application number string. The purpose was to address a readability issues - at a glance certain characters and numbers are indistinguishable, so removing the offending characters from the string they were generated from. A simple enough coding exercise.
This card, however, sat in the testing column for a long while. It didn't need to, as we had built a unit test that did what a regular tester would do - throw lots of data at the generation and see whether those offending characters were present.
An automated test, in this case, could do what a human could not: quickly generate and accurately assess the functionality. It's checked 1000 times every time the code is compiled - overkill as far as what's needed to be tested, but that overkill is computationally negligible.
Writing the unit test first (TDD) gives the developer the target to aim for, as well as creating the before and after test artefacts. At the very least, the unit tests results could supplement the functional spot-checking that would constitute normal testing for such a story.
Writing code that verifies functionality is not simply for helping developers to write better code, but can be the best tool at the disposal of the testers for doing testing. It's leveraging technology at tasks it excels, which is what we ought to be using technology for.
Wednesday, October 5, 2016
Book Recommendation: The Nature of Software Development
Development has more than enough to keep anyone occupied on immediate concerns. But it's worth standing back in order to reflect on what it is we are doing.
The Nature of Software Development by Ron Jeffries can best be seen as an expansion on the agile manifesto. Like the manifesto, it's high level but gets right to the point. Like the manifesto, it's concise but profound. And like the manifesto, the aim is to identify what makes for successlooks like.
What we are doing when building software is trying to add value. The word can mean different things in different contexts, but in this case it is a strength rather than a weakness.
We build software because the software will serve a purpose. We should focus on delivering that which serves that purpose, and not waste time on features that are superfluous to that.
We should develop such that there will be value if the project winds up tomorrow. The code should always be ready to deploy. We should fix problems as we go along because that makes development quicker in the long run.
There's much more to the book than I'll cover here. It's a short book, and conversational in tone, and that fits with the message the cook conveys. Deliver what adds value, and don't waste time with filler that doesn't. This book is very valuable.
The Nature of Software Development by Ron Jeffries can best be seen as an expansion on the agile manifesto. Like the manifesto, it's high level but gets right to the point. Like the manifesto, it's concise but profound. And like the manifesto, the aim is to identify what makes for successlooks like.
What we are doing when building software is trying to add value. The word can mean different things in different contexts, but in this case it is a strength rather than a weakness.
We build software because the software will serve a purpose. We should focus on delivering that which serves that purpose, and not waste time on features that are superfluous to that.
We should develop such that there will be value if the project winds up tomorrow. The code should always be ready to deploy. We should fix problems as we go along because that makes development quicker in the long run.
There's much more to the book than I'll cover here. It's a short book, and conversational in tone, and that fits with the message the cook conveys. Deliver what adds value, and don't waste time with filler that doesn't. This book is very valuable.
Tuesday, October 4, 2016
90% of Development is People
When I started out in software development, my biggest concern was my lack of technical ability. Each annual review I flagged technical proficiency as an area of improvement.
After about 5 years and a few projects, I was put on an assignment with absurd deadlines on a different technology stack. This forced me to talk to people more about what I was doing. This was a gestalt moment for me, as it forced me to see software development in a new light.
Raw technical proficiency has its place, and some jobs require it - 3D engine programmers, for example. But for the enterprise work I do, soft skills matter.
The ability to negotiate matters because ideas are seldom unambiguous. It also matters in navigating what can be achieved in the face of unrealistic expectations.
The ability to foster trust matters because it's essential for team cohesion and fostering working relations.
Being able to effectively communicate matters because if we can't express ourselves to others, then we can't get our ideas across.
Being able to listen matters because others need to be able to get their ideas across to us.
And being able to put all that together matters because effective collaboration requires all those skills. A developer who can collaborate is in a far better position than one where those skills are lacking.
After about 5 years and a few projects, I was put on an assignment with absurd deadlines on a different technology stack. This forced me to talk to people more about what I was doing. This was a gestalt moment for me, as it forced me to see software development in a new light.
Raw technical proficiency has its place, and some jobs require it - 3D engine programmers, for example. But for the enterprise work I do, soft skills matter.
The ability to negotiate matters because ideas are seldom unambiguous. It also matters in navigating what can be achieved in the face of unrealistic expectations.
The ability to foster trust matters because it's essential for team cohesion and fostering working relations.
Being able to effectively communicate matters because if we can't express ourselves to others, then we can't get our ideas across.
Being able to listen matters because others need to be able to get their ideas across to us.
And being able to put all that together matters because effective collaboration requires all those skills. A developer who can collaborate is in a far better position than one where those skills are lacking.
Monday, October 3, 2016
Shuhari: first learn, then detach, and finally transcend
From Wikipedia:
As we progress, we are able to identify where the rules work for us and where they don't. We are in a position to know the difference between what works and what doesn't, so breaking the rules gives us valuable feedback. This is ha.
Finally we are in a position where the rules no longer matter. We have developed through experience practices that serve us well, as well as learnt many bad practices to avoid. This is ri.
When we learn something new, the accumulated wisdom of those who came before is the best place to start. We follow what they say because they know what they are talking about. This is shu.
- shu (守) "protect", "obey" — traditional wisdom — learning fundamentals, techniques, heuristics, proverbs
- ha (破) "detach", "digress" — breaking with tradition — detachment from the illusions of self
- ri (離) "leave", "separate" — transcendence — there are no techniques or proverbs, all moves are natural, becoming one with spirit alone without clinging to forms; transcending the physical
As we progress, we are able to identify where the rules work for us and where they don't. We are in a position to know the difference between what works and what doesn't, so breaking the rules gives us valuable feedback. This is ha.
Finally we are in a position where the rules no longer matter. We have developed through experience practices that serve us well, as well as learnt many bad practices to avoid. This is ri.
Friday, September 30, 2016
Kinds of Technical Debt
I think there are three main ways technical debt accumulates. They are:
Not all technical debt is bad, and some kinds of technical debt are more or less benign in the scheme of things, but it's worth recognising what factors lead to technical debt, so that the decision to address it can be a informed and deliberate one.
Sometimes we should know better, sometimes we can't do anything about it, and sometimes it's worth it.
- Debt by negligence
This is the everyday accumulation of technical debt in a project. It's the little things that can really add up over time. Examples of this include...- Leaving code in giant indecipherable methods.
- Not giving variables meaningful names.
- Bloating an existing class.
- Leaving in commented out code.
- Not giving meaningful commit messages.
- Failing to write or update unit tests.
- Debt by constraints
The best of intentions and due diligence among the developers won't matter if there's too much being asked of them. Examples of this include...- The amount of work cannot be achieved in the time allotted.
- The developers with the wrong skill-set are assigned to the project.
- Interference in the process from management.
- The technology stack is less than optimal for the solution.
- The requirements repeatedly shift.
- Debt by investment
There is the very rare occasion that taking on technical debt is the wise decision. Examples of this include...- Responding to an urgent deadline.
- Getting a prototype up and running quickly.
Not all technical debt is bad, and some kinds of technical debt are more or less benign in the scheme of things, but it's worth recognising what factors lead to technical debt, so that the decision to address it can be a informed and deliberate one.
Sometimes we should know better, sometimes we can't do anything about it, and sometimes it's worth it.
Thursday, September 29, 2016
The Virtue of Brevity
"If it is possible to cut a word out, always cut it out." - George Orwell
On my last project, there was a substantial guidelines document for how to do code review. I read that document multiple times, but I could not tell you anything that was in there.
This is not to say that the guidelines were disagreeable or poor, but that there was way too much information for the document to be useful.
I contend the guidelines would have been better if they were shorter and more concise.
For the document, and anything else, being brief serves two purposes.
When we write, it's to get something across. Successful communication is the goal, and brevity is the key to successful communication.
On my last project, there was a substantial guidelines document for how to do code review. I read that document multiple times, but I could not tell you anything that was in there.
This is not to say that the guidelines were disagreeable or poor, but that there was way too much information for the document to be useful.
I contend the guidelines would have been better if they were shorter and more concise.
For the document, and anything else, being brief serves two purposes.
- It becomes more readable, and thus more useful.
- It respects the time and abilities of those who read it.
When we write, it's to get something across. Successful communication is the goal, and brevity is the key to successful communication.
Wednesday, September 28, 2016
Dealing with Waterfall Change Management
No development team can ever just decide to 'go agile'. They can adopt agile practices into development, but the full benefits of agile are realised when that adoption impacts the whole software development lifecycle.
For us, the main sticking point is change management. To get anything approved to even go to preproduction requires seven different approvers and 2 weeks lead-in time, only for the process to be repeated to get into production.
The way change management works affects the way we make our deliverables. From the end of the sprint to production, we have a 3 week lead time, contingent on everything going right. (The product owner makes the final decision whether we are going to production.)
But it also affects how we handle production issues. One thing that cropped up last sprint was a request to change how a screen behaved. Programmatically it needed to flip a boolean on a read-only field, but because of the time and effort to get that change in, multiple avoidable data fixes will accompany the next production release.
We know that there's no getting around this, so we adjust accordingly. It makes us less effective to respond to minor issues, but it's manageable for the day-to-day.
For us, the main sticking point is change management. To get anything approved to even go to preproduction requires seven different approvers and 2 weeks lead-in time, only for the process to be repeated to get into production.
The way change management works affects the way we make our deliverables. From the end of the sprint to production, we have a 3 week lead time, contingent on everything going right. (The product owner makes the final decision whether we are going to production.)
But it also affects how we handle production issues. One thing that cropped up last sprint was a request to change how a screen behaved. Programmatically it needed to flip a boolean on a read-only field, but because of the time and effort to get that change in, multiple avoidable data fixes will accompany the next production release.
We know that there's no getting around this, so we adjust accordingly. It makes us less effective to respond to minor issues, but it's manageable for the day-to-day.
Tuesday, September 27, 2016
The Heisenberg Product Owner
"We're done when I say we're done" - Heisenberg
From the development's team perspective, product owner sign off is the one stage of the development process they can't control. Ideally, this should a minimal step - all the work has been done before this point. But the product owner can always say no.
As scrum masters, our job is to try to sort this out, as this is indicative of failure somewhere along the way. The product owner is a part of the team, and thus involved in the day-to-day delivery of the project. So the question is where the failure is taking place.
There are many reasons within the team this could happen, and many ways a team could deal with it. One thing to watch out for, however, is a product owner who simply cannot make up their mind.
If this is the case, it's up to the scrum master to intervene. Simply acquiescing to the product owner is not a good idea as it breaks the flow of development. It also erases the trust and enthusiasm of the development, as their work is undermined.
If new aspects of a user story are required, they are to be turned into new user stories, prioritised, and put on the backlog. This gives a path to improvement that ensures those features are added, while not holding up the flow of development. The development team then doesn't have to worry about moving targets or feature creep.
Similarly, if user stories aren't being fleshed out enough, then there's grounds for spending more time in backlog refinement. Or, perhaps, the developers can swarm with the product owner during development.
As scrum masters, it's our job to ensure the process is running smoothly, and that includes how the product owner works with the team.
From the development's team perspective, product owner sign off is the one stage of the development process they can't control. Ideally, this should a minimal step - all the work has been done before this point. But the product owner can always say no.
As scrum masters, our job is to try to sort this out, as this is indicative of failure somewhere along the way. The product owner is a part of the team, and thus involved in the day-to-day delivery of the project. So the question is where the failure is taking place.
There are many reasons within the team this could happen, and many ways a team could deal with it. One thing to watch out for, however, is a product owner who simply cannot make up their mind.
If this is the case, it's up to the scrum master to intervene. Simply acquiescing to the product owner is not a good idea as it breaks the flow of development. It also erases the trust and enthusiasm of the development, as their work is undermined.
If new aspects of a user story are required, they are to be turned into new user stories, prioritised, and put on the backlog. This gives a path to improvement that ensures those features are added, while not holding up the flow of development. The development team then doesn't have to worry about moving targets or feature creep.
Similarly, if user stories aren't being fleshed out enough, then there's grounds for spending more time in backlog refinement. Or, perhaps, the developers can swarm with the product owner during development.
As scrum masters, it's our job to ensure the process is running smoothly, and that includes how the product owner works with the team.
Monday, September 26, 2016
Following the Boy Scout Rule
"Always leave the campground cleaner than you found it."
In all my years as a software developer, the only times I've ever gotten to start from scratch is when I've worked on personal projects. And even then, starting from scratch is a last resort.
My experience of being a programmer is working in someone else's architecture. I get to expand it out, fix it up, and sometimes refactor it away. And each time I end up having the same question - why is it built that way? And each time there's the same answer - it's because the code needed to be delivered quickly.
It feels at times that all we can do as programmers is try not to make things worse. One thing I've done is tried to fit in with the existing way of doing things. That way the code is consistent for those who maintain it in the future. But it doesn't solve the problem, merely prevents it from getting any worse.
The answer is meant to be refactoring, though it's really not an answer for fragile code. It's a sad irony that the code in most need of refactoring is the code least able to be refactored...
Any complex system is going to seem impenetrable at first glance, but there are ways to improve it. A system doesn't need to be overhauled to be fixed, and something as simple as tidying up the code that's being worked on is enough to make things better.
For this project, we have pretty abysmal unit testing. So one rule we've set for ourselves is to put unit tests in wherever we find ourselves enhancing the code. Coverage is still woeful, but it is less woeful than it was a month ago. Another thing is to make sure no commit has commented-out code.
Code doesn't get to a messy state by a single act, nor can it be fixed in one. The complex mess that is the average codebase (at least average from my experience) is the way it is because of a long path to that state. Making things better is a similarly long path, but thankfully one that be addressed as part of enhancing a system. Fragility is quantitative, not qualitative.
In all my years as a software developer, the only times I've ever gotten to start from scratch is when I've worked on personal projects. And even then, starting from scratch is a last resort.
My experience of being a programmer is working in someone else's architecture. I get to expand it out, fix it up, and sometimes refactor it away. And each time I end up having the same question - why is it built that way? And each time there's the same answer - it's because the code needed to be delivered quickly.
It feels at times that all we can do as programmers is try not to make things worse. One thing I've done is tried to fit in with the existing way of doing things. That way the code is consistent for those who maintain it in the future. But it doesn't solve the problem, merely prevents it from getting any worse.
The answer is meant to be refactoring, though it's really not an answer for fragile code. It's a sad irony that the code in most need of refactoring is the code least able to be refactored...
Any complex system is going to seem impenetrable at first glance, but there are ways to improve it. A system doesn't need to be overhauled to be fixed, and something as simple as tidying up the code that's being worked on is enough to make things better.
For this project, we have pretty abysmal unit testing. So one rule we've set for ourselves is to put unit tests in wherever we find ourselves enhancing the code. Coverage is still woeful, but it is less woeful than it was a month ago. Another thing is to make sure no commit has commented-out code.
Code doesn't get to a messy state by a single act, nor can it be fixed in one. The complex mess that is the average codebase (at least average from my experience) is the way it is because of a long path to that state. Making things better is a similarly long path, but thankfully one that be addressed as part of enhancing a system. Fragility is quantitative, not qualitative.
Friday, September 23, 2016
The Shopping Analogy: Learning to Shift Your Thinking
I don't like to go food shopping often, so what I end up doing is buying more than I need. I'm anticipating my future needs - short term for meat and vegetables, mid-to-long term for everything else.
So when I buy, I'm rarely thinking of a specific dish, but what kinds of foods I may want in the future. This has two apparent consequences. First, I end up getting things that go bad before they're used. Second, I'm susceptible to the specials and whims of possibility.
Any food I don't end up using is a waste of money. I'm paying for the potential it provides rather than the reality of consumption. And anything that goes into a packed fridge or pantry is out of sight (i.e. out of mind) so it can easily be lost. While I always have food around - which may be of psychological comfort - I'm paying for things I don't use.
There is a way of solving this: buy ingredients on a needs basis rather than an anticipatory one.
What this requires, however, is a change of mindset. That when I shop, I buy based on an actual need rather than an anticipated one. So each time I buy less, but make more trips to the shop. This way I know what I'll buy will be consumed, and that my freezer and pantry won't be full of spontaneous anticipated purchases.
In software design, the Big Requirements Up Front approach is similar to trying to shop in anticipation of what one might need in the future. Like with shopping, what sounds like a good idea at the point of sale won't necessarily translate into what's needed now, or even useful in the future.
The agile approach is for the business to value and prioritise what is needed, not what might be nice to have at some stage. A pantry full of uneaten food would rightly be considered a waste, just as software with unused features ought to be considered a waste.
And like shopping, recognising the problem of waste isn't enough without that change of mindset. The process of 'going agile' is not about following scrum (or any other process) to better outcomes, but embracing a mindset that makes the process effective. It's the shift from "I want this someday" through "I'll better plan to use what I buy" to "I'll buy what I need when I need it."
So when I buy, I'm rarely thinking of a specific dish, but what kinds of foods I may want in the future. This has two apparent consequences. First, I end up getting things that go bad before they're used. Second, I'm susceptible to the specials and whims of possibility.
Any food I don't end up using is a waste of money. I'm paying for the potential it provides rather than the reality of consumption. And anything that goes into a packed fridge or pantry is out of sight (i.e. out of mind) so it can easily be lost. While I always have food around - which may be of psychological comfort - I'm paying for things I don't use.
There is a way of solving this: buy ingredients on a needs basis rather than an anticipatory one.
What this requires, however, is a change of mindset. That when I shop, I buy based on an actual need rather than an anticipated one. So each time I buy less, but make more trips to the shop. This way I know what I'll buy will be consumed, and that my freezer and pantry won't be full of spontaneous anticipated purchases.
In software design, the Big Requirements Up Front approach is similar to trying to shop in anticipation of what one might need in the future. Like with shopping, what sounds like a good idea at the point of sale won't necessarily translate into what's needed now, or even useful in the future.
The agile approach is for the business to value and prioritise what is needed, not what might be nice to have at some stage. A pantry full of uneaten food would rightly be considered a waste, just as software with unused features ought to be considered a waste.
And like shopping, recognising the problem of waste isn't enough without that change of mindset. The process of 'going agile' is not about following scrum (or any other process) to better outcomes, but embracing a mindset that makes the process effective. It's the shift from "I want this someday" through "I'll better plan to use what I buy" to "I'll buy what I need when I need it."
Thursday, September 22, 2016
Keeping Track of Tasks with SplenDO
A few jobs back, my monitor would be covered in post-it notes. It was my way to keep track of my tasks, or my progress in particular tasks. It's a practice I wish I kept up, and I'm disappointed I let the lack of office-supplied stationary get in the way of doing it. For my development, it was the time I felt the most productive.
Between Notepad, Sticky Notes, and actual notepads, I've never managed to find a sustainable suitable replacement. Memory only goes so far.
For the last few weeks, I've been trialling out the SplenDO on my mobile. This is what I found so far.
The useful:
The not-so-useful:
The verdict:
As far as keeping organised goes, this is far superior to my old post-it note system. It's simple to use, and it's notifications keep it relevant. It's great for someone who always has a smartphone around.
Between Notepad, Sticky Notes, and actual notepads, I've never managed to find a sustainable suitable replacement. Memory only goes so far.
For the last few weeks, I've been trialling out the SplenDO on my mobile. This is what I found so far.
The useful:
- Putting rooms on meetings - Knowing when meetings are is useful enough (though mostly taken care of by Outlook), but having the place of the meeting at your fingertips is invaluable.
- Tasks I'd normally delay - Some things I know I tend to procrastinate on, such as anything involving chasing up people. Having it as a task (and one that becomes overdue) compels me to get it out of the way sooner rather than later.
- Temporal tasks - For the occasional task that has a deadline, it's very useful to have a reminder close to when it's needed.
- Tasks that come up in conversation - So much of what needs to happen at work is stated during the course of conversations. This alleviates the need to carry a notepad and pen everywhere, or trying to remember every action item after the conversation is over.
The not-so-useful:
- Habitual tasks - The daily stand-up happens at 9:45am each weekday. While I have the tasklist to remind me of this, it doesn't do much to have it as a task. I find I do it just to tick it off that it is done.
- Aspirational tasks - If the task isn't necessary, then putting it on the list becomes a source of frustration as the deadline passes.
- Setting times on non-temporal tasks - This is a case of over-planning. If a task is not time-dependent, it makes no sense to try to give a deadline to it.
The verdict:
As far as keeping organised goes, this is far superior to my old post-it note system. It's simple to use, and it's notifications keep it relevant. It's great for someone who always has a smartphone around.
Wednesday, September 21, 2016
Our Training Sprint
I suppose there's never a good way to start an agile project. On the project I was put on, almost everyone was new to the codebase, new to the project, and new to each other. So I called the first sprint Sprint 0 - our training sprint.
The difficulty was to ramp up with learning the existing product, learning the business rules, and learning to work with the tools and processes provided. With this sprint complete, here are some of the lessons I came away with:
The difficulty was to ramp up with learning the existing product, learning the business rules, and learning to work with the tools and processes provided. With this sprint complete, here are some of the lessons I came away with:
- Being on the project doesn't mean buy-in - The standard way to illustrate commitment in agile is the fable of the chicken and the pig. As a development team, the ideal is to get collective commitment, where all individuals work together to get the product over the line. It was clear when individuals were merely involved rather than committed, be it some or most of the time.
- The daily scrum is important - Perhaps the daily stand-up felt too much like a status meeting, but it was really useful for the team to know what was going on with the project as a whole. When information was shared outside of this, it didn't always get to everyone.
- Intervention should be done early - We had one member who only ever turned up for some of the meetings, and missed a lot of the daily stand-ups. Intervention about what was expected as a team member came too late in the sprint to rectify the problem. And even then, it did little to change things.
- Business As Usual consumes time and energy - We had two issues with BAU. First, when production issues arose, it took time out of the sprint to deal with them. Second, there were non-sprint activities for nearly all members dealing with that took focus away from making progress.
- It's easy to revert into a management-like demeanour - The ideal of the scrum master being a servant-leader is really difficult to achieve in practice. At times, it felt like I was driving the process too much, rather than simply guiding a self-organising team.
- The day-to-day can obfuscate accumulated achievement - At times, it felt like the sprint moved really slow. Yet when I added up what we had built by the end of the sprint, we had made quite a number of improvements. And that wasn't including some of the background tasks we did, like setting up automated deployments or getting better code coverage tools.
Tuesday, September 20, 2016
Effective Bitbucket: 10 Tips to Better Utilisation of the Tool
One can simply use Bitbucket as a basic source repository, configured to behave like SVN. But with a few configurations and proper usage, it can be very powerful. Here are 10 things I've found for getting the most out of it.
- Use the feature-branch branching model
Although Bitbucket can be configured to work in master, creating branches should be the default. It allows you to separate out all the muck that happens through the development process from the clean deployable code in the master branch. Merge back into master when the code is ready, and only when it is ready.
(The feature-branch model is good, the gitflow model is better.) - Prevent changes to master except by pull request
Setting up a feature branch workflow doesn't guarantee people won't develop in master. Putting a rule restricting commits to master will. This can be achieved by going to "Settings -> Branch permissions" then adding a rule that master can only be changed by a pull request. - Create branches in JIRA
Working in branches is good. Working in meaningfully named branches is better. If you are integrating Bitbucket with JIRA, then you can create branches from a ticket. From the JIRA issue, click "Create Branch", then set the right branch type and which branch to branch from. - Don't take ownership of branches
The main point of source control is collaborative development. Creating branches is a good way to isolate changes from the main codebase (until they are ready), but that's no reason to isolate yourself. - Write meaningful comments in commit messages
This is true for any source repository, but it's well worth repeating. Writing a good commit message may seem tedious at the time, but it's invaluable for understanding that code in the future. The code is the how, the commit message should be the what and the why. - Turn on mandatory code review approval
In "Settings -> Pull requests", tick "Requires [1] approvers". Bitbucket has code review built into the web interface, so make sure they happen before the code is merged back into master. Having code reviews at this point dissolves the pre-commit / post-commit dilemma, as the gateway moves from entry into the repository to entry into the main codebase. - Turn on mandatory successful build
All source repositories should be hooked up to a continuous integration tool. Bitbucket goes one better in that it can use the results of the build as a necessary step in the merge process. In "Settings -> Pull requests", tick "Requires a minimum of [1] successful builds. But it also requires some setup on the CI tool. For Jenkins, it required the Stash plugin (the Bitbucket plugin is for the cloud version). And to set the job up right, build branches with the prefix features and bugfix. If you're using Maven, set the goals as clean test as this will compile / run the unit tests, but not build the final artefact. - Treat the code reviews as a conversation
Comments can be put against files or against lines. Multiple commits contribute to a pull request. The history of pull requests can be viewed at any time by changing the filter on the Pull Request screen. So treating the code reviews as a conversation between developers is a way of documenting specific changes in a way that most code review processes lack. - Don't hang onto old branches
If you're finished with a branch, and the code has been successfully merged into master, then clean up the branch right then and there. Deleting the branch signals that there's no work left to be done on it. And even if a defect is found on that feature, just create a new branch in its place. This way, the repository does the job of keeping the branch up to date with all the other code changes. - Search is a fast way to find code
Searching on Bitbucket is much faster than searching in an IDE, so using the Bitbucket interface can assist in being able to navigate through an unfamiliar codebase. If there are multiple projects / repositories on the same server, a global search can be performed. To search, just use the search box located on the top navbar. This, by default, will be a global search. To search within a particular repository, navigate to that repository before performing the search.
Monday, September 19, 2016
From Feature Branch to Gitflow
When we migrated from SVN, one of the first acts was to lock down the master branch. This was mainly to get developers used to the Git way of doing things, so hopefully to reap the benefits that comes with proper usage of branches.
The ideal, however, has been to get to a Gitflow model. The idea is to have easy access to the Production code - should the need arise - and that fits well with the Git branching model. This could have been retrofitted to our migrated codebase, but we didn't do it as we thought the best thing to do was get the team used to Git first.
Here's the path we took to incorporate Gitflow:
*I really don't know why we do this after production rather than immediately before the production deployment. It seems odd to deploy SNAPSHOT code into production.
The ideal, however, has been to get to a Gitflow model. The idea is to have easy access to the Production code - should the need arise - and that fits well with the Git branching model. This could have been retrofitted to our migrated codebase, but we didn't do it as we thought the best thing to do was get the team used to Git first.
Here's the path we took to incorporate Gitflow:
- Create a release branch - This branch incorporated all potential fixes found in the preprod environment while preparing for production. This branch could only be changed by pull request, so the developers would branch off it if needed. A job in Jenkins was set up to build the release branch.
- Do the release - The latest artefact built from the release branch was promoted to production.
- Create a production branch - A new branch called production was set up in Bitbucket. On the branching model tab, this branch was set as the production branch. The current release branch was merged into it.
- Finalise the release - At this point, we put an exception on the restrictions to the release branch: the release manager. The release manager removed the SNAPSHOT from the version number and committed, so Jenkins would build the final version*. The release version was then incremented to the new SNAPSHOT version.
- Merge back to master - All the release changes were merged back into the master branch. As the branch had served its purpose, it was deleted.
*I really don't know why we do this after production rather than immediately before the production deployment. It seems odd to deploy SNAPSHOT code into production.
Friday, September 16, 2016
Embracing Shadow IT
Back in 2009, I was put on a project with a very short time frame and a lot of work to deliver. A couple of the senior developers looked at the resources and technologies at our disposal, and they realised they were inadequate for the task at hand. So we set up a completely open source solution - developed on Eclipse instead of RAD, tested on Tomcat instead of WebSphere, used SVN instead of ClearCase, and hooked into Hudson.
All this was running on a spare machine sitting on my desk. And of course, we had to keep it a secret. It all came to a head two project managers later, but it is a vindication that when I returned to the department years later, many of those tools had become the norm.
There is a term for this: Shadow IT. And it comes about because no enterprise-wide solution is going to cover everything that needs to be done. In 2009, the computers we had weren't powerful enough to effectively develop on RAD or WebSphere. ClearCase didn't give us continuous integration, and is a generally burdensome source repository compared with SVN.
(In late 2010, I switched projects in the department to one that used the official set of tools. The team had just lost 7 days to a ClearCase outage, despite it having paid support. We never lost any time running SVN.)
What surprised me about our switch to the Atlassian suite of tools is we are still in a semi-secretive state about it. It's a tool for our section, that we are meant to deny knowledge of to outsiders. The tools and their relative effectiveness are of interest to others, and better practices only come by demonstrated utility. Otherwise we're beholden to sales pitches to people who aren't directly affected by the the products being bought.
The whole point of going agile is that teams individually can reflect on what works (and what doesn't!), so they can respond for the future. Having severe constraints on their tools limits their capacity for change, thus making the exercise less effective.
For us, we are certainly benefiting from using our own tools. JIRA is far better for collaboration than the physical boards we had. Confluence is more intuitive and powerful than Sharepoint. Bitbucket has many advantages over SVN. But most of all, we've got a development team excited and engaged with the tools - a far better situation than the usual stream of complaints that have been ubiquitous in every workplace I've been in.
All this was running on a spare machine sitting on my desk. And of course, we had to keep it a secret. It all came to a head two project managers later, but it is a vindication that when I returned to the department years later, many of those tools had become the norm.
There is a term for this: Shadow IT. And it comes about because no enterprise-wide solution is going to cover everything that needs to be done. In 2009, the computers we had weren't powerful enough to effectively develop on RAD or WebSphere. ClearCase didn't give us continuous integration, and is a generally burdensome source repository compared with SVN.
(In late 2010, I switched projects in the department to one that used the official set of tools. The team had just lost 7 days to a ClearCase outage, despite it having paid support. We never lost any time running SVN.)
What surprised me about our switch to the Atlassian suite of tools is we are still in a semi-secretive state about it. It's a tool for our section, that we are meant to deny knowledge of to outsiders. The tools and their relative effectiveness are of interest to others, and better practices only come by demonstrated utility. Otherwise we're beholden to sales pitches to people who aren't directly affected by the the products being bought.
The whole point of going agile is that teams individually can reflect on what works (and what doesn't!), so they can respond for the future. Having severe constraints on their tools limits their capacity for change, thus making the exercise less effective.
For us, we are certainly benefiting from using our own tools. JIRA is far better for collaboration than the physical boards we had. Confluence is more intuitive and powerful than Sharepoint. Bitbucket has many advantages over SVN. But most of all, we've got a development team excited and engaged with the tools - a far better situation than the usual stream of complaints that have been ubiquitous in every workplace I've been in.
Thursday, September 15, 2016
Walking for Health and Sanity
One thing I always make sure to do before lunch every day (it isn't raining) is get outside for a walk. I consider this the most important thing I do at the office each day.
The obvious reason to walk is that it's a form of exercise, and that has all sorts of health benefits. It certainly is better than sitting at a desk all day. One of the reasons I started doing it was seeing the occupational consequences for others, and realised I was heading down the same path.
It's not always easy to make time to exercise, so my lunch break serves as a time I can dedicate to it without needing to juggle other things around. If it becomes habitual, one never needs to make time for it.
And as far as exercise goes, walking is not that strenuous. One can walk (and even walk fast) without breaking a sweat, which makes it a convenient form of exercise for the office.
There are benefits besides the fact that it's exercise, and exercise is good for your health. Walking has been found to help with learning - that it can increase memory retention. For my personal experience, I've found it's a great way to collect my thoughts, and even solve problems. Many times I've been stuck on a problem, only to solve it 5 minutes into my lunchtime walk.
If you are walking outside, there's the added bonus of getting vitamin D. Not getting enough vitamin D is an unexpected occupational hazard of working in an office, as incidental exposure to the sun (our primary source of vitamin D) is much less than in other professions. There's even research suggesting that walking through areas with trees has benefits over walking through streets.
Finally, walking provides an excuse to get away from the office, away from coworkers, and away from the work itself. It gives me a break, and a chance to take my mind off work-related matters. And best of all, it's undisturbed.
The obvious reason to walk is that it's a form of exercise, and that has all sorts of health benefits. It certainly is better than sitting at a desk all day. One of the reasons I started doing it was seeing the occupational consequences for others, and realised I was heading down the same path.
It's not always easy to make time to exercise, so my lunch break serves as a time I can dedicate to it without needing to juggle other things around. If it becomes habitual, one never needs to make time for it.
And as far as exercise goes, walking is not that strenuous. One can walk (and even walk fast) without breaking a sweat, which makes it a convenient form of exercise for the office.
There are benefits besides the fact that it's exercise, and exercise is good for your health. Walking has been found to help with learning - that it can increase memory retention. For my personal experience, I've found it's a great way to collect my thoughts, and even solve problems. Many times I've been stuck on a problem, only to solve it 5 minutes into my lunchtime walk.
If you are walking outside, there's the added bonus of getting vitamin D. Not getting enough vitamin D is an unexpected occupational hazard of working in an office, as incidental exposure to the sun (our primary source of vitamin D) is much less than in other professions. There's even research suggesting that walking through areas with trees has benefits over walking through streets.
Finally, walking provides an excuse to get away from the office, away from coworkers, and away from the work itself. It gives me a break, and a chance to take my mind off work-related matters. And best of all, it's undisturbed.
Wednesday, September 14, 2016
What does a Scrum Master do?
As a developer, it's always been really easy to know what my responsibilities are (and what they aren't). I do tech-related stuff, answer tech related questions, and try to be as articulate as possible at the intersection of the technical stuff and the wider business processes.
With moving into the scrum master role, this has shifted. I'm still developing, but developing nowhere near as much as before. Rather, my standard tool has switched from RAD to JIRA. I'm even finding that RAD remains unopened until time during the day.
I also find myself doing a lot of coordination tasks. The documentation about scrum masters repeatedly use the word 'facilitator', but it seems like coordinator is more appropriate. Perhaps that's indicative of less-than-complete buy in among the team, or simply the teething issues that come with trying to start from scratch. Either way, a lot of time is trying to get the right information from the right people.
The final big thing for me is the sense of responsibility. On the last project where I was purely in a development role, I had the collective commitment attitude, but I knew a lot of what was being done by my Scrum Master was obscured from me. It's a big step up - I have to be the one to make all the things happen that I took for granted when my focus was purely development.
With moving into the scrum master role, this has shifted. I'm still developing, but developing nowhere near as much as before. Rather, my standard tool has switched from RAD to JIRA. I'm even finding that RAD remains unopened until time during the day.
I also find myself doing a lot of coordination tasks. The documentation about scrum masters repeatedly use the word 'facilitator', but it seems like coordinator is more appropriate. Perhaps that's indicative of less-than-complete buy in among the team, or simply the teething issues that come with trying to start from scratch. Either way, a lot of time is trying to get the right information from the right people.
The final big thing for me is the sense of responsibility. On the last project where I was purely in a development role, I had the collective commitment attitude, but I knew a lot of what was being done by my Scrum Master was obscured from me. It's a big step up - I have to be the one to make all the things happen that I took for granted when my focus was purely development.
Tuesday, September 13, 2016
Utilising Continuous Integration
When I came into my current role, we had our repository hooked up to Jenkins. Every time we would commit, it would build the code and run our unit tests. We'd know almost right away if someone had broken the build.
Yet if we wanted to deploy, that was a manual process. Someone would download the EAR and manually log into the Test environment to upload it. Continuous deployment was deemed a bad idea as it would disrupt testing, but there was no good reason to keep it as a manual process. Not least that the power to deploy was limited to a couple of people.
In our new setup, we have three deployment jobs:
The first two are automatic and will run whenever there is a code change. The Branches job is integrated into our code review process, as Bitbucket is configured to only allow merging upon success. The Master job ensures we always have a potentially deployable artefact.
The Deploy job is manual, and controlled by the testing team, who run it whenever they are ready for more features to be tested. This gives the advantage that testers decide when they are ready for new work.
This doesn't solve all deployment problems. Any intervention with databases is manual. The necessity to deploy shared EARs is invisible to the testers and requires intervention on our part. And deploying to hither environments involves the cumbersome change management process. However, it's important to recognise all the wins the current setup gives us over what was there before.
Yet if we wanted to deploy, that was a manual process. Someone would download the EAR and manually log into the Test environment to upload it. Continuous deployment was deemed a bad idea as it would disrupt testing, but there was no good reason to keep it as a manual process. Not least that the power to deploy was limited to a couple of people.
In our new setup, we have three deployment jobs:
- Branches - builds a maven deploy up to the unit tests, and informs Bitbucket of the results.
- Master - builds the master branch and stores the artifact on Archiva.
- Deploy - takes the last successful build and deploys it to the Test server.
The first two are automatic and will run whenever there is a code change. The Branches job is integrated into our code review process, as Bitbucket is configured to only allow merging upon success. The Master job ensures we always have a potentially deployable artefact.
The Deploy job is manual, and controlled by the testing team, who run it whenever they are ready for more features to be tested. This gives the advantage that testers decide when they are ready for new work.
This doesn't solve all deployment problems. Any intervention with databases is manual. The necessity to deploy shared EARs is invisible to the testers and requires intervention on our part. And deploying to hither environments involves the cumbersome change management process. However, it's important to recognise all the wins the current setup gives us over what was there before.
Monday, September 12, 2016
Building a Minimum Viable Product
When developing on a Waterfall project, the criteria by which you develop to are determined by the specifications. This is both a blessing and a curse, in that the decisions about what to do are external to the development team.
The general issue with design by specification is each part is of equal value, in the sense that it's presented as an all-or-nothing proposition. And since everything is designed upfront, it has to cater for all conceived features - no matter the actual value.
With agile, features are organised and developed by value. The idea is to create a minimum viable product, where the features developed are those that will be the those that add maximum business value.
There are two really good reasons for doing this:
To me, both issues come down to the problem of trying to sort out the needs of a system upfront. Design, especially on detail, is constrained by the development costs - something difficult to work out in advance. And the value of features is something that can only really be seen once the system is live, so anticipating that is going to involve guesswork.
Keeping in mind the possibility that development could wind up tomorrow puts pressure on the actions of today. Each day a product is not delivered is a day the value of the product is not being realised. By focusing on MVP, the outcome is delivering sooner, delivering something that addresses the needs, and thus represents the best value of the time and effort of everyone involved.
The general issue with design by specification is each part is of equal value, in the sense that it's presented as an all-or-nothing proposition. And since everything is designed upfront, it has to cater for all conceived features - no matter the actual value.
With agile, features are organised and developed by value. The idea is to create a minimum viable product, where the features developed are those that will be the those that add maximum business value.
There are two really good reasons for doing this:
- If development needs to stop at any time, then what's delivered will reflect the greatest value.
- Most software features turn out to be either never or rarely used.
To me, both issues come down to the problem of trying to sort out the needs of a system upfront. Design, especially on detail, is constrained by the development costs - something difficult to work out in advance. And the value of features is something that can only really be seen once the system is live, so anticipating that is going to involve guesswork.
Keeping in mind the possibility that development could wind up tomorrow puts pressure on the actions of today. Each day a product is not delivered is a day the value of the product is not being realised. By focusing on MVP, the outcome is delivering sooner, delivering something that addresses the needs, and thus represents the best value of the time and effort of everyone involved.
Friday, September 9, 2016
Busier Than Ever
One thing I'll lament to anyone who is in earshot is how little I get done. What I mean by that is that the Java tasks I've pulled into the development queue.
On the other hand, I now need to remind myself to get water. I haven't even made tea since I started in this role. And it's a deliberate and rare act to go get a coffee with a coworker from the café just outside.
So the rules of inference suggest that I must be doing something. And the most obvious answer is dealing with people.
However autistic I feel at times, dealing with people is natural in a way that development has never been. The years of developed intuitions and heuristics I have about coding don't surmount the weirdness of the task cognitively. We're evolved social interactors, not programmers.
The point I'm getting at is that it doesn't feel like work to communicate. Having a coding task interrupted by a conversation never requires much effort to switch context, but it's very difficult to break free from a conversation to do coding.
The feeling of being busy is not being able to set aside time for the tasks I want to be doing, as what I need to be doing is something that doesn't feel like work. It is work, and it is necessary work, and it sure is time-consuming work.
On the other hand, I now need to remind myself to get water. I haven't even made tea since I started in this role. And it's a deliberate and rare act to go get a coffee with a coworker from the café just outside.
So the rules of inference suggest that I must be doing something. And the most obvious answer is dealing with people.
However autistic I feel at times, dealing with people is natural in a way that development has never been. The years of developed intuitions and heuristics I have about coding don't surmount the weirdness of the task cognitively. We're evolved social interactors, not programmers.
The point I'm getting at is that it doesn't feel like work to communicate. Having a coding task interrupted by a conversation never requires much effort to switch context, but it's very difficult to break free from a conversation to do coding.
The feeling of being busy is not being able to set aside time for the tasks I want to be doing, as what I need to be doing is something that doesn't feel like work. It is work, and it is necessary work, and it sure is time-consuming work.
Thursday, September 8, 2016
The Cycle of Process
- The solution to alleviate human error or overcome personal limitations is the addition of process to avoid / minimise it.
- The longer the timespan, the greater the accumulation of processes.
- The more processes that are in place, the less intrusive adding new process seems.
- Working within such an environment entrenches it as the status quo.
- Eventually the processes become so taxing that any changes in personnel or circumstances pose a problem in itself.
- Outside perspectives offer an entirely new way of doing things that will eliminate all the wasted effort gone to maintaining the process rather than the outcomes the process was there to manage.
- The new way of doing things turns out not to eliminate all human error or overcome personal limitations.
Wednesday, September 7, 2016
Going Agile: The Elevator Pitch
What we are trying to achieve is to change the way IT interfaces with business. Instead of business trying to work out ahead of time all the details they want, the business collaborates with IT to work out what the requirements are.
This is further broken down into time, with development continuously delivering functionality. The business doesn't need to have every detail set out ahead of time, but just enough of an understanding of the features that development can take place.
The advantage of this is that development becomes a feedback loop, where features can be gradually introduced, changed, enhanced and even removed based on business requirements. In short, it allows business to concentrate on what is needed, and for IT to respond to these needs as they arise.
This is further broken down into time, with development continuously delivering functionality. The business doesn't need to have every detail set out ahead of time, but just enough of an understanding of the features that development can take place.
The advantage of this is that development becomes a feedback loop, where features can be gradually introduced, changed, enhanced and even removed based on business requirements. In short, it allows business to concentrate on what is needed, and for IT to respond to these needs as they arise.
Tuesday, September 6, 2016
How to Git: Branch, Code, Merge, Delete, Repeat
Switching technologies under people is a great way to learn about implicit understandings people have about how to use the technology. With the branching model in Git, it brought out just how people see their own development as a solo enterprise.
The common reaction was to think that each person has their own branch to work with, rather than a branch serving as a unit of work. This highlights one barrier to better collaboration, but also can limit the effectiveness that branching in Git provides.
The best way I've found to deal with branching in Git is as follows:
The ideal is that branches are short-lived. The longer a branch lives, the more likely there are to be merge conflicts when it's brought back.
With JIRA and Bitbucket, it's easy to create a branch named for a particular feature. So anyone seeing a branch will at a glance know what unit of work that branch is for. And since this can be done multiple times, having to recreate a branch when more aspects of a user story emerge is simple.
The common reaction was to think that each person has their own branch to work with, rather than a branch serving as a unit of work. This highlights one barrier to better collaboration, but also can limit the effectiveness that branching in Git provides.
The best way I've found to deal with branching in Git is as follows:
- Branch - When starting a new feature, branch from the head of the development branch. This ensures you are working with the latest deployable code.
- Code - Your regular development work. Commit as needed as it backs up your work, as well as allowing for collaboration.
- Merge - When your coding is done, merge it back into the development branch.
- Delete - After you merge the branch back to the development branch, delete it. Leaving it just creates clutter, and clutter that goes quickly out of date at that. (if needed) Repeat - If it turns out steps 1-4 didn't cover the feature, start the process again. Don't worry about maintaining a branch for the possibility that feature still needs to be worked on.
The ideal is that branches are short-lived. The longer a branch lives, the more likely there are to be merge conflicts when it's brought back.
With JIRA and Bitbucket, it's easy to create a branch named for a particular feature. So anyone seeing a branch will at a glance know what unit of work that branch is for. And since this can be done multiple times, having to recreate a branch when more aspects of a user story emerge is simple.
Monday, September 5, 2016
The Beauty and Necessity of The Agile Manifesto
Of all the documentation I've read on agile, the one thing I keep referring to in conversation is the Agile Manifesto. For reference, here it is:
Whenever I'm pulled into meetings to discuss the agile process, I think of this manifesto. Whenever someone asks me about how we are collaborating with business, I think of this manifesto. Whenever the very question of agile comes up, I think of this manifesto.
I even have the manifesto printed out and stuck to my desk!
The beauty of the manifesto is how much it says in so few words. The terms are broad enough such that they don't get bogged down in any particular aspect of the SDLC. Yet the terms are also targeted enough that they hit the relevant factors associated with software development.
Furthermore, they are a statement of intent - of what a software developer ought to strive for. The goal is working software, and to get that requires interaction, collaboration, and responding to change.
Those things on the right too often get in the way of creating working software. We've all been in projects where managers put pressure on the team because it deviates off their unrealistic "plan". That bad ideas get coded in because the phase to negotiate that was before coding began. That a effort is put into elaborate documentation that goes out of date the moment a system enhances it.
Even when going agile, the agile manifesto works to dampen the tendency to think any problem can be solved with more process. It can stop those in-grained waterfall tendencies from answering the inevitable "now what?" question.
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
Whenever I'm pulled into meetings to discuss the agile process, I think of this manifesto. Whenever someone asks me about how we are collaborating with business, I think of this manifesto. Whenever the very question of agile comes up, I think of this manifesto.
I even have the manifesto printed out and stuck to my desk!
The beauty of the manifesto is how much it says in so few words. The terms are broad enough such that they don't get bogged down in any particular aspect of the SDLC. Yet the terms are also targeted enough that they hit the relevant factors associated with software development.
Furthermore, they are a statement of intent - of what a software developer ought to strive for. The goal is working software, and to get that requires interaction, collaboration, and responding to change.
Those things on the right too often get in the way of creating working software. We've all been in projects where managers put pressure on the team because it deviates off their unrealistic "plan". That bad ideas get coded in because the phase to negotiate that was before coding began. That a effort is put into elaborate documentation that goes out of date the moment a system enhances it.
Even when going agile, the agile manifesto works to dampen the tendency to think any problem can be solved with more process. It can stop those in-grained waterfall tendencies from answering the inevitable "now what?" question.
Friday, September 2, 2016
The Switch to Git
Apart from 9 months using Rational ClearCase (and a couple of dark years using SAP WCEM), I've used SVN as a source repository. I even use it for my personal development projects. So I feel pretty comfortable in the SVN paradigm.
The strength of SVN is its simplicity. It's fairly easy for any developer to pick up and master its basic functions. That said, it has some pretty major shortcomings in a collaborative environment:
None of those are showstoppers, of course, but something definitely feels amiss when you have to use alternatives to a collaborate tool to do what the collaborative tool is for.
So as part of our switch to a more agile framework, we took on migrating from SVN to Bitbucket. Most of our team hadn't used Git before (myself included!), so one issue with doing this would be the reskilling of developers in a new way of collaborating. So we needed a good reason to disrupt what people are familiar with. And with Bitbucket, these are the advantages I can see:
This is not to say that all of it is easy. Git has a higher learning curve than SVN, and working with branches can be confusing. It's also hard to break out of the silos of development mindset that SVN encourages, thus robbing Git of the collaborative advantage it has over SVN. And no code tool is going to stop poor code from being committed.
That said, Git fits much better into the feature-driven development of scrum than SVN. It encourages keeping features discrete by their creation as branches, and keeps features from being prematurely deployed by keeping a master branch as the endpoint of development.
The strength of SVN is its simplicity. It's fairly easy for any developer to pick up and master its basic functions. That said, it has some pretty major shortcomings in a collaborative environment:
- Everything goes to trunk - All the checks that go into ensuring quality code happen after the code is shared. A bad commit can break the build and hold up deployments.
- Branching is difficult - While it's possible to branch in SVN, it's a non-trivial process that's largely avoided in practice.
- It encourages sitting on code - The flip-side of a central repository and the difficulty of branching is that developers tend to sit on uncommitted code.
- Collaborating in an unfinished state is tricky - Committing unfinished code is risky because it breaks the source of truth. It encourages workarounds like emailing code or zipping up entire repositories.
- Code reviews are after the fact - .
None of those are showstoppers, of course, but something definitely feels amiss when you have to use alternatives to a collaborate tool to do what the collaborative tool is for.
So as part of our switch to a more agile framework, we took on migrating from SVN to Bitbucket. Most of our team hadn't used Git before (myself included!), so one issue with doing this would be the reskilling of developers in a new way of collaborating. So we needed a good reason to disrupt what people are familiar with. And with Bitbucket, these are the advantages I can see:
- Branching is easy - Creating branches in Git is a trivial process. Switching between branches is a trivial process. Working with branches can avoid prematurely merging code before it's ready.
- Integration with JIRA - JIRA Agile tickets can be used to create branches, as well as linking to commits against a number. This is good for chasing history.
- Commits don't need to worry about breaking the build - When working in a branch, checking in code that breaks the build doesn't affect the "source of truth". So developers don't have to keep code on their desk for fear of "polluting" the master code.
- Collaboration is made easier - Since multiple people can use the same branch, there's no need to resort to emailing code between developers.
- Master can be protected - Bitbucket can be set up so the only way into master is through a "pull request", and that can be protected by setting rules about code reviews, as well as ensuring the code will successfully compile.
- Code reviews are post commit, but pre-merge - Bitbucket puts code reviews as part of a pull request, so with the right rules only code that passes review will be put as part of the build.
This is not to say that all of it is easy. Git has a higher learning curve than SVN, and working with branches can be confusing. It's also hard to break out of the silos of development mindset that SVN encourages, thus robbing Git of the collaborative advantage it has over SVN. And no code tool is going to stop poor code from being committed.
That said, Git fits much better into the feature-driven development of scrum than SVN. It encourages keeping features discrete by their creation as branches, and keeps features from being prematurely deployed by keeping a master branch as the endpoint of development.
Thursday, September 1, 2016
From Java Dev to Scrum Master
My previous workplace did classic Waterfall, which meant coming on and going to spec review meetings while waiting for the final green light to start coding. And I came on just before development was due to ramp up - in this case, about 6 weeks.
That was 6 weeks of reading elaborately crafted specs set out in a systematic way, and occasionally going to long tedious meetings where minor details were argued over. All this for what turned out to be a few weeks of actual development, before the long wait for testing to be complete and into production.
By the time I left, we were a few months into the design phase for the next project, with coding due to start just as soon as the specs were signed off.
The place I entered had just started taking on an Agile scrum methodology in software development. In the space of a month, I had gone to arguing over the minutiae of 100 page specifications to being asked to size stories based off a two line description and 5 minutes of discussion among the team. It was quite the culture shock!
It took a few sprints, but we slowly established a good team working ethic. When I came on, there was pessimism about whether the agile process could deliver anything other than more process for us to follow. But over the next 6 sprints, we made it work and got a high quality product out on time and with a good working relationship between business and IT.
So that brings me to where I am now. I've come out the other side of a scrum agile project as a scrum master for another. That means taking on a lot more of the organisational side of things, and consequently having a lot less time doing actual development. And it's meant spending time working out how to integrate the tools we have seamlessly into our workflow.
The way I see this blog is this - it's a way to collect my thoughts on the agile process as a whole. To write out some of the things I'm learning as I'm learning them, and to reflect on the practical realities of agile software development as I see it. I've been happy to wear the developer hat for close to a decade now, so will be an interesting journey for me to be immersed in a different aspect of the process.
That was 6 weeks of reading elaborately crafted specs set out in a systematic way, and occasionally going to long tedious meetings where minor details were argued over. All this for what turned out to be a few weeks of actual development, before the long wait for testing to be complete and into production.
By the time I left, we were a few months into the design phase for the next project, with coding due to start just as soon as the specs were signed off.
The place I entered had just started taking on an Agile scrum methodology in software development. In the space of a month, I had gone to arguing over the minutiae of 100 page specifications to being asked to size stories based off a two line description and 5 minutes of discussion among the team. It was quite the culture shock!
It took a few sprints, but we slowly established a good team working ethic. When I came on, there was pessimism about whether the agile process could deliver anything other than more process for us to follow. But over the next 6 sprints, we made it work and got a high quality product out on time and with a good working relationship between business and IT.
So that brings me to where I am now. I've come out the other side of a scrum agile project as a scrum master for another. That means taking on a lot more of the organisational side of things, and consequently having a lot less time doing actual development. And it's meant spending time working out how to integrate the tools we have seamlessly into our workflow.
The way I see this blog is this - it's a way to collect my thoughts on the agile process as a whole. To write out some of the things I'm learning as I'm learning them, and to reflect on the practical realities of agile software development as I see it. I've been happy to wear the developer hat for close to a decade now, so will be an interesting journey for me to be immersed in a different aspect of the process.
Subscribe to:
Posts (Atom)