Welcome to Part 3 of my blog series on Real World Software Development Process. Warning: this is gonna be a long one, but a GOOD one!

In Part 1, I talked about the member roles and responsibilities in a software team. In Part 2, I talked about the environments and servers you will typically find in a software shop that exercises any degree of discipline. These 2 parts have kind of helped set up the stage for what I will cover in this blog post, which is my favorite of the 3 parts. So if you are new to the software industry or are not familiar with the concepts covered in Part 1 & 2, then please spend some time reading those first. Part 3 will make a lot more sense to you if you do that first.

Introduction

If you are reading this, it means you care about the process of developing software. But why do you care? Well…it’s not hard to come up with answers to that question. In my own words, software development is a very challenging, dynamic and disciplined process. Without a good process in place, there’s no hope of long term sustainability of the team members or the software. Having a process takes away lots of guess-work and streamlines tasks so that team members can focus on the core, important business logic instead of distractions or even harmful drama. There’s enough drama in life already. We don’t need more from work! 🙂

In this blog post, I am going to walk you through all the steps (that I can think of) involved in a real world software development process. These steps happen in chronological order for the most part. I will talk about each step in plain English with a good amount of detail (some more than others).

I want to emphasize the key phrase “real world” used in my blog title “Real World Software Development Process” because it is the main reason why I wrote this blog post series. I believe what I have written down in this blog post series on software development process is not something you can easily find anywhere else. I’ve been in the software industry for more than 8 years and I’ve never come across anything like this, at least nothing this comprehensive. That’s why I wanted to blog about this myself.

Sure, you will read articles, blogs, and books that talk about this topic, and a lot of them are very helpful, but a lot of times they only talk about a piece of the puzzle, or it’s got too much theory and not very practical. On the contrary, what I am trying to accomplish in this blog post series is to present a complete picture of the process with plain English and practical steps that have proven to work for me through experience.

Not convinced? Well, read on.

The List of Steps I’ll Cover

Here’s the list of steps I’ll talk about summarized here for you to quickly glance at.

1. Gather and analyze requirements.
2. Document requirements.
3. Create UI mockups & prototypes (assuming the project has UIs).
4. Create PBIs (and bugs if existing software).
5. Have project planning meeting with devs.
6. Analyze requirements and translate into technical implications.
7. Create high level software architecture (MSDN link)
8. Create low level, detailed code design
9. Write tasks and estimate man-hours for each PBI and bug in the sprint
10. Have sprint kickoff meeting
11. Software construction
12. Write unit tests
13. Do peer code reviews
14. Deploy to Dev/QA environment for QA
15. QA will test deployed software in Dev/QA environment
16. Devs fix bugs and make modifications according to feedback
17. Demo software internally.
18. Deploy to staging environment for final QA
19. QA will test again in staging environment
20. Devs fix more bugs possibly
21. Have pre-deployment meeting before deploying to prod
22. Deploy to prod & lots of testing
23. Rollback to previous state if deployment goes bad
24. Sprint retrospect meeting
25. Demo to key users and stakeholders
26. Continue to engage our users for feature requests and bugs
27. Plan for next sprint
28. — the process repeats —

For each step, I will talk about 3 aspects of it:

• Who: who is involved in this step?
• Why: why is this step needed?
• What: what will happen during this step?

These 3 aspects in each step give me a consistent framework of mind to talk about each step. You certainly don’t have to describe the process this way. This is just how I choose to do it.

Ok. Are you ready? Let’s get to it!

Note: I’ll be using acronyms throughout this blog post. Most of the acronyms are defined in my 2 previous blog posts in this series.

Gather and Analyze Requirements

• Who
  • PO, BA, architect, DBA/database lead, CTO. Possibly others.
• Why
  • we need to go through a process to figure out exactly what we want to implement. This is usually harder than it seems.
• What
  • probably several meetings with all relevant parties.
  • lots of engaging conversations during this step.

Document Requirements

• Who
  • PO, BA
• why
  • after talking is over, we need to write them down in precise and unambiguous words so that others (such as devs) can read and understand the requirements.
  • after lots of brainstorming and meetings, lots of great ideas are in people’s heads, but those ideas need to be concrete and put into words now.
• what
  • use an advanced text editor (such as MS Word), or may use a software to create these requirement documents to make them look more professional and easier to understand.

Create UI mockups & prototypes (assuming the project has UIs)

• who
  • PO, BA, UX/UI designers. This can happen at the same time as the previous step.
• why
  • ideas need to be translated into words, and words need to be translated into actual UIs.
  • the main reason is that actual software UIs are very time consuming and expensive to create. So when devs start writing code for UI, it’d better be pretty darn close to the final version. It’s ok and expected to make minor tweaks for the UI, but not major changes.
  • compared to actual UIs, mockups are a lot easier and cheaper to create and modify. They are also great for vision casting, brainstorming, fine tuning ideas, etc.
  • also, more often than not, the UI can dictate how the software is architected and designed on a high and low level, so that’s another reason why we want to spend the time to get the UI/UX right before devs start writing code.
• what
  • the PO, BA will collaborate with the UI/UX design team.
  • they will more than likely work on low fidelity mockups first using pen & paper, or white board. Nothing fancy. Just brainstorming UI ideas.
  • Once they have a pretty good idea of the general UI, the UI/UX team will use graphics design tools to create the high fidelity mockups. Once they are done, the outcome will be pretty close to the finished product.
  • if there’s already an existing UI/UX guideline, or existing software products that share a common UI/UX, then this step will be easier.
  • need to think in terms of the overall UX of the software, not just individual UI screens.
  • the design team usually creates both low and high fidelity mockups.

Create PBIs (and bugs if existing software)

• who
  • PO, BA. Possibly devs
• why
  • write down the big to-dos: major features to implement.
  • the devs will use these PBIs as starting points and create tasks for each PBI.
• what
  • PO, BA will use some kind of online project mgmt tool to create these PBIs.
  • this project mgmt tool needs to be web-based so that everyone can have access to it and see progress updates.
  • this project mgmt tool is also geared toward Scrum based projects.
  • For eg, we use TFS (Team Foundation Server) client that integrates with source control and Visual Studio.

Have project planning meeting with developers

• who
  • PO, BA, architect, devs, database devs, QA. Possibly: CTO, UI/UX.
• why
  • go over the finalized (or close to final) requirements with devs so devs can find out what they are supposed to accomplish.
• what
  • go over the PBIs, requirements docs, using the online project mgmt too.
  • usually also includes all the mockups that have been created so far.

Analyze requirements and translate into technical implications

• who
  • architect, devs, database devs. Possibly: PO, BA
• why
  • Now that the devs and architect know what they are expected to accomplish, they need to carefully analyze the requirements in order to fully understand the technical implications. In other words, they need to figure out what technical implementations need to happen in order to fulfill the business requirements. This stage in the development process is time well-spent before jumping into designing architecture or writing code.
• what
  • Devs and architect will need to spend time understanding requirements thoroughly. This involves reading through requirement docs, meeting notes, looking at UI mockups, etc.
  • There can also be a lot of back and forth discussion with business folks in order to fully understand requirements b/c sometimes documentation alone is not enough.
  • During this process, requirements might change due to wrong assumptions and technical limitations, or other scenarios. It’s much better to iron out these things now instead of later b/c it becomes a lot harder and more expensive to change later.

Create high level software architecture (MSDN link)

• who
  • architect. Possible: lead dev.
• why
  • based on the new requirements, we need to decide on the high level software architecture before devs starting coding.
  • this is the ground work and foundation for the software project.
  • The software architecture will affect many things, such as
    • what the code construction stage is going to look like.
    • code complexity.
    • software longevity and maintainability.
    • software UX.
    • timeline and cost.
    • scalability.
    • …the list goes on…
  • the architect will strive to get it just right the first time, but it’s a hard task.
• what
  • high level architecture consists of
    • utilization of server and network infrastructure.
    • selection of technologies: both software and hardware.
    • decomposition of system into modules, components, subsystems, interfaces, services, etc. These components are mostly abstract and need to be further defined.
    • layers or tiers for clear code execution boundaries and other reasons.
    • architecture and database design diagrams.
    • creation and design of assemblies, frameworks, libraries, and APIs.
    • long term software longevity and maintainability considerations.
  • architect will create the high level architecture based on the following
    • his/her experience,
    • understanding of requirements,
    • technologies and resources available
    • cost constraints
    • timeline
  • architect will then explain the architecture and design to devs.
  • if an existing architecture is appropriate for the new requirements, then this step will be trivial.

Create low level, detailed code design

• who
  • lead dev. Possibly: architect, other devs
• why
  • If high level architecture is the 10 thousand foot view on a map, then low level code design lets you see the street names.
  • High level architecture tells us what the high level components will be, whereas low level code design tells us how each component will be implemented.
  • High level architecture outlines how each component will interact with each other, whereas low level code design describes in detail the inner workings of each component.
  • Low level code design serves as a blueprint for devs. Devs can take the low level code design and start writing code, knowing that the end result will conform to the high level architecture.
  • Kind of like an instructions manual for devs.
• what
  • we usually can’t start working on the low level design without finishing the high level design first.
  • Generally speaking, we create the low level design by carefully analyzing all the functional and non-functional requirements of each use case and then design individual public classes/methods/interfaces to support those requirements.
  • Low level design is responsible for
    • utilization of coding best practices.
    • utilization of design patterns.
    • creation of public classes/methods/interfaces. (not concerned with private classes/methods)
    • interactions of classes within a high level component.
    • project file/folder structures.
    • code organization and namespacing.

Write tasks and estimate man-hours for each PBI and bug in the sprint

• who
  • all team members who have tasks to do, but mostly software and database devs.
• why
  • gives people a rough idea how long it might take for the sprint.
  • forces devs to break down each PBI into tasks and figure out what it will take to get it done.
• what
  • usually the devs will get together and go through each PBI and write tasks together.
  • during this time we will also assign each task to a dev. Usually related tasks are assigned to the same dev for obvious reasons.
  • usually we are not required to write tasks for bugs, but sometimes we can when it is helpful.
  • usually we also don’t have to assign each bug to a dev yet. Devs can assign bugs to themselves as they have time or prefer to work on a bug.
  • doing this together has several benefits. It will give every dev a chance to discuss their thoughts and contribute. It will allow best ideas to win, hopefully. It will help eliminate extremes in implementation and time estimations. It will give newbies a great chance to learn from more experienced devs.
  • It can be a fun time together as a team, but it can also get frustrating if you got a very opinionated and inconsiderate team member. This is a good time to build team culture.

Have sprint kickoff meeting

• who
  • PO, BA, software devs, db devs, architect, QA. Possibly: CTO.
• why
  • now that devs are done with writing tasks and estimating hours, it’s time to inform others.
  • this meeting also gives everyone a chance to catch any misunderstood requirements, poorly written tasks, etc.
  • after this meeting, PO will be able to set some timelines and expectations for the sprint.
• what
  • during this meeting the devs will quickly go through the tasks and explain why each task was created, and why it was given the estimated hours.
  • b/c most of the tasks are technical, and this is a meeting with non-technical folks, the devs don’t need to go into all the technical details about each task. Just some rough ideas so that people have a basic understanding of the tasks.
  • sometimes devs don’t even need to go through the tasks. They might just need to show the tasks are written and hours are estimated.
  • at the end hours are summed up and the length of the sprint will be decided. If the sprint is too long or too short, it will be an indication to add or remove PBIs/bugs from the sprint.

Software Construction

• who
  • software devs, db devs.
• why
  • after all that good planning, someone has to write the code and get it done! 🙂
• what
  • at this stage all the devs should have a good understanding of the high level architecture and low level code design, although understanding the high level architecture is not required. This is b/c if devs only have a good understanding of the low level code design without knowing the high level architecture, they can still do their job and get it done.
  • the devs will utilize all the resources available to them to write code on a daily basis. The resources include requirements docs, mockups, PBIs, tasks, PO, BA, architect, existing code, each other, and of course, Google.
  • usually if the dev team has several members, we can divide up the tasks and work in parallel to speed things up. For eg, the front end team can work on UI related tasks, and the back end team can work on server side tasks. As long as they adhere to the plan, which has clearly defined boundaries/interfaces/data contracts, the front end and back end code will work together seamlessly.
  • most of the time there will be lots of dependencies in code. For eg, you can’t complete class A until class B is written first. Or you can’t complete the data access layer code until the db devs are done with writing stored procedures (sprocs) code. In those cases what we typically do is that we would create a mock layer to mimic the real thing. For eg, we can create a mock data layer with hardcoded data and write our data access layer code against that mock layer. This will allow us to keep writing our code without being blocked by db devs. When the real data layer is done, then we can just swap it out, assuming you are coding against interfaces instead of concrete classes.
  • This is generally known as IoC principle within the SOLID principles. IoC is commonly implemented using Dependency Injection pattern. The DI pattern is a powerful and useful pattern in many ways, and all devs should use it whenever appropriate. So learn it if you haven’t already.

Write unit tests

• who
  • software devs.
• what
  • unit test: tests each “unit” of a software, usually it’s a public method/property of each public class.
  • usually unit tests code are in their own projects. For eg, for the data access layer project, there will be a corresponding project that contains all the unit tests code.
  • for each unit test, there are 3 things to do: arrange, act, assert.
• why
  • to make sure your code does what it’s supposed to do.
  • to ensure you don’t break existing software when you modify code or fix bugs in the future.
  • this has several long term positive impacts such as ensuring software quality, longevity, and gives devs confidence to make changes. Otherwise the software can become stale b/c devs are afraid to make changes and break things.

Do peer code reviews

• who
  • devs
• why
  • check each other’s code to catch bad code (wrong algorithms, inefficient code, hard-to-read code, etc)
  • share knowledge among devs.
• what
  • devs review each other’s code.
  • code review software is integrated with source control software. For eg, TFS.

Deploy to Dev/QA environment for QA

• who
  • software devs, db devs.
• why
  • We need a place for QA to test newly implemented software that dev team has been working on.
  • QA can test newly implemented software without affecting actual users.
• what
  • when the dev teams is mostly done with the sprint tasks or feels like they have added enough features to the software, they may choose to push the working software to QA server. The keyword here is “working software”.
  • usually the steps it takes to deploy to dev server are very simple b/c we want to make it easy for devs to frequently push updates to dev server so that QA can test frequently. These steps should be automated by development tools, which means the tools can do the build and deployment process all in one step.
  • For eg, in VS, it has a feature called “Web Publish” for asp.net web projects. Once I set it up, it is literally one click to build and deploy the software to our dev server. It’s very powerful!
  • We could use this tool to deploy to staging and prod environments as well, but it’s probably not a good idea b/c of several reasons. For one, only IT should have access to prod environments. Secondly, we shouldn’t make it so easy for devs to make changes to staging and prod environments, b/c if we do, mistakes will happen and our users will be affected.

QA will test newly deployed software in Dev/QA environment

• who
  • QA
• why
  • Quality control: catch bugs before deploying to actual users.
• what
  • manual or automated testing
  • QA team needs to have a very good understanding of requirements. That’s why they are involved with the sprint planning in the beginning.
  • they usually use PBIs, tasks, requirements docs, etc as a reference when testing.
  • they will document newly discovered bugs.

Fix bugs and make modifications according to feedback

• who
  • software devs, db devs.
• why
  • obvious reasons
• what
  • devs will look at the documented bugs and decide which ones they will work on. If a bug is related to the code they wrote, then they will most likely work on that bug.

Demo software internally

• who
  • PO, BA, lead dev, UX design team. Possibly: other devs, CTO, CEO.
• why
  • at this point the sprint is mostly done and we have fairly stable working software. Sometimes it might be a good idea to demo it to key internal stakeholders so they get an idea of what working software looks like.
  • This will allow PO/CTO/CEO to update external stakeholders on the progress of the project, and set certain expectations on the timelines and features of the software.
  • will give our internal team and external stakeholders confidence in our software.
  • helps build excitement and momentum about the upcoming release.
• what
  • the dev team will give a demo of all the new features of the software.
  • The demoed software is usually hosted in the Dev/QA environment.

Deploy to staging environment for final QA

• who
  • IT, software devs, db devs
• why
  • staging environment is supposed to be identical or very similar to prod environment, whereas the dev/QA environment is not necessarily, although it could be. Therefore, when we deploy to staging, it’s like we are getting a practice run of deploying to prod. This can help us catch issues we may not have encountered when deploying to dev/QA environment.
  • When our software is running in the staging environment, it is very much like running in the prod environment. So when we do testing, it is very much like testing the software in the prod environment.
• what
  • depending on the setup of the staging environment, IT may or may not be involved in this step.
  • if want this process to closely mimic that of deploying to prod, then the devs will prepare the software binaries and resources and hand them to IT.
  • If you want devs to be able to deploy to staging quickly, then IT probably won’t be involved in this step. Devs will deploy to staging in a manner similar to that of deploying to dev/QA environment.

QA will test again in staging environment

• who
  • QA
• why
  • last line of defense so to speak before software goes live.
  • There might be new bugs that emerge in this step due to the fact that the software is now running in the staging environment.
• what
  • QA team will test and document bugs.

Fix more bugs possibly

• who
  • software devs, db devs.
• why
  • fix more bugs discovered during testing in staging environment
• what
  • devs will look at the documented bugs and decide which ones they will work on. If a bug is related to the code they wrote, then they will most likely work on that bug.

Have pre-deployment meeting before deploying to prod

• who
  • PO, BA, IT, software devs, db devs. Possibly: architect, CTO.
• why
  • get all relevant parties together to prepare for deployment
• what
  • by this time all the testing should be done and we feel good about releasing our software into the wild.
  • in this meeting we will decide who needs to do what before deployment time, and when deployment will happen.
  • pre-deployment tasks might include
    • schedule downtime based on internal team and external users’ schedules.
    • inform all relevant parties of scheduled downtime.
    • IT: Set up necessary infrastructure (folders, servers, firewalls, etc).
    • devs & IT: build deployment binaries and other assets, create deployment folders, etc

Deploy to prod & lots of testing

• who
  • IT, QA, devs
• why
  • obvious reasons
• what
  • This depends on the software we are deploying. The process for deploying web, desktop, mobile apps are not the same.
  • For deploying web apps, we need to take things down and put up a “Under Maintenance” page. Sometimes we can deploy without taking things down and users will just all of a sudden see new features. (This can happen when we are deploying brand new things that don’t touch existing stuff).
  • IT: First thing – backup, backup, backup.
  • IT will update server resources/files/folders, etc with new ones.
  • DBA will update db with new sprocs/tables/data, etc.
  • Once new things are deployed, IT will bring things back up and testing will begin.
  • IT can manipulate network rules such that external users still see the “Under Maintenance” page while internally things are back up and we can do testing.

Rollback to previous state if deployment goes bad

• who
  • IT, DBA
• why
  • this is really the last resort. Sometimes things can go haywire when deploying to prod, and we can’t fix it within the scheduled downtime. We cannot afford to leave things down any longer. At this point the best thing to do is to roll back to previous state and try our luck another day. This is why backing everything up before deployment is so important.
• what
  • IT & DBA will replace the newly deployed resources with backups created right before the deployment.
  • Everything must be restored to its exact previous state in order to guarantee the success of the rollback.
  • This sucks!!!

Sprint retrospect meeting

• who
  • pretty much everyone involved in the process.
• why
  • To discuss what we did well and poorly, and what we learned from this experience.
• what
  • in this meeting we can talk about what went well and poorly. We can decide what we should keep on doing and what we should stop doing.
  • we don’t have to always have this meeting after every sprint. If sprints are similar from one to the next, then this meeting is probably unnecessary.

Demo to key users and stakeholders

• who
  • PO
• why
  • solicit user feedback.
  • generate excitement.
• what
  • PO will schedule meeting with key users/stakeholders to show them our latest and greatest.

Continue to engage our users for feature requests and bugs

• who
  • PO, BA, customer support. Possible: CEO, CTO.
• why
  • to learn from actual users what they like/dislike about our software.
  • may dictate what to do in the next sprints.
  • keep our software products and ecosystem alive and well. Survive and thrive!
• what
  • This can vary from business to business. Do whatever it takes to engage your users b/c they are the reason your company exists and you have a job.

Plan for next sprint

• who
  • PO, BA. Possibly: CTO.
• why
  • start thinking about what to do in the next sprint.
  • get a head start and give the devs something to think about.
• what
  • based on our product roadmap and user feedback, write down ideas and requirements for next sprint.

Wrap Up

Wow! I know! That was a lot to go over! If you made it this far, I applaud you. You must care about the process a lot. I hope you learned something here that can help you and your team improve your development process. If you are a noob to software development, I hope this series of blog posts did its job, which is to give you a somewhat comprehensive look at what is involved in the software development process from both business and technical perspectives.

I hope the information I covered in this blog post series help you keep the big picture in mind and keep you engaged as you wade through the maze of software development on a daily basis, as it can be the difference-maker for you as a software developer.

Thanks for reading and best of luck to you in your future development endeavors!