How to do estimates
"Software estimation is the most difficult single task in software engineering - a close second being requirements elicitation." ~ Fishtoaster
"Budgeting is one of the most difficult and stressful things for makers and consultancies. More commonly referred to as estimating, it’s often the least favorite aspect of their jobs. Few of us enjoy it or feel confident with it." ~ Carl Erickson
"Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law." ~ Douglas Hofstadter (see also Planning Fallacy)
When do we do estimates
We need to estimate tasks when we write a discovery document for a client or for ourselves. If the proposal in the document is accepted, we move its epic to "Accepted" and set a time budget and a timeline based on estimates from the discovery. Therefore, discoveries and epics are related through their estimates: tasks in an epic use the estimates and budget of the features the client selected from the discovery document.
To decide the type of discovery we do for each type of client, see our lead qualification and discovery pipeline:
How to do a ballpark estimate
We don't know if we're going to get the job, and the client doesn't know if they want to hire us, and isn't really sure about their requirements. It's time for a ballpark estimate!
Look at our list of epics and how long they took, and look for similar work, to help get an idea. If there are similar epics, one technique for doing a ballpark estimate is to pick 3-5 similar epics and take the average, add 15%, and use that as the ballpark estimate. Show your work by including this calculation when you deliver the ballpark estimate.
If the project requires changes to edX codebases such as edx-platform, then reach out to the relevant edX Product lead and/or Open Source Coordinator on Slack and ask them if they think edX would be willing to accept that sort of feature, and whether or not edX is planning anything similar or planning any changes that could affect the proposed new feature. There is no need for an in-depth upstream review at this stage, unless explicitly stated on the discovery/estimation task.
How to calculate estimates in a discovery and thus set an hours budget for an epic
- Talk with the client and/or review documents and ask questions until you have a good understanding of what they want.
- (If the project requires changes to edX codebases such as edx-platform:) Talk to edX upstream early about the project and the technical approach you have in mind.
- Scoping: make a list of the deliverables (if the client has not provided one) - this will be what is in the contract. Be only as specific as the client requires. Clone the Google Docs and Google Sheets templates linked below into new documents inside OpenCraft Shared Drive » Discoveries & Specs or a subfolder. Link to the docs from JIRA (do not put these items into JIRA directly at this stage as it can be hard to tell what the latest information is if it is found in multiple places).
- Make a list of all the assumptions you are making.
- Model the system. A model might be a written description or a diagram (e.g. using Google Drawings). (This step does not need to be done for projects where the architecture won't change or the implementation plan is well-established, e.g. an epic of bugfixes to an existing system.)
- (If the project requires changes to edX codebases such as edx-platform:) Get written agreement from edX about the proposed approach. (Unless we are unlikely to get the project, because that erodes our credibility with edX.)
- For each component in the model, create a high-level list of tasks, with a range for each one (e.g. "Create basic XBlock with student_view: 1-2 days"). Put these task descriptions into the Google doc of deliverables created in step 3, and the numbers into the corresponding Google spreadsheet.
- It can be easier (and more accurate) to estimate in days rather than hours.
- Double-check that all the requirements/deliverables will be covered by the proposed list of tasks.
- Every epic should include several optional stories that are included in the estimate but not in the requirements/deliverables promised to the client (examples: refactoring, tech debt clean up, UX polishing). Indicate which stories these are on the list of stories, which is internal. This can be skipped on small epics (<4 stories), but should be at least 5% of the time for larger epics.
- Include any required service from the list of standard work we offer (like deploying an Open edX instance), and get the estimates from that template.
- Include time for the following: (Account for these not by creating a specific point for each, but by factoring the time through each individual task or at least for each major part of the project, since we often quote each major feature separately and clients often choose to go ahead with some parts and not others.)
- Setup and onboarding
- Becoming familiar with existing functionality and code
- Coordinating with the client
- Manual testing
- Writing/expanding tests (typically double the time for the coding)
- Creating any applicable documentation
- Creating the pull request and sandbox
- Code review (which may involve setup, onboarding, and manual testing again)
- Addressing feedback from the code review
- Delivering to the client
- Addressing post-delivery feedback from the client and fixing bugs
- Upstream contributions (typically this takes a lot of time)
- CEO/CTO management time related to that client
- The time spent for initial discovery (this includes the current estimation process) + budgeting (e.g. epic planning)
- In general: the estimates should include anything that can require logging hours in the epic
- The end result should be a budget hours range (the sum of all the low estimates vs. the sum of all the high estimates, with the time spent for initial discovery/budgeting time added on top).
- The proposed budget will be submitted to the client, to ask if they agree or if they want to adjust the scope. If they agree, we will create new tasks in its epic.
Style of a good discovery document
We write most discoveries for the client, and this means that the language we use should match their level of understanding.
It's preferable to assume a non-technical audience and to avoid technical jargon (like "merged to the repo", "upstream", "SSO" etc.) unless we're sure that the client understands it (or if it's an internal discovery!). Even if the client is technical, it's often good to explain some concepts, because they may not know about Open edX development.
The client will directly read what you write, so it's better to mention the client name instead of writing "they" or "the client". And because it is produced by OpenCraft as a team, "we propose" is preferred to "I propose".
Also make sure you use well-written, full sentences and include some details when describing each task - these task descriptions will be used the quote.
Matching the client's budget
To avoid wasting our time estimating work that the client would then not be able to afford, it is important to try to match the client's goals with their budget. Instead of quoting what the clients ask for blindly, try to understand the underlying needs behind the requirements they give, and figure out alternate/simpler ways to accomplish this that they might not have thought of. Help the client understand what they can afford, and find an approach that will work within the client's budget (even if it's not the way they first thought they should do it).
When we don't know their budget, there will need to be some collaboration between the developer doing the estimate, Gabriel and the client. For instance, we may ask questions about the items that are big, to make sure the client expects them to be big. Start with some quick ballpark estimates that Gabriel can check with the client to see whether it's worth it, and suggest smaller ways to do the big items when possible.
Useful, quick read about this: The XY problem
Tips for smoother, more successful discoveries
We sometimes need to do complex discoveries within a short timeframe. A good example of this is when we answer requests for proposals that come from public organizations. Such projects often have a huge scope, no known budget, and a short submission deadline. In those cases we need to be extra careful to match or exceed the client's requirements, all within a reasonable budget. Here are a few tips to achieve this, and to make the discovery process go as smoothly as possible: We should always ask for a meeting with the prospect, and meet as early as possible (most public RFPs won’t allow individual meetings, but it never hurts to ask). Some RFPs include a pre-bid meeting. * If a meeting happens, we should send a summary afterwards to make sure everyone is on the same page, and to challenge any mistaken assumptions early in the process. * If we can’t meet, we should prepare a list of questions (scope, budget, etc.) and send them as early as possible in the process. We should always aim for the simplest and most affordable viable solution by default, but mention that we can offer more complex solutions if needed (and provide estimates for those as well).
Doing this should make things more clear early on, reduce internal stress, and increase our chances of matching the prospects’ requirements and budget.
Estimation worksheet templates
A full discovery produces a list and description of the tasks to implement, and estimations in hours. With fixed-cost projects, the individual estimates aren't sent to the client but we still use them internally. To make it easier to separate the client part and the internal part of the estimation, we keep them in different documents.
Start your estimations by duplicating the content of the following templates: