How to do estimates

"Software estimation is the most difficult single task in software engineering - a close second being requirements elicitation." ~ Fishtoaster

"Budgeting is one of the most difficult and stressful things for makers and consultancies. More commonly referred to as estimating, it's often the least favorite aspect of their jobs. Few of us enjoy it or feel confident with it." ~ Carl Erickson

"Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law." ~ Douglas Hofstadter (see also Planning Fallacy)

When to do estimates

We need to estimate tasks when we write a discovery document for a client or for ourselves. To decide the type of discovery we do for each type of lead, see our lead qualification and discovery workflow:

Support or consultancy requests

Leads often approach us with development needs, or simply need us to unblock them in their development work. Their requirements range from well-defined to fuzzy (in which case we help them define their requirements). We like to start helping leads with a small iteration (usually 1h of work), to assess the client's needs and give them a small piece of work to start with. If needed, we can then move on to a bigger iteration. The support and consultancy workflow defines how we handle such requests, as show below:

Estimating large projects

"…Systems requirements cannot ever be stated fully in advance, not even in principle, because the user doesn't know them in advance – not even in principle. To assert otherwise is to ignore the fact that the development process itself changes the user's perceptions of what is possible, increases his or her insights into the application's environment, and indeed often changes that environment itself."

– 'Lifecycle Concept Considered Harmful' by Daniel McCracken and Michael Jackson, ACM Software Engineering Notes, April 1982.

Software Development: How The Traditional Contract Model Increases The Risk Of Failure

We don't use fixed-price or fixed-scope for large projects. They don't work because the customer perception will change as the project evolves, breaking the original assumptions used for the estimated price and scope.

At the start of a project, usually only very high level requirements will be available. Different stakeholders will understand those requirements and their relative importance differently and both these will evolve as the project progresses. This means that there cannot be estimates of the size of the work in principle.

We recommend that clients follow an iterative approach in which we start with a minimum viable product (MVP) and then incrementally add enhancements to it. This approach delivers the best value to the client since it means the work done will incorporate learnings (both engineering and user-focused) from the earlier phases of the project.

Always clarify that a fixed budget or timeline does not imply that whatever details they envision now or will think of later will fit into it, but that they will need to work with the client owner to work out what features their constraints can accommodate.

For example, in some cases, clients have a fixed budget available or they will want a hard number that they can get approved for the project. Usually the requirements will already have been prepared by the client or there will be limited time available to develop them further. In such cases, it is important to have an understanding with the client about what the estimates mean, and the effect of adding more details to the requirements. Similarly for fixed timelines: our capacity constrains how many hours we can dedicate to a particular project, and so we need to work with clients to come to an agreement on what can be delivered and when.

To demonstrate to the client how the iterative approach will work, and how enhanced requirements affects the estimate:

• Prepare an initial ballpark estimate for each requirement.
• Choose only one or two key components or features with the client to detail what features or changes they envision. Don't estimate in detail all requirements.
• Prepare detailed estimates for the different versions of these key features, beginning with the MVP base, and create Feature Milestones by adding features and complexity iteratively to meet the full feature description.

Make sure to include detailed user stories for each milestone so that it is easy to differentiate what each version will be concretely, and this detail can be reused when creating implementation tickets.

• Use this detailed estimation for these key features to estimate the upper bound for all the client's requirements.

So if the total ballpark estimate for all requirements is the lower bound, then the upper bound of the estimate for all requirements is roughly ballpark estimate * (max estimate for the feature / min estimate for the feature).

• Document these calculations and details so that they can be referenced later.

Most client budgets will not be able to accommodate the full estimated upper bound; the sweet spot lies somewhere in between the lower and upper bounds. This process will help the client to understand that the work needed will depend on the level of complexity of each of the features and that is something they cannot predict or know right now but will need to tune as the project progresses to make sure the project fits into the budget that works for them.

It is our job as project/client owners to watch the budget usage against the desired feature set, and communicate with the client when decisions need to be made about how best to use the remaining time. Depending on the client, it may also be possible to adjust the budget as the project progresses. We will learn more about what the client wants, and the client will learn more about what we can do. We're building a relationship, and so it will change with time.

Every system has trade-offs. Our clients:

• Often need a detailed Statement of Work (SOW) that their organization can approve.
• Often have budget ranges or need budget estimates before work can start.
• Our team needs approval of the clients for all (non-technical) details since they are the ones who interact with the users.

This system helps with all 3 of the above. However, it does not help with creating great UX, which requires a holistic view of the completed system. For this, the project needs a UI/UX designer, who is also on board with the iterative nature of the project.

Create manageable deliverables

Large projects should be split up into deliverables of less than 2000h or 6 months (whichever will be reached earlier depending on the pace of the work).

Deliverables should be decided in a way such that all of the work at the end of it is production ready. This gives an accurate picture of the rate of completion of the project, and provides the client with visible progress towards their goals. It's much better to ship an MVP than to have a large feature half-ready, because it is usually difficult to estimate how much more work it will take to make an unfinished work production ready.

Complexity of the deliverables should remain steady, or increase slowly. Steep ramp-ups in the pace of work should be avoided; not only do they result in inefficiency but there may be bottlenecks that will only become obvious after commitments have been made.

Setting realistic timelines

High-level timelines given to clients should be relative to when the client completes their part of the specification or design. This will require splitting up the work into phases and clarifying dependencies explicitly. A good way to do this is to express the phases in the timeline as being "X months" long. If it looks like a client may not meet the timeline for a dependency, check with them (well before the deadline) and review the impact on the following phases.

The timeline that comes out of the discovery is usually the ideal timeline. The discovery writer should add a 25% buffer for the "unknown" unknowns (external blockers, team members needing unexpected off-time, needing to slow down to re-evaluate approach or process). This should be done after the discovery has been reviewed and approved (but before we send a quote to the client), otherwise "known" unknowns will tend to get counted under this buffer. Capacity planning at the cell level should take this buffer into account, even if initial planning at the epic level is more optimistic and doesn't explicitly allocate time for dealing with unknown unknowns before they occur. It is better to have a reputation to under-promise and over-deliver than the other way round.

And because team capacity also influences the timelines we promise, make sure you discuss the capacity required for the project with your cell's epic planning manager. Depending on other project commitments or sustainability concerns, we may need to adjust the number of people involved, which will affect the timeline.

Consider your team

When doing the discovery for a large project, the number of people expected to work on the epic should be specified and the expected onboarding time per person listed. Onboarding can then be split out among the deliverables in the quote.

When determining onboarding time, consider any new technologies that are being introduced and time required to review project discoveries. Also allocate sufficient time to prepare and document a devstack to reduce the per-person onboarding time.

• Limit number of people involved to a subset of the whole cell, who will commit at least 75% of their time to the project.
• Allow time to onboard one or two newcomers.
• For really large projects, a developer will need to work full time on the project with no other major project commitments, dedicating most of their time to the project's management, along with reviews and small development tasks necessary to keep a hands-on perspective.
• Ensure the client has staffed the product owner role, and can provide copy for the UI, or make sure to assign that role on our side, to one of the UX/product managers in the OpenCraft team.
• Ensure that the project has assigned a UX specialist to manage design and user experience - if the client doesn't already have someone on staff, assign the role to one of the UX designers in the OpenCraft team.

Developing a shared understanding of requirements

When managing any project, it is critical to form a shared understanding with the client on what their project goals are. Especially with large projects, those requirements are likely to shift as the project evolves, and that's ok. As the project manager, it's important to spend time with the client discussing their vision, so you can convey their intention to the development team. When the team and product owner share the same mindset, you don't need ultra-detailed requirements.

Anti-patterns to watch for

• The entire project is already spec'd out in great detail before any engineering work begins.
• Thorough review and iron-clad sign-off from all teams are required before work even starts.

Remember, be agile in your evolution of requirements for a project. It's okay to change user stories as the team builds, ships, and gets feedback.

– Product requirements documents, downsized, Atlassian

During the initial discovery phase for large projects, detailed specifications are not required. But these will need to be created for the various phases and features of the project, and so time must be allocated in the budget and timeline for repeated occurrences of this exercise. See Epic management: Capturing the technical requirements.

It's also important to factor in time for discoveries for each successive phase of the project, to reconnect with the client on their new understanding of what they want from this project, and to further refine requirements.

Components of a large project

Large projects aren't just scaled-up versions of the small projects we do. Our small projects generally build on the architecture that edX is designing and reworking – we usually fix bugs, add features, and write plugins. These additions sit on the periphery, and though they require design, it's at only a basic level.

But large projects are different – we need to architect them, build development and deployment infrastructure, onboard people, and have design phases for each major feature before development begins, as well as an overall plan for how these features will fit together. All of these things take time, which needs to be factored into any estimates and timelines presented to the client.

Large projects generally involve the following components:

• Software architecture: frontend, backend, API layers, build process.
• Operational architecture: setup and deployment should be automated (CD) for production environments, but also for end-to-end and pre-release or stage testing.
• Development environment: devstack must be reliable, automated, stable, and easy to set up to reduce onboarding time.
• Testing: automated tests must be run for all change requests (CI) so they must be fast and reliable, and test data must be added to the development enviroment to assist manual testing and QA.
• Internationalization: user-facing strings must be extracted for translation, and a strategy in place for displaying localized content.

How to do a ballpark estimate

We don't know if we're going to get the job, and the client doesn't know if they want to hire us, and isn't really sure about their requirements. It's time for a ballpark estimate!

Look at our list of epics and how long they took, and look for similar work, to help get an idea. If there are similar epics, one technique for doing a ballpark estimate is to pick 3-5 similar epics and take the average, add 15%, and use that as the ballpark estimate. Show your work by including this calculation when you deliver the ballpark estimate.

If the project requires changes to edX codebases such as edx-platform, then reach out to the relevant edX Product lead and/or Open Source Coordinator on Slack and ask them if they think edX would be willing to accept that sort of feature, and whether or not edX is planning anything similar or planning any changes that could affect the proposed new feature. There is no need for an in-depth upstream review at this stage, unless explicitly stated on the discovery/estimation task.

How to calculate estimates in a discovery and thus set an hours budget for an epic

1. Talk with the client and/or review documents and ask questions until you have a good understanding of what they want.
2. (If the project requires changes to edX codebases such as edx-platform:) Talk to edX upstream early about the project and the technical approach you have in mind.
3. Scoping: make a list of the deliverables (if the client has not provided one) - this will be what is in the contract. Be only as specific as the client requires. Clone the Google Docs and Google Sheets templates linked below into new documents inside OpenCraft Shared Drive » Discoveries & Specs or a subfolder. Link to the docs from JIRA (do not put these items into JIRA directly at this stage as it can be hard to tell what the latest information is if it is found in multiple places).
4. Make a list of all the assumptions you are making.
5. Model the system. A model might be a written description or a diagram (e.g. using Google Drawings). (This step does not need to be done for projects where the architecture won't change or the implementation plan is well-established, e.g. an epic of bugfixes to an existing system.)
6. (If the project requires changes to edX codebases such as edx-platform:) Get written agreement from edX about the proposed approach. (Unless we are unlikely to get the project, because that erodes our credibility with edX.)
7. For each component in the model, create a high-level list of tasks, with a range for each one (e.g. "Create basic XBlock with student_view: 1-2 days"). Put these task descriptions into the Google doc of deliverables created in step 3, and the numbers into the corresponding Google spreadsheet.
8. It can be easier (and more accurate) to estimate in days rather than hours.
9. Double-check that all the requirements/deliverables will be covered by the proposed list of tasks.
10. Every epic should include several optional stories that are included in the estimate but not in the requirements/deliverables promised to the client (examples: refactoring, tech debt clean up, UX polishing). Indicate which stories these are on the list of stories, which is internal. This can be skipped on small epics (<4 stories), but should be at least 5% of the time for larger epics.
11. Include any required service from the list of standard work we offer (like deploying an Open edX instance), and get the estimates from that template.
12. Include time for the following: (Account for these not by creating a specific point for each, but by factoring the time through each individual task or at least for each major part of the project, since we often quote each major feature separately and clients often choose to go ahead with some parts and not others.)
    • Setup and onboarding
    • Specifying tasks
    • Becoming familiar with existing functionality and code
    • Implementation
    • Coordinating with the client
    • Manual testing
    • Writing/expanding tests and fixture data (typically double the time for the coding)
    • Provide continuous integration (CI) to automate regression and feature testing.
    • Creating any applicable documentation
    • Creating the pull request and sandbox
    • Code review (which may involve setup, onboarding, and manual testing again)
    • Addressing feedback from the code review
    • Delivering to the client: provide Continuous Delivery (CD) to automate releases.
    • Addressing post-delivery feedback from the client and fixing bugs
    • Upstream contributions (typically this takes a lot of time)
    • CEO/CTO management time related to that client
    • The time spent for initial discovery (this includes the current estimation process) + budgeting (e.g. epic planning)
    • In general: the estimates should include anything that can require logging hours in the epic
13. The end result should be a budget hours range (the sum of all the low estimates vs. the sum of all the high estimates, with the time spent for initial discovery/budgeting time added on top).
14. The proposed budget will be submitted to the client, to ask if they agree or if they want to adjust the scope. If they agree, we will create new tasks in its epic.
15. In the case of discoveries which will result in a new project or require sending an official quotation to the client, do not include the estimated time in the discovery; instead, use the estimation worksheet templates. The template will help reduce the chance of inconsistencies if an estimate needs to be adjusted during the review.

Style of a good discovery document

We write most discoveries for the client, and this means that the language we use should match their level of understanding.

It's preferable to assume a non-technical audience and to avoid technical jargon (like "merged to the repo", "upstream", "SSO" etc.) unless we're sure that the client understands it (or if it's an internal discovery!). Even if the client is technical, it's often good to explain some concepts, because they may not know about Open edX development.

The client will directly read what you write, so it's better to mention the client name instead of writing "they" or "the client". And because it is produced by OpenCraft as a team, "we propose" is preferred to "I propose".

Also make sure you use well-written, full sentences and include some details when describing each task - these task descriptions will be used the quote.

Matching the client's budget

To avoid wasting our time estimating work that the client would then not be able to afford, it is important to try to match the client's goals with their budget. Instead of quoting what the clients ask for blindly, try to understand the underlying needs behind the requirements they give, and figure out alternate/simpler ways to accomplish this that they might not have thought of. Help the client understand what they can afford, and find an approach that will work within the client's budget (even if it's not the way they first thought they should do it).

When we don't know their budget, there will need to be some collaboration between the developer doing the estimate, the Business Development Specialist, and the client. For instance, we may ask questions about the items that are big, to make sure the client expects them to be big. Start with some quick ballpark estimates that the BizDev specialist can check with the client to see whether it's worth it, and suggest smaller ways to do the big items when possible.

Useful, quick read about this: The XY problem

Costs for shared infrastructure components

In general, if the needs of a new or existing client require adding, replacing, or modifying an infrastructure component that is (or will ultimately be) shared by multiple clients, we will do a technical discovery to determine an approach to billing that is appropriate for the specific case at hand.

The discovery shall consider costs for:

• Initial setup
• Recurring maintenance (i.e., developer time required to keep the infrastructure component running and up-to-date)
• Hosting (i.e., recurring costs for allocating necessary resources from cloud providers such as AWS, OVH, etc.)

Other aspects to take into account when coming up with a suitable approach for individual scenarios include:

• How much effort will be required for the initial setup? (In other words, are we dealing with a small, medium, or large project?)
• Did any other clients or prospects ask for the same or similar infrastructure changes?
• Will the infrastructure changes mainly benefit a specific client (or group of clients), or will they ultimately reach all of our clients? Will OpenCraft benefit from the change?
• What are the budget constraints of each client that expressed interest in the infrastructure changes?
• For scenarios where initial costs would be shared between OpenCraft and one or more clients (cf. below): What are OpenCraft's budget constraints? Are we currently in a position to contribute enough hours to cover our part of the setup costs?

To help bootstrap discoveries for shared infrastructure components and give clients an idea of what to expect, we include recommendations for approaching some specific scenarios in the following sections.

Scenario A: The client's needs match OpenCraft's plans

In this scenario, a client asks us to add, replace, or modify a component of our infrastructure in a way that aligns with existing plans of OpenCraft. In other words, we would add/replace/modify the component anyway (though the timeline for completing the necessary steps might be different if it weren't for the client's specific needs).

For example, we may know that we want to replace some component X with a new component Y, and the client wants to see this replacement happen, too, because it would improve the performance of their Open edX instance(s). Similarly, we might have plans to add some new component Z to our infrastructure, and the client may be asking for this addition to be made, as well.

In this case OpenCraft and the client should share the initial setup costs in a way that benefits both parties. As a result, the client would get their request fulfilled at reduced cost, and OpenCraft would be able to save some funds for other projects and/or infrastructure improvements.

Scenario B: The client's needs do not match OpenCraft's plans

In this scenario, a client asks us to add, replace, or modify a component of our infrastructure that we had not been planning to add/replace/modify ourselves.

To determine initial setup costs for the client, OpenCraft would reach out to other (existing) clients to see if any of them are interested in the proposed changes involving the infrastructure component in question. The initial costs would then be split between all clients who agreed to contribute funding for the work.

(This means that if no other clients besides the one who originally requested a specific change are interested in said change, the first client would need to cover the initial costs themselves: Even if other clients might benefit from the work later on, the first client is the only one who needs it now, so we think it is fair for them to pay for the acceleration of the work. This is very similar to what happens when requesting development of custom features that will later be upstreamed: Features that are funded by a single client are made available to the entire community for free; but in turn, the client gets the feature that they want, when they want it, and they also get to benefit from similar contributions of other community members in the future.)

Monthly hosting costs for new infrastructure components should be based on resource types. For example, in the case of Kubernetes, we could charge based on the node pool or similar.

Tips for smoother, more successful discoveries

We sometimes need to do complex discoveries within a short timeframe. A good example of this is when we answer requests for proposals that come from public organizations. Such projects often have a huge scope, no known budget, and a short submission deadline. In those cases we need to be extra careful to match or exceed the client's requirements, all within a reasonable budget. Here are a few tips to achieve this, and to make the discovery process go as smoothly as possible:

• We should always ask for a meeting with the prospect, and meet as early as possible (most public RFPs won't allow individual meetings, but it never hurts to ask). Some RFPs include a pre-bid meeting.
• If a meeting happens, we should send a summary afterwards to make sure everyone is on the same page, and to challenge any mistaken assumptions early in the process.
• If we can't meet, we should prepare a list of questions (scope, budget, etc.) and send them as early as possible in the process.
• We should always aim for the simplest and most affordable viable solution by default, but mention that we can offer more complex solutions if needed (and provide estimates for those as well).

Doing this should make things more clear early on, reduce internal stress, and increase our chances of matching the prospects' requirements and budget.

Estimation worksheet templates

A full discovery produces a list and description of the tasks to implement, and estimations in hours. With fixed-price projects, the individual estimates aren't sent to the client but we still use them internally. To make it easier to separate the client part and the internal part of the estimation, we keep them in different documents.

Start your estimations by duplicating the content of the following templates:

• Template for the discovery document (feature list)
• Template for the estimation spreadsheet (estimation of each task in the discovery document)

Estimation errors

If you find that the original estimate won't be enough and that the end of the task or epic is farther than expected, it's advised to take action soon. Check this forum thread¹ for ideas or discuss it there.

---

1. Private for OpenCraft Employees ↩

---

Last update: 2023-10-16