Skip to content

Cell budgets

To give generalist and project cells autonomy to make decisions about the work being done, create and schedule some tasks independently, without approval, some of the budgeting is delegated to them. The idea is to ensure that someone close to the work can be taking more of the decisions regarding the internal budgets, someone who can have in mind all the details of the work. It keeps our efficiency high and allows a better use of our internal budgets.

For this to work, a cell needs a simple way to check that it remains sustainable while it takes these decisions. A cell is only sustainable when enough hours worked can be billed to clients, proportionally to the hours that aren't billable. Rather than requiring management approval for each new task or internal project budget, each cell ensures that it doesn't exceed a budget of 1h of internal/non-billable budget for every 2.5h the cell bills to clients.

The only exception is the DevOps cell (Serenity): It focuses on internal work, so it can't achieve the ratio of non-billable to billable hours mentioned above. To keep the number of non-billable hours generated by the DevOps cell from growing in an uncontrolled manner we keep its size limited to a small number of members. The current limit is 3; this number might change in the future if we find that more (or less) capacity is needed to complete relevant work at an appropriate pace.

Attribution of hours

Work is accounted per Jira project, no matter which team member logged it. For example:

  • If time is logged on a ticket with an 'SE' prefix, it counts towards 'Serenity'.
  • If time is logged on a ticket from a non-cell project (e.g. 'MNG'), it's counted only there.

Counting it this way allows cells to ask for help from other cells on the tickets without affecting sustainability of other cells. Thanks to this, members of a cell which is low on billable hours can find more work without negatively impacting its sustainability by reviewing tasks from another cells. However, counting it like this means that the owner of a task should agree to external members logging the time there.

Things to keep in mind:

  • Task budgets are all-inclusive: As stated in Time logging best practices, all work should be logged. If two people meet for 1h over a task, they log 1h each, for a total of 2h.
  • Time logged on tasks impacts the parent epic's budget (and by implication the cell's too) even if that budget is not strictly binding. So, when creating tasks or requesting somebody's time, think about where that time will be logged and what budget it will affect. This, and warning before going over a task's original estimate, will help everyone have a more effective and predictable workflow.

List of budgets managed by the cells

Hours from client epics (including fixed-price projects as well as ongoing maintenance and support) are billable. The rest isn't - like internal projects, forum discussions about internal matters and internal team meetings are non-billable. Not all the internal/non-billable budgets are the responsibility of cells; some are handled by management.

All cells

The budgets for which cells of any type are responsible are:

  • Cell management: Time spent fulfilling the cell management roles & discussing cell-related matters.
  • Conference: Time spent preparing and attending the conference.
  • Contributions: Making contributions to the platform for which we aren’t being paid by clients.
  • Learning: Time spent learning things, outside of the context of specific projects (almost every task requires learning something, that learning is part of that task's scope).
  • Meetings: Time spent in things like sprint planning or all hands.

Cells with clients

In addition to the budgets listed above, all cells that own at least one client are responsible for the following budgets:

  • <Client> instance maintenance: Each cell is responsible for maintaining its client deployments, including handling Open edX stable release upgrades and applying security updates when necessary. Maintenance accounts for individual clients usually have a monthly recurring budget, and cells must make sure that they don't exceed the total number of hours that the budget provides for the duration of the maintenance contract. (I.e., going over budget for a couple months is fine as long as the excess is compensated for in the following months.)
  • Overhead: Time spent in work for clients that exceeds a fixed budget we agreed to for the completion of an epic, and which will thus not be paid by the client.

Generalist cells

The following budgets are the primary responsibility of generalist cells. This does not mean that other types of cells can't help out with relevant tasks when the need arises - but they're not required to:

  • Open edX release upgrades: When a new stable version of Open edX is released, all Open edX instances that we host must be upgraded soon after. While cells are responsible for their own clients' upgrades (as explained above), a general upgrade epic will be created for each release in order to drive the corresponding code and infrastructure changes common to most clients (Ocim, etc.), as well as the actual upgrades of all self-hosted instances. Responsibility for owning this type of epic will alternate between generalist cells. They will need to take this into account in their annual epics/budgets planning, to include the budget for supervising one upgrade (or more, depending on the number of generalist cells that currently exist).
    • Note that it is still the responsibility of each individual cell, regardless of whether it is a generalist cell or not, to ensure that the upgrade work for instances belonging to its client(s) is done as soon as possible after the release: preferably within 1-2 weeks of the release, and at most within 2 months. This means that it'll be in the best interest of even a project cell to assist with the common upgrade work (assuming that it has an external client).
  • Prospects: Generalist cells decide how much time to allocate to prospect work - the more prospect work, the more billed hours coming in later on, so this is a way to invest to increase future internal budgets and cell size.
    • In cases where a project warrants the creation of a project cell to house it, all prospect work specific to it - a discovery and its associated budget, for instance - is to be moved to and managed by the new cell.

Generalist cells and DevOps cell

Lastly, primary responsibility for the following budgets is shared between generalist cells and the DevOps cell (Serenity). Again, other types of cells may help out with relevant tasks when the need arises but are not required to:

  • Ocim/DevOps: Improving our architecture, fixing ongoing issues, developing new features, etc.
    • The DevOps cell (Serenity) sets the priorities for work that uses these budgets and defines the scope of new developments, but the budgets for any epics that it delegates are to be monitored by the cell taking them on.

Cell budgets independence

The proportion of the time spent on individual budgets can vary per cell - that’s part of the decisions individual cells will be taking, with the decisions and monitoring of the budgets being coordinated by the person with the sustainability management role for the cell.

They are each proper to each cell, and they aren’t influenced by the other cells budgets. No matter what happens, each cell has 1 hour available for its internal budgets for every 2.5h billable hours logged. To keep accounting and coordination work sane, budgets can't be exchanged between cells.

Firefighting and Cell Budgets

Firefighting tasks can pose a unique challenge when it comes to cell budgets since in most cases these are not tasks that can be delayed. In the case of firefighting tasks you can keep the following guidelines in mind:

  • The task is related to our infrastructure:
    • Is the task urgent? For example, are one or more instances down? Is there risk of them going down any moment?
      • If so, whichever FF encounters it first should begin work immediately, irrespective of the cell that they belong to and any budget or sustainability concerns.
    • Is the task less urgent? For example, is there a pending certificate rotation in a few days?
      • If so, discuss and pass on the task to whichever cell has better sustainability numbers.
  • The task is related to a client:
    • Client budgets will often cover these types of tasks, so sustainability is not a concern.
    • Is the task urgent? For example, is the client's instance down, or does the client need to immediately scale their instance, or do they have some other urgent request?
      • It should be completed by whichever FF encounters it first, irrespective of the cell that they belong to. (This may be harder/impossible in case of more specialised clients like Yonkers or LX.)
    • Is the task less urgent? For example, did the client request a minor change, fix, etc. that needs to be completed in the same sprint that it was raised?
      • It should be completed by the FF from the appropriate cell.
  • The task is neither related to a client nor infrastructure.
    • These are rare, but if such a task comes up it should go to whichever cell has the capacity to handle it. In case multiple cells do, then we can take sustainability into account.
    • For example: One of our XBlocks needs a fix that is currently breaking on edx-platform:master, or a secret (password etc.) has been leaked and needs to be changed ASAP.

Note that Firefighting tasks are often small, a few hours at most and they will not greatly impact sustainability when dealing with many hundreds or even over a thousand hours. Often, the more important thing is to begin work immediately.

Rolling budgets

To account for the fact that there are sometimes variations in needs, or in volume of work, budgets are accounted for on a rolling basis. If a cell uses less budget one month, it can spend it in the next months, and if the budget is exceeded one month, it is to be caught up on in the following months. It is recommended to build a bit of budget excess, to be able to weather surprises without having to make cuts every time.

The budgets for the next month will be recorded in the epic planning spreadsheet, since that directs the time assignation for individual sprints. Updating the spreadsheet to include the expected volumes is something the epic management role would do each sprint, and the sustainability role reviews and is responsible for.

Management

As always, Xavier or Braden can still make decisions on those topics, and these decisions would still take precedence over individual cells’ decisions. But like now for the rest of the decentralized responsibilities, ideally we’ll limit this to as few cases as possible - it’s simply a way to ensure we have a way to efficiently handle issues where self-management isn’t enough.