Yeti Lunch and Learn: Scoping and Expectation Setting

Video Transcipt

Rudy Mutter: Hi everyone! Welcome to my talk on Scoping: Expectation Setting. My name is Rudy. For all of you who don't know me, I'm the CTO and co-founder of Yeti. I also organize the SF Django meetup here at the city.

So the plan for today is kind of three things we are going to walk through. So one is a bit of the common knowledge and discussion around why estimating software development projects sucks and what’s the pitfalls of scoping and estimating, so kind of just like the basic, a really quick run through of why it’s so difficult. I’m going to talk of about our process that has evolved over many years here at Yeti and how we scope out our projects. And then kind of running through a laundry list of lessons learned over the years of doing this, of scoping then actually executing on the project.

So, a quick show of hands, who has ever had trouble estimating a project or who has under-scoped things on a project? Yeah, so pretty much probably any developer or designer or anyone who’s dealt with building products would say that they have. Funny meme and a lot of if you talk to people about this is, is they are called estimates, and yet they are treated as deadlines. Talk about the nature of that for us as a business in a second, about the duality there, but it’s understandable why a bunch of people have issues with it.

So a quick note, this talk is a bit from my experience, which is from an agency perspective. A lot the suppliers, also, if you’re in maybe a really large enterprise organization and also trying to scope out a product. But for us it’s a lot of new projects, so we’re scoping things kind of from scratch that haven’t been built yet, which is, has some differences than if you were adding new features to an existing product already. So kind of just a disclaimer there.

So first we are going to talk about some of the pitfalls. So number one is I put people and not necessarily developers, because people are overly optimistic when we start to talk about a project and implementing some kind of feature, we usually default to thinking that everything is going to go hunky dory and that what we estimate is actually pretty much the best case of how it could go. And sometimes if we are asked for the worst case, to estimate the worst case for implementing something, people often give the medium case because they also don’t want to think about how poorly things can go or can go wrong. So that’s the number one pitfall.

Hours and days are faulty and it also goes along with people being overly optimistic. It’s easy to say “Oh yeah, that is going to take me a couple of hours” or “Oh yeah, it’ll just take me a day to implement that.” So those are bad. We’ll talk about it more. But essentially that’s why in agile development a lot people use points and talk about the complexity of their work instead of talking about hours and days. So yeah nothing crazy there.

So I think this is one that’s really hard to wrap your head around a little bit. So there’s the things you know and then there’s the things you know you don’t know. And then it’s hard to quantify the things that you don’t know that you don’t know. When you go into a project, there’s probably plenty of things you don’t know you don’t know yet about that project but the whole thing is, you don’t know that you don’t know. So when you are scoping it, you just miss this entirely. You might think of plenty of things you know you don’t know about what maybe needs to get implemented, or some libraries or some new technology, but yeah then you get the point. There’s kind of unknown unknowns and it’s really hard to capture that in estimates if you already don’t know what the unknowns are.

Requirements change with knowledge. So often you estimate a thing, there’s a certain kind of an agreed upon scope of what you are estimating, and then once you get into it, that changes. Whether it’s because you start to build something, you learnt that there’s maybe a better way to build it. Maybe you start building something, you start testing it with users and you realize what you are building is not going to be successful. Maybe you start to build it and then some competitor of your company comes out and does something differently so now you need to pivot a bit what you are building. Hard to really account for this when you are first estimating the scope of a project. So that’s kind of like the summary of just a lot of the reasons why scoping is hard.

So, I’m going to talk next about our process and what we do. At the end of the talk we are going to collaboratively scope a mock-up, which will be fun. So there’s not going to be necessarily too many examples through this, it’s mostly going to be me talking and then at the end we’re going to do a little bit of exercises. So as a consultancy helping, we work with clients to talk about product strategy; what they are building, and then ultimately executing it on ourselves. We start in a jillion different starting positions with projects. Sometimes it’s just an idea a client has, and all they have to first give us information is conversation. Sometimes they make some kind of sketches or wire frames. Sometimes they’ve operated in bit more of a waterfall fashion and there’s already a bunch of mock-ups to be developed. Sometimes they have 10 to 20 page PDFs. Maybe there are requests for proposals and there are just specifications and documentation. We start with things all over the board. It’s kind of just the nature of what we have to deal with. So usually Tony is fielding this. From our clients specifically at a larger org these might be some kind of executive or other stakeholder. It could be a product manager or a designer coming to you with a slew of things they’d like you to say, “How difficult would it be for us to build this thing? How long is it going to take?” So we try to distill this in any way possible. We get what we get.

So what I normally do is I start to outline the big pieces. So I create an outline, I try to think about everything at a high level. Maybe we are talking about a mobile app, and there’s going to be a backend that’s going to hook-up with this other system that we need to build that’s more logic-focused. So there’s going to be that thing that’s also kind of a separate piece. So I start to try to outline what I see are the big chunks at first in the project. I then start to break those chunks down into smaller tasks. And this is where it can be hard depending on how much detail you have. But generally speaking what we found on projects is if you can’t break the tasks down into chunks that are maybe a couple of days, the biggest at most, then likely you are probably going to have faulty estimations. If you estimate some chunk of work is going to take a week to implement that thing, you are probably already wrong. Because you haven’t really thought it through far enough, into enough detail, to figure out what it’s going to be.

So here’s like a really simple example though of what we are talking about. Maybe some of part of a flow, of an app we are building is on-boarding authentication. Some really basic user stories here. And then so there are some points. I won’t dive too much into it but we are using points instead of hours or days here. And we create these spreadsheets with all of our big chunks, and then our tasks and the scoping process. Then we do some excel formulas and math down at the bottom to try to come up with an estimate of how many weeks we think it’s going to take to implement the project we are scoping. Usually once we have this spreadsheet, and I have put in a bunch of effort to try to understand what we are building, and try to break it up as far as I can, what the different pieces are, usually this is when we go back and we talk to the product-people, the stakeholders, our clients, designers, to make sure that anything we’ve identified, things that we haven’t been able to break down, we try to be diligent about asking as many questions and breaking it down as much as we can.

We come back and we revise. I like this one. We revise the budget and the scope of tasks we put together. We’ve been the most successful when someone else looks at it besides just me. So kind of, I put this meme here - would you carry - because a little bit at this point in the project, we might not have a lot of detail and I am just making up stories and putting points on it. And a lot of that is kind of in the ether at this point and there’s not a lot of definition. There mightn’t be no design yet, there mightn’t be no real specs. But to get the project started we have to get through this, we have to work in an area with a lot of ambiguity.

After this, going into a project and starting to kick it off, it’s important to identify what my or our riskiest assumptions were in putting that scope together. So usually as I’m going through and making these scopes I [inaudible 0:10:18] notes or flag. Like, this third party integration with this company, we weren’t able to get their documentation. We have no idea how it’s going to work. This is an area of concern. I really don’t know how much time it’s going to take. Or it might be like we need to use this new technology that we’ve never used before. I don’t really have the expertise to know how long it’s going to take and so again my estimates are probably the weakest in these riskiest areas. And so likely when we go into the project, development and design should work together on probably tackling these riskiest areas first, so that we can clear up ambiguity. And so we can get any kind of misaligned expectations or time out of the way upfront in the project. Often times, people like to get quick wins and they want to work on the easiest and simplest stuff first so it feels like we are making progress, the riskiest stuff get back-loaded to the end of the project and then we get in bad situations where we start going over our estimates. And it feels like it’s at the last hour that that’s happening in the project instead of being upfront about it and tackling those first. That’s a lesson we’ve learnt plenty of times.

We’ve tackled this several different ways at this point, and we still don’t know exactly the right way. But if you talk to some developers they always say like they put a two, three or four x multipliers and all of the rest of it. That’s certainly like an easy way to add buffer to your project plan. But in our situation and working with clients, I mean it’s not realistic to just say everything is going to take twice as long as we implement. The idea’s why some people might do that is because they know some of the things they’ve estimated might actually be quicker and easier and that some of the things are going to take exponentially longer than what they’ve imagined. So adding 2X to everything will create enough buffer for where things change or go wrong.

We try to add buffer. We try to make it more transparent on like where we’re doing QA, bug fixing, polishing, we try to add buffer or more points to the specific tasks that we are unclear about, that are the riskiest. But sometimes it’s difficult and we still make mistakes. I budget things and then we go and we do it and things will go the right way. So you should always add buffer. You should be careful about what you are adding. We can’t just add buffer to everything for the sake of doing it. It needs to be deliberate and understandable and your stakeholders, PMs etcetera need to understand why there’s some buffer here. Because you’ll probably get pushed back if you say I’m going to multiply all my estimates by three. That probably won’t fly with your team and your organization.

So finally in the process at the end of this is there might be real other deadlines, timelines, things outside of our control and other constraints, and also money as a constraint as well. So what we come to, and what we scoped can essentially either take too long or can be too many resources or too much money. So, next is deprioritizing or prioritizing the scope. Whichever way you want to look at it, it’s working with the stakeholders, the clients, the team to figure out alright we have these things that we have now scoped. We feel fairly confident that we are on the same page on generally what we are building. We’ve identified what’s the riskiest assumptions, what things are we totally not clear about that we are building, and it looks like it’s not going to fit in to essentially the constraints of the project. So it’s time to prioritize what needs to be in there, and try to essentially eliminate or push off the things at the bottom of that priority list for a future phase of the project. Or even, well best case scenario, we have don’t need this buffer room, and maybe we’ll get to some of this at the end of the project. But having shared understanding with clients, stakeholders and where your estimates came to.

So that’s kind of our process. The tools and stuff we’ve done and the way it’s worked has evolved a lot. As an agency with fixed price projects versus agile development it’s difficult, so I’m going to go from scoping and running projects and seeing what went wrong about scoping, just some general lessons learned and tips for what to do and what not to do.

So first one that came to mind is estimating doesn’t stop once the project starts. What we’ve learnt is we create these estimates. We start to design and develop the product. Inevitably tons of new user stories get thrown into JIRA.  We work on another features that come up and if we lose track of the net difference and what has been added or removed from the project based on the original scope and we lose control of seeing what the timelines are, how our estimates are stacking up against deadlines, things are going to go wrong. There’s going to be conflicts with stakeholders, your team, your clients. The project’s going to go off the rails. And so being diligent around noticing when you’ve finished some phase, some part of the project, new tickets are being created, things are being redefined, it’s time to re-estimate.

In recent memory at Yeti, we’ve been sticking to this, to our guns on this a lot more. And just making sure that we are being diligent about the project process being transparent to our clients. When something came up that we didn’t know about, our stakeholders didn’t know about, it’s new information for everyone so what are we going to do about it.

I’ve already harped on this one a little bit, but beware of third parties. So getting agreement quickly on what third parties you are using. Looking at the documentation of those third parties and assuming that whatever they are doing or using is going to be simple, that’s probably not the case. Checking out specifically their APIs, their SDKs, maybe talking to a support person at the third party. Getting all of that like identified and locked in as soon as possible is great. Something we do here sometimes is we have a spike, which is a ticket that goes in the JIRA for us that’s like a time-boxed, maybe be it half a day or a day, researching whatever is related to that third party.

Whenever we are working on a new project that’s going to use a new technology, it’s an area where we should be concerned in scoping, and setting expectations with the clients. It also might not be a new technology to Yeti or the team but it might be a new technology to a team member, so it’s good to keep in mind and try to think about who on your team is working on this project and how familiar are they versus you. Again it’s a little bit more about that in a second.  If we are starting off a project, and we are doing design and development, design never fits neatly inside your estimates. We’ve seen this on plenty of projects that we scope out a thing with a client or a stakeholder. But then once we get into the design process, there’s always new ideas, things that my development brain didn’t think about or account for that come up during the design process. And so again always be estimating the scoping. This is where really agile development fits into the scoping process and where they kind of overlap. So I think Wes is still working on here to account for it.

So estimate and account for bugs. There’s a little bit of it, I don’t know how much anyone cares anymore about should you add points to your bugs in JIRA and should you count them in their velocity? We do it. And I think it’s important because when we are building new products, we are not following this absolute pure agile methodology where every sprint isn’t done and you don’t take on new features until all the bugs in the product are fixed. I don’t know who is that diligent about that process and being like really pure agile. So we put points on bugs. Generally, people can get a sense of is this bug going to be tricky to fix or do we think it might be line of code change? Often we can be wrong but generally people have a good idea.  If a bug exists only in production on a certain person’s phone, and a developer cannot reproduce that bug on a computer, that’s probably going to be trickier one, then you should point it higher and account for it in your estimates. So again this is as bugs get created. We should be doing QA for other process and we should be accounting for them in updated timelines and estimates that we are making on the project.

Everyone has a different velocity makeup. So when I go and I estimate a project, a point of contention is that I’m assuming a certain person on the team can get a certain amount of points done every single week based on my estimates. My background is a backend developer. So when I have a certain bias, when I am estimating backend tasks versus front-end tasks, you have people on the team who are front-end or backend or full stack, you have people who are senior or junior, you might have people who know JavaScript or maybe they know React or they know Vue or Angular and this project is using React. So there’s a lot variables that also go into scoping about the team, and sometimes it’s difficult to know exactly the makeup of the resources on the team that are going to do the project when we are doing the scoping. That’s just another thing to keep in mind. Also like some developers are good at logic and some developers are good at maybe more like styling and polishing. Different people have different strengths and weaknesses and it’s hard to sometimes account for that in scoping when you are trying to make it more generic for a team.

Don’t overlook this handful of tasks. So sometimes it’s easy to… Sometimes stakeholders, clients, product managers will assume that your estimates include setting up analytics, setting up error monitoring, setting up maybe some other thing that you didn’t realize. Maybe it seems like an easy task on the outside but it’s still worth capturing it inside of what you are scoping. Even just the time to setup the code base, deployment, setting up a testing suite, doing continuous integration, continuous deployment all of that takes time. We are not reinventing the wheel but every new project needs some setup time allocated for doing that. If you forget to scope any of this in your plan, you all of a sudden might’ve just missed a couple of weeks’ worth of work.

So reset expectations constantly. If far we’ve been doing all the things that I’ve just talked about, that means that you should also be chatting with your PM, the stakeholders, executives, your clients and like letting them know what is happening. I think it’s… I have another slide in a couple that talks about this. But basically it’s easy, so maybe just let those conversations slide. And then that just puts you potentially more in the whole if things are changing and you are not resetting expectations.

There’s a whole another slew of bucket of things that you don’t think about often so you get towards the end of a project and ready to launch a new product, so how does this look on a smaller screen size? Are we building this to be cross-platform? Have we tested well and looks and works well on both iOS or Android? Do we care about supporting Internet Explorer? Is there a use case where users of this app might have poor internet wherever they are? Does this app need to work offline? Sometimes you as a developer might not have thought about or scoped out any of these things but another stakeholder might just assume that these are table stakes for some reason or another. This is again where scoping can just add on several weeks to the end of the project if this was unaccounted for upfront.

Recently, we noticed some things basically when scoping. We just forgot some simple things. So are we thinking about - if this is a mobile application - creating a privacy policy page and a terms of use in a bunch of like legalese pages. That’s not awfully complicated but still it might take someone some time to implement. We forget about the need to create a footer and a navigation bar. It’s pretty straight forward but sometimes when thinking about user stories, we don’t scope and create tickets related to this. Forgetting your password functionality, resetting your password. Maybe we need to setup email notifications and we didn’t think about it. That requires setting up a third party email notification service. It’s all simple stuff that’s been developed thousands of times by different teams, but it still takes time to implement.

Balancing optimizations. So we don’t want to pre-optimize what we are building. But when we think about and when we’re scoping, another thing we often don’t see is there’s going to be a concern when this is ready to go on the app store; that something’s going to take a really long time to load. Are we uploading big images? Do we need to cache this thing that we might be integrating with? Again, they fall into sometimes the unknown unknown category, so it’s just difficult to account for it.

So last one, which all of this summarizes up to is embracing healthy conflict and disappointing people. So I think there is a stereotype that developers are maybe a bit more introverted. I don’t think that’s necessarily the case but in general people don’t want to disappoint other people, and generally people don’t want to be having conflict constantly. So I would encourage everyone to really think about this and if they feel like something is going off the rails, or there’s something they missed when scoping and even though it might create some turbulence with stakeholders and clients, it’s important to bring it up as soon as you think about it. And not see, not realize something is missing and then just be like “Ah! We can just [inaudible 0:25:15] for a month until it’s the end of the project.” Maybe the client won’t think about it. All of these lessons learnt there are still things that maybe I personally don’t follow to this day every single time, so I’m not a saint by any means. It’s just a collection of things over time.

So we are going to do an exercise. Here’s a mockup of a product that we built. So some quick context for people that are not familiar. This is for a backend portal of a chatbot. So let’s say people have been using a chatbot, they’ve been asking it questions, the chatbot has been answering questions. People are then also asked did they get a satisfactory answer, are they happy et cetera et cetera? So there’s bunch of like statistics, and a dashboard and a training part of it. So we are going to collaboratively try to come up with user stories that people see on this page and I’m going to try and point out what I think people are missing or what I disagree with. Who wants to start?

AM: As a user I want to see the results of my stats from the satisfaction [inaudible 0:26:40] in pie chart format.

Rudy Mutter: So you are specifically just talking about the top part?

AM: Yeah.

Rudy Mutter: And then specifically in that one user story, what do we need to consider in actual development work?

AM: You need research and graphing algorithms.

Rudy Mutter: I’ll also say there’s two components to it. This is the running application. Yeah, so really there are two tickets to this scope. There’s backend providing the data for the graphs, then there’s the frontend tickets related it and Tony pointed out that there’s some assumption about how we are building those little pie charts and all those things.

AM: [inaudible 0:27:37 to 0:28:09]

Rudy Mutter: Yeah, we don’t need to say the exact points right now.  [Inaudible 0:28:11] is more picking up the stories, but you nailed a couple of other things. Search is a big can of worms, about how complicated of the search are we doing? Are we just literally searching some text fields in the database? It could be really straightforward; we could implement really quickly. But if we need to setup Elasticsearch and we are trying to do fuzzy searches and we also want to do predictive search where you basically auto complete what the user is searching and get them suggested results, it’s a whole lot more work. What else?

AM: How long [inaudible 0:28:43 to 0:28:45]

Rudy Mutter: So what’s some concerns about that?

AM: [inaudible 0:28:52 to 0:29:05]

Rudy Mutter: Agreed, time zones suck. So definitely if our user base is viewing this on the West Coast and the East Coast, for example, as there are going to say the right morning, afternoon or evening. We are not seeing it here but we are going to assume that there’s some kind of authentication, right? So it’s like we need to make sure that we’ve got the user data. What else? We’ve only covered like a third of the screen so far.

AM: [inaudible 0:29:33 to 0:29:40]

Rudy Mutter: Yeah, that button.

AM: [inaudible 0:29:41 to 0:29:58]

Rudy Mutter: Yeah, also what do you think happens when we click that button?

AM: [inaudible 0:30:01 to 0:30:15]

Rudy Mutter: Yeah, it’s not mocked up here so we don’t know for sure. But likely it’s going to navigate you, I think, to another screen at this point, where you can edit the answers to the question. What else?

AM: [inaudible 0:30:30 to 0:30:44]

Rudy Mutter: Is there some concerns about the logic that actually produce that data?

AM: [inaudible 0:30:53 to 0:31:04]

Rudy Mutter: Yup. We can assume there’s some other similar [inaudible 0:31:07] unanswered questions too. What else?

AM: [inaudible 0:31:15 to 0:31:20]

Rudy Mutter: What are the concerns of that one?

AM: [inaudible 0:31:23 to 0:31:47]

Rudy Mutter: On mock up, there’s usually a thing people might want. Also, do we have a graphing library that we’re going to use that’s easy enough to style it to make it look exactly like the mock up? So it’s a concern with the stakeholder, designer, client to be okay if we use a library that’s close enough to how styled in the mock up? So there’s one more missing here that was under the “let’s not miss the simple things”. Yeah, there’s a nav bar at the top in general. Anything else?

AM: [inaudible 0:32:29 to 0:32:39]

Rudy Mutter: I don’t know

AM: If there a forgot password?

Rudy Mutter: Yeah. And kind of we are looking at one mock up in the application, obviously. But there’s a whole slew of other… If this is a new product we are building, all of those setup miscellaneous deployments, really error monitoring related things all might apply as well.

AM: Pagination

Rudy Mutter: That’s a good one. What if there’s hundreds of unanswered questions? Pagination a good one that’s often overlooked by me.

AM: [inaudible 0:33:20 to 0:33:32]

Rudy Mutter: Yeah those are good points. I feel that it’s getting into scope creep territory. I’m the design [inaudible 0:33:37] but that’s what happens. You are also a designer and not a developer. Anything else or is everything covered?

AM: [inaudible 0:33:39 to 0:34:01]

Rudy Mutter: So just to recap, we went through some of the common pitfalls that I talked about. Why scoping is hard. Our process that’s evolved here and then a whole slew of lessons learned. I [inaudible 0:34:14] all of you, and we had the exercise. Does anyone have any questions?

AM: [inaudible 0:34:28 to 0:34:57]

Rudy Mutter: Yeah. So I think one part of the process would be to try get design’s voice more involved in the original scoping of the project because often it’s taken from a development perspective. So it’s difficult when people are working on projects, do people have time? Sometimes, like this week I had three different projects that I was scoping.  So it’s like is there enough bandwidth to get more voices into the original part of it? When to let me know that I’m missing something in the scope? But naturally no matter what, it’s going to hard to think of everything in just like 30 minutes of looking at that spreadsheet.  Once we get to the design process how can we work better together.

I think it’s still a thing we’re figuring out. I think we do some things good, which is checking in with development. Certainly it’s bad for design to work with stakeholders, show designs, design themes, and then once kind of stakeholders have been like “ Oh yeah, this looks great, I’m ready to build this.” And then development comes in and has to be like “No, we can’t do this. No we didn’t scope this. No this is going to take too long”. So I think it’s good that we have some check-ins. I think we could have more process or scheduling. Rounds like when development and design check in and it being like less like off-headed and more like, we check in twice a week on whatever’s a work in progress. And it doesn’t matter how finished or perfect it is yet, we sit down twice a week on scheduled days and development, design just chat through what’s been designed so far. Probably running design through scoping documents in more detail and maybe giving a lot more context around them before diving in would also be helpful.

AM: [inaudible 0:36:45 to 0:36:50]

Rudy Mutter: The bigger the problem for scoping, the more often we’ve been way outside. I feel generally speaking, if I scope a thing that comes to an estimate that’s going to take two, three or more months, chances are we are going to miss the mark, historically speaking. When we’ve started to estimate things that are more like sub two months, we have not been missing deadlines as badly. Generally speaking, we always probably end up, historically, missing our original estimates by at least a week or two. But it just gets exponential once we are trying to estimate a thing that’s like several months long.