The lord of the delays

Intro

In the tech industry it’s common to have long run projects, whether code refactoring or new features. If a company wants to be competitive, it must innovate and control its budget. It is how it works. That sounds obvious to have visibility, and to help drive, we have the project management….

On the paper, everything is perfectly, and tightly controlled. I mean one task follows another, we can provide feedback and globally everyone is happy.

But wait, is this reality? What if something goes wrong?

How to manage a project that will drift ? Whatever you try, you feel it, you know it. And obviously you think of the consequences. I’m not talking about methodology, no, beyond that, how to face it ? How to deal with it from a purely human point of view ?

Before giving my point of view, let me share with you one of my experiences (true story)

A bit of context

Some years ago, I was working as a data team leader in a non-tech company. Not a big one, but big enough to have offices across Europe at least. Anyway, I joined this company because the promise and the challenges were really interesting (And I was bored in the company where I was).

Long story short, they needed to ingest a certain amount of data each day (something like 100Gb, heterogeneous) in a shorter time possible. They initially choose to create a home made orchestration in PHP, data temporarily stored in a tiny MySQL. Everything hosted on crappy and outdated on-premise instances with custom “frameworks”. Surprisingly, there were a tons of issues with this setup 😞, a lot of operational task, and barely maintainable.

Honestly, the promise was cool : almost unlimited budget and open to new technos / tools. I remember the discussion with the CTO at the time : “You can choose whatever you want, but we need to move out from this legacy / on-premise stuff, and more than everything it needs to work !“. Got it captain.

Well. With the team we worked on a new design. The exercise was really nice, and we concluded it with a lot of ideas. But nobody wanted a Big Bang project, including me. We provided a solution. Nothing fancy, an ETL with an orchestrator like airflow and Snowflake for the storage - In fact Snowflake was a company decision. Again nothing exotic, but still a complete revamp. After the official chiefs validation, we played the tasks estimation game. We wanted to be able to iterate and keep everything under control.

Baby steps

To roll out the entire existing solution, and I’m not talking about “only” load data in a big db, we estimated that it required 9 months with 6 full time employees (No one worked on Snowflake before). - Please keep in mind that the point is not to say it’s too big or not - And we ended the exercise with a marvelous excel sheet list of tasks. As a good team leader, after making sure that the team was completely aligned with it, I gave this estimation to the project sponsor to present it to the stakeholders. He came back with an incredible : “We have 6 months”…

The point of no return

What to say from that, ⅓ times less than what we have estimated, - and even if we were a bit generous - it’s not a challenge, that’s hunger games 😂. It was probably my biggest mistake, to not amend this decision. The reason behind, (or at least what was given to me) : “I [the sponsor] - want to show how performant we are and make a wow effect”. That’s a lot of consideration when we think that no one in the team already worked with Snowflake. And then the project started. With a lot of pressure of course. In the first month, we were 2 weeks behind schedule. Because of the project, the documentation, the conception, or simply because of the reverse engineering we tried to understand the business rules (or maybe everything together). That’s life, shit happens. But we fall into nonsensical and fuzzy explanations. The kind of thing that only makes sense for techies. Tension, friction and frustration increased, and quickly the methodology came on the table. The sponsor tried to micromanage and I stopped him. We started to take shortcuts to try to fit in the 6 months schedule but clearly it wasn’t doable. The further we went, the more I felt that the project itself was unsuitable. Things got worse as we discovered smaller details, like we needed to take more time to understand the data transformation. Quickly, the atmosphere became toxic, insults, tension, finger pointing. The sponsor decided to go into another kind of methodology, like he said : “deliver with a kick in the ass and make them work” - this time, unfortunately I wasn’t able to block it - After 4 weeks of intense battle, a limit was crossed. It went pretty far, too far (I guess that the only thing that kept us from fighting was the remote work). I concluded that It was too much for me (I’m not giving details on purpose). I resigned after the very last discussion we had, only 3 months after the beginning of the project and I still feel bad for the team I left. As I write, I still don’t know how it all ended.

Back to the present

Again I don’t want to talk about methodology, I think, whatever you want to use (scrum, kanban, v-cycle) it should be aligned with the business needs AND the team should adopt it (it will be a topic for another article). What we do agree on is the profusion of “tools” - I see you scrum masters, no, poker is a game not a prediction thing - and with these we can provide estimation, then having a budget etc. Again on the paper it’s neat.

I think the biggest challenge is accuracy (Yeah, I like to be obvious). It is difficult for stakeholders to understand the reality of an estimation and for an engineer to be extremely precise. It can be wrong, but not only because of numbers, it can depend on the point of view as well. We can easily make an analogy with construction building and software development and I suppose everybody has a story to tell about a similar experience.

My point is people oriented, how can we manage this in a humane way ? I’m not trying to find excuses, or anything like : “You should put everything on your blablabla board, and follow your burndown chart”. (The reflexion is the same about capacity, it’s not because you have 10 more people that you will deliver 10 times faster)

And so what ?

Stay close to your stakeholders: Keep the communication with them, explain the truth with simple words. An estimation can be wrong, but it’s even worse when you can’t explain it.

Know your topic and your interlocutors: Don’t try to talk about something you or they don’t understand. That’s a good way to lead to mistakes.

Don’t let someone else talk about your plan: You wrote your plan, you should know how to talk about it. Don’t let others do it for you (easy to say, we can’t really prevent people from talking).

Avoid toxic and micro-management strategies: Adding pressure on people is not a good sign. Neither is being pressured.

Think of an iterative process: Split in steps, be clear with objectives, and don’t promise the moon. Don’t be afraid to say that you need more time to design a solution and ask help for an estimation. Again giving something you are not comfortable with is not a good idea.

Don’t use too many tools: Even if the ultimate tool doesn’t exist. you have a risk of missing something important if you spend too much time updating your tools. (like team mood, progress or behavior). A tool should help you, not waste your time.

Protect yourself: You don’t have to suffer other people’s concerns. Yes, sometimes you can’t avoid people, because of hierarchical positions. But I encourage you to say if you’re not feeling safe or comfortable if needed.

Please don’t let people change your estimation: Either the scope change or maybe another round of discussion is needed. But, if with the team you agree on a charge, it shouldn’t be reduced (the opposite is true as well). At the same time, don’t overestimate. It should be a trust game.

Honesty, trust and transparency: I usually say that I have nothing to hide and that I trust my team. And I do. Be as clear as possible with your expectations and results.