free html hit counter
Posted on: Friday, November 4, 2022 by Rajiv Popat

Problems Needing Binary Tree Searches Are All Around Us.

The other day someone asked me why I was going through masters curriculum at this age. Their argument:

Most of what you are learning is theoretical and useless. What’s the point of revising data-structures at this point in your life?

Then last week, someone at work was working on a specific problem. In the last release they had 21 check-ins and one of the check-ins had introduced a bug. The team was trying to find which check-in introduced the bug.

To find the erroneous check-in, they were applying each check-in one at a time to the old codebase and then running the application to see if the bug could be reproduced. When I heard that they were doing that I spontaneously responded with:

You’re trying to use linear search algorithms to solve a problem that would be solved much more efficiently and with much less time complexity using a simple binary tree search.

What I meant was that instead of taking each check-in one at a time applying it and then running the code, the team could have just taken the first 11 check-ins together, applied them on the server in a single shot and then tested. Either ways, if the bug was found or not in the first 11 check-ins, we would know if we should look at the left side of the tree or the right side of the tree and with that one attempt, we would have eliminated 10 tests right away. That’s 10 times of applying check-ins, firing a build, running the code and trying to reproduce the bug.


Put simply, you merge 11 check-ins in a single shot and run. If you find the bug, the bug lies in one of the check-ins between check-in 1 to check-in 11 and if you don’t find the bug you know the issue lies with check-in 12 to 20. And then you repeat this process depending on which side of the tree (1-11 or 12-21) the bug lies in. Exactly how the binary tree search algorithm works.

During the times of Covid a collogue of mine had pointed me to an article that showed how you can apply binary tree search to do effective Covid testing and save a bunch of testing kits. It’s the exact same logic. If you have 100 blood samples, mix the first fifty samples and test. If you don’t find Covid in the mixed sample you’ve just saved yourself 50 test kits. If you do find Covid, mix the first 25 samples and do another test. Again, binary tree search.

Of course we don’t hand write code for binary tree search in the real world when we are coding these days. Most modern day languages provide us out of the box search functions which use the most efficient algorithms under the hood.

But that doesn’t mean we can’t see binary tree search as a thought construct. Real world Binary tree search based use-cases exist in our day to day life, and identifying them, helps you solve these problems way faster, even when the problems have nothing to do with writing code. Problems that benefit from binary tree searches, are all around you. The real question is, can you spot them?

Oh and yes, there is a point of revising some of your data structures at any point in your life because that knowledge itself is not as useless or impractical as you might think it is.

posted on Friday, November 4, 2022 9:47:07 AM UTC by Rajiv Popat  #    Comments [0]
Posted on: Thursday, November 3, 2022 by Rajiv Popat

The Art Of Knowing That You Don’t Know.

An old Chinese proverb reads:

He who knows not and knows not that he knows not is a fool, avoid him.

In reality however, most of us, don't know. And we don't know that we don't know. What a professional phycologist would call, “lack of insight”. Putting it colloquially, when it comes to skills and competence, most of us suck. And we don’t know that we suck.

And why do we lack insight about how good we are at most things? Because the neural circuits required to become good at a craft are the exact same circuits required to qualitatively judge how good you are at that craft.

Put simply, if you don't have the brain circuits to draw well, you also lack the neural circuits to help you evaluate just how bad you are at drawing. This also means that if you learn just a little bit of drawing, you start believing you know a lot about drawing and your brain starts to think of yourself as an artist. This happens because with your limited knowledge of drawing, you can only see painting with limited scope of putting the brush to paper and you still don't know the intricacies of the craft.

As you spend years drawing, the nuances of the craft open up to you and for the first time you start realizing the vastness of what you don't know. The irony here? It is only when you become a decently good artist, that you realize that you are a really bad one. In the chart below for example, you don’t reach the “I know nothing about this” realization till you are substantially competent at a craft.

dunningkrugerskillgraphThis is why as people become better at any craft their self rating in that craft keeps coming down. The less you know the craft, the higher you rate yourself in it. The more you know the craft, the more you know what you don’t know and the less you rate yourself.

There is a name for this. It's called the Dunning Kruger effect.

It's why managers with literally no organizational skill, can give you a long lecture on how you should organize your tasks better. It's why CEOs and founders, give coding advice to programmers who have coded for two decades, without blinking an eye or without realizing the inherent underlying irony and humor in the entire situation. It's why the most unproductive guy in the room usually goes around telling everyone how unproductive they are.

The effect may seem funny at one level, but ignore it and it has the potential of ruining companies, relationships, careers and even destroying lives. Its why the movie The Big Short (loosely based on the book which in turn is based on real life housing industry collapse) begins with the caption from Mark Twain:

One way to overcome the Dunning Kruger effect, is to have actively develop insight of how we can all fall prey to the effect. Mindfulness, and simply accepting that the effect is a scientific reality and happens to all of us, is a good starting point.

For example, I had driven for few years in the US; but I recently learned how to drive in the streets of India and after a month of practice, I genuinely started believing I knew everything my driving instructor knew. Then one fine evening, I had to brake a manual stick shift car on a hill slope without using hill assist. That’s when I realize how misguided my evaluation was on how much driving I really knew.

Another approach, is respecting results and payouts. Put simply, don't pass judgements on working styles of people who are more effective than you and particularly those who are paid more than you are in a given field.

The simplest way to start this practice, is by stopping to give commentary on how that expert soccer, baseball or cricket player ‘should have’ hit the ball, next time you are watching a match and they miss scoring.

Then take the concept to your professional life.

For example, stop passing remarks on how your multi-millionaire CEO should run his company and instead try to learn from him on how he runs his company and makes millions.

Working with an expert designer? Don't tell him to change the color of your website to light blue because dark blue just doesn't "feel right" to you. You feel you have a sense of esthetics, but in reality it's just Dunning Kruger effect at play, making you think that you are a designer. You’re not. You know nothing about the science of esthetics. Something your designer has probably read and practiced for years.

Instead, ask questions. Why did he pick dark blue? What was his rational? What basic rules of designing has he used. Practice humility. Learn from him rather than trying to teach him the craft he has already practiced for multiple decades and you know nothing or very little about.

Insight, humility and genuine curiosity are three biggest weapons you have against the Dunning Kruger effect and given how dangerous this effect can be to your professional and personal growth, I suggest you use all three.

posted on Thursday, November 3, 2022 2:12:36 PM UTC by Rajiv Popat  #    Comments [0]
Posted on: Friday, October 21, 2022 by Rajiv Popat

The Fallacy Of Being Independent.

Overheard someone say this on a call in a café:

Why should be depend on or work with them? As a team we’re not dependent on other teams!

Even at an individual level, the Idea of a ‘self made man’ (or woman) seems to be catching on. Men and women hold their head highs when they proudly proclaim:

I’m not dependent on anyone! I don’t like depending on others. In my entire life, I’ve not taken any favors from anyone.

Even though these statements provide a sense of accomplishment, the fallacy in the idea, lies in evolution.

A few million years ago, when we were nomads, much like elephants or monkeys, we sapiens, lived and moved in herds.


And for some reason, if you were ousted or out-casted from your tribe and the herd moved on without you, no matter how powerful you were, you would, in all certainty, be lion lunch within a day. Your chance, of surviving in the wild for more than a week, without ‘depending’ on others in the herd was minuscule at best.

Everyone understood the significance of being dependent on others in the herd, we acknowledged that dependence and we reciprocated by acting out our responsibilities and respecting the older and more powerful alphas in the group. In return for the protection they provided us, we provided them with respect, support and our protection; the interdependence loop was closed and we made it thus far in-spite of having rather weak bodies compared to any other predators in the wild.

Then came in modern media – with stories of self made men and women who don’t take favors from others and are not dependent on others. The stories gave us a dopamine spike and soon, we became echo chambers for those messages.

Dependence on each other is what fundamentally defines any relationship; be it personal or professional. I’m a programmer, you depend on me for code and in return I depend on you for my very survival and food. If we work in a team or are in a personal relationship, the dependence on each other is even higher.

If your claim to fame is that you’re ‘self made’ and that you’ve never been dependent on anyone; well, let me just say that if your ancestors in the savannahs thought like you do, you wouldn’t even have existed to have that chain of thought.

Celebrate dependence. Ask for favors. Be shameless about how dependent you are on your loved ones, your team, your company, your clients and everyone around you. And then, reciprocate. Because without interdependence, we never stood a chance. We still don’t.

posted on Friday, October 21, 2022 10:44:23 AM UTC by Rajiv Popat  #    Comments [0]