Not all data are created equal

We live in a world awash in data. Most of us walk around with devices that are radiating and receiving signals from cell towers, satellites, Bluetooth, near field readers, and more. Organizations have inputs from clients and suppliers and operations. There is no shortage of data. For many managerial efforts, like Change, even more data is desired. We could send surveys, hold focus groups, check training scores, or do a myriad of other activities that would add to our piles of data, but should we? Getting all that data costs something, certainly time and money to collect it.

Often, we have so much data that we don't use it all. There are surveys that go unexamined, training metrics that never see the light of day, or qualitative data that we are unsure how to analyze.  All of this, because of a missed step. We have not taken the time to careful Choose the Data.

We are so used to having data readily available that we often miss thinking about its usefulness. The fantastic amounts of data may produce interesting insights, but are they useful? Useful in the sense that a leader would make a decision from the data. Often, a decision is not even possible, let alone useful. The solution isn't collecting more data. It is being more deliberate about what data we choose.

A useful mindset as you Choose the Data is to think of yourself as a Cultural Anthropologist or a Detective trying to understand the change. Success in those roles comes not from looking at everything, but in finding just the right things, the mindset of data that reveals the facts of the crime or the culture of a group. A detective doesn’t look for fingerprints everywhere. They check door handles and light switches to know who has been in the room. To make sure that the data you collect for a Change or any decision is useful, follow these three steps. Brainstorm > Reduce > Test

Brainstorm the Data

Start with the aperture as wide as possible. At this point, it costs only brain power and a bit of time to bring in as many data points as possible. Think about all the data that could help answer the question:

How will you know the change is progressing as intended?

This framing of “How will you know…” helps to keep your thinking wide yet still directed at the problem. It is also the beginning of a useful prompt for any Large Language Model (LLM) such as Claude or Copilot. These tools are fantastic as aids to brainstorm. Give the LLM the background on the Change, what successful adoption would look like, then ask it to brainstorm metrics that would help you know that the Change is progressing as intended.

At this point, the idea is to capture as many ideas as possible. Once you have a full panel of ideas that span different stakeholders, then it is time to edit. It will also be useful to consider different phases of change. The model we use in our consulting and courses is Engage> Understand> Test & Learn, and Adopt.

Reduce the Data

Ensuring a focus on reducing the data is an expression of one of our Four Core Philosophies. 

Leading Change Intentionally is simply a gesture of respect.

In this case, it means not throwing in the “kitchen sink” of all available data, which is not only time-consuming, but also demoralizing when someone on the team finds that their great effort to Collect the Data was of no value because it was never used to analyze the change.

The first place to start is to find the metrics that essentially cover the same thing and pick just one of them. Often, there will be metrics that are nuances of the same idea. It is unlikely that all of them will give meaningfully different views. So choose one. 

Then go up a level and gather the metrics by stakeholder. Ensure that there are metrics to cover all of those who will have to do something different for the Change to be successful. Now, you can go back to the LLM and ask it to review the metrics you have by stakeholder and point out any gaps. This is an effective way to uncover blindspots that you or the team may have had in the creation of the proposed list. Fill in those blind spots, if any, and then look for any other places of redundancy in the metrics. The LLM can help with this as well.

Test the Data

Here is where the separation of useful data from merely interesting data occurs. To do this, imagine the likely range of values that the data can take. For instance, if employee attrition is something to watch as part of the change, then consider the likely range of values. It will almost certainly not be 0 to 100%. These are conceptually possible but not all realistic. If current attrition is 5%, perhaps it could go as high as 10% and as low as 3%. Once you determine a realistic range, then ask this question.

Should the data show the highest value or the lowest value, what action would you take?

If the answer is that you would not take any action, then the data is merely interesting and not useful. That metric can confidently be removed because it is not going to spur a decision. 

You may find that this removes a significant number of metrics. Good! Those metrics will not help you. In fact, you may find that you have relatively few metrics left. Good! If only a few are needed, be grateful for the time and cost savings that will come from being selective.

Finalizing

To get everyone comfortable with the metrics you have selected as part of Choose the Data, consider conducting a Pre-Mortem. Popularized by Gary Klein, psychologist and research scientist, in a September 2007 Harvard Business Review article, a pre-mortem asks you to look forward and imagine the effort has failed. Then ask what all the mechanisms of that failure are. With that foresight, you can take action to prevent them. Carefully chosen metrics that provide useful data are part of that prevention process. Klein found that by framing failure as definite rather than asking "what could go wrong?", you give people psychological permission to voice concerns they might otherwise suppress to maintain team harmony. 

These well-chosen metrics will lead to useful data, which will answer not only if the effort is moving forward as intended but also if it is avoiding the issues that could readily derail it. The extra effort taken upfront to choose these metrics will pay significant rewards over the course of the change.