For Data Collection, "When" Matters Just as Much as "What"

You’ve formed a great hypothesis and selected just the right change metrics. Now you begin to Collect the Data, right? Not quite yet. When you Collect the Data is as important as what you collect. In the first of our four phases of Data-Driven Change Management, Choose the Data, you have executed on the what. The hypothesis or question has been formed, and you have applied the “Minimum Set” Principle to get just the metrics that give useful data. You are well on your way to a robust and efficient dataset to measure your change, but the “Minimum Set” Principle also applies to timing. Gather the data too early, and it may be merely interesting, not useful (useful is when you would make a decision based on the data, while for interesting data, you would not). Gather it too late, and it’s looking in the rearview mirror at the turn you should have taken. Again, interesting, but not useful.

Just in Time

From Ecclesiastes 3:1, "There is a time for everything, and a season for every activity under the heavens,” to William Shakespeare’s Julius Caesar to a multitude of modern-day pop philosophers (I’m looking at you, Hollywood actors), timing is a key component of success. Good to know, but how do you do it? The phrase “How do you know…” provides the right framing. “How do you know…” questions provide insights into movement through the four phases of our change curve model. What we call the Change As Experienced Model:

Engage > Understand > Test & Learn > Adopt

Asking “How will I know the team is in the Understand phase,” for example, includes not only the markers of behaviors people exhibit but also the timing of those behaviors. Expecting to see behaviors that people understand the change just after it is announced (too early) or just before the change is implemented (too late) is not a useful framing. The data need to be collected “just in time”. This concept from Total Quality Management, specifically Lean Management, was famously successful in Japan, soon after World War II. Taiichi Ohno employed the idea at Toyota as part of the Kanban pull system. This was the heart of the Toyota Production System, which was instrumental in propelling Toyota to be one of the premier automobile companies.

In the context of Change, it is about getting the data to convert to information through analysis. For example, asking a tactical readiness question like 'I understand how the new communication flow works' during the Engage phase is useless. People don't even know why the Change is happening yet. Early on, you need to ask open-ended questions like 'My biggest concern is...' to uncover risks. Save the tactical readiness questions for the Test & Learn phase, right before go-live, when you can actually take immediate action on the answers." With Data-Driven Change Management, only get the data when it is useful for decision-making.

Many change models are normative, meaning they tell you what to do. Useful, but not complete. Since our Change as Experienced Model is an experience-based model, it tells you when behaviors are changing. The metrics give you the markers of the extent to which those behaviors are changing so that you can make adaptive moves if the behaviors are not moving fast enough to meet the needs of the change. They provide useful data because they arrive just in time to make a decision. 

Change Measurement Schedule

One of the more mundane yet strikingly powerful tools in organizational life is a project plan. The manifestation of that in this context is the change measurement schedule. In The Fifth Discipline, Peter Senge outlines a useful hierarchy that can help with forming a change measurement schedule.

1. Events (What has happened): are the visible incidents we react to. Decisions are made in the context of events, but focusing only here leads to reactive decision-making.
2. Patterns of Behavior (What are the trends): are the repeated actions that drive events. Focusing here provides perspective on why events happen, but does not give insight into how to impact them.
3. Underlying Structures (What drives behavior): are the mental models, rituals, and incentives that drive behavior. Focusing here provides the lever to change behavior and drive the change.

Change Measurement will pick up the patterns of behavior. It provides insights as to why the events are happening. The Change Plan needs to impact the underlying structures or at least influence them, since those underlying structures drive behavior. This is why Change Management is not only communication and training. It is a form of influence. The change measurement schedule needs to account for the underlying structures and the timing of their influence. If the change is happening in phases, then the change measurement schedule needs to align with those phases. It cannot be set on a regular schedule without regard to when the underlying structures of the organization, as well as the change, will impact patterns of behavior. It is also important to avoid the “Black Hole” effect of asking for data and never getting back to the people who provided you with such valuable data. A truly respectful measurement schedule doesn't just dictate when you will collect data. It also dictates when you will report the findings back to the organization. Scheduling this feedback loop builds trust.

Although this timing certainly applies to Self-Reported data (like surveys), it also applies to Observable and Existing Company metrics, the other two components of our three-tiered metrics approach. There is no need to gather metrics from the help desk if there has not been a change to an underlying structure that will drive a behavior change. It may produce interesting data to look at what questions the help desk is getting but not useful data.

If mapping out a schedule for a complex change feels overwhelming, scale your effort based on risk. Focus your data collection schedule on the stakeholder groups that are most critical to the mission, or those you expect will have the most difficulty. You don't have to measure everyone’s behaviors at every milestone.

Timing Changes Everything

Just as Toyota proved with the Kanban pull system, timing changes everything. When you align your data collection with the Change As Experienced Model (Engage > Understand > Test & Learn > Adopt), you stop guessing. You start knowing exactly when a behavior is shifting, delivering useful data right when you need it to make a decision.

Creating a change measurement schedule that respects this timing is the ultimate expression of one of our Core Four Philosophies: 

Leading change intentionally is simply a gesture of respect. 

When you carefully map out when to collect data, you actively protect your organization from survey fatigue and wasted energy. You prove to your team that you aren't just checking boxes, but that you are gathering insights to influence the underlying structures of the organization. By respecting their time, you are actively laying the groundwork to grow Joy at Work.

Your Next Step: Take a look at your current change initiative. Before you send out your next survey or pull another system report, stop and look at your change measurement schedule. Ask yourself: "Am I collecting this data just because it's interesting right now, or is it arriving 'just in time' for me to make a decision?"

Once you lock in your change measurement schedule and master the "When" of the Collect the Data phase, you are well-positioned for the third step of our Data-Driven Change Management process: Analyze the Data. That is where we take this perfectly timed data and convert it into the powerful information you need to lead your change to success