Agile methodology has transformed the process of software development to be iterative, where requirements and solutions evolve through collaboration between self-organizing cross-functional teams. The ultimate value in Agile development is that it enables teams to deliver value faster, with greater quality and predictability, and greater aptitude to respond to change. We call these “quick wins”.
This same methodology can be applied to data governance or, really, whenever an activity can be clearly identified, defined, prioritized, broken down into releases and each release further dissected into smaller tasks known as sprints.
Provides the framework behind delivering business value to the right resources at the right time.
Enables an organization's ability to adapt to changing needs and perform cost benefit analysis on their data governance efforts.
Establishes quantitative and qualitative value of data governance, driven by accessible and trusted data.
But let me ask you this, is agile governance the right approach for you? Before I can help you answer that, we need to look at the conventional or legacy approach to data governance.
Legacy Approach to Data Governance
We have found that many organizations follow a linear path that begins with creating a data dictionary, defining data domains across the enterprise, and appointing a data steward to run the entire process. Those are all important to data governance, but the approach creates bottlenecks with the data governor, data governance council, data steward and stakeholders upfront.
In the short term, top management’s decision to initiate the set-up of data governance offers no added value to field teams and may even be seen more as a disturbance or an additional constraint than an urgent need. With this asymmetry of input versus output, it’s not hard to see how the initiative can quickly run out of steam before it has taken off the ground.
Agile Approach to Data Governance
What is agile data governance? We think our friends over at data.world said it best.
Agile Governance is the process of creating and improving data assets by iteratively capturing knowledge as data producers and consumers work together so that everyone can benefit. It adapts the deeply proven best practices of Agile and Open software development to data and analytics.
We start by having the customer identify small, bite-sized initiatives that hold strategic importance for them. From there, we put these through multiple iterations to perfection then build on the successes to embark on progressively more complex initiatives. For example, start with deploying a data catalog to stop your data lakes from turning into data swamps. That way, the data governance council can keep all stakeholders abreast of decisions, milestones and wins while keeping work inputs focused and limited.
The business benefit is realized much faster and thus generates even more data governance enthusiasm among stakeholders, getting this buy-in is key to perpetuate the maturity of your Business Intelligence Program.
I would be remiss to not point out that a common pitfall of agile data governance is that due to the small scale of focus initiatives, there’s a risk that the decision-making process will be inadvertently confined in a silo. However, this can be overcome if the entire program is infused with a ‘big picture’ mindset from the onset.
Determining if Agile Data Governance is Right for You
This is simple, have you ever heard one of the below statements come from one of your business users?
- “Endless emails to find the right person who has or knows about the data that I need”
- “I get multiple answers for the same business question”
- “I need to combine these datasets and have to wait for IT to help”
Or maybe you have heard one of these statements from your IT team.
- “Everyone comes to us asking for data”
- “We don’t know who has access to what data”
- "We don't have a way to share the data we have"
- “If we make a change to the data, we don’t know what it’s going to impact upstream”
- "The data in our lake (or anywhere) is a big mess” We like to call this a data swamp.
If you can say “yes” to any of the above, then you are a great fit!
Building the Foundation
So now, assuming you are a fit, let’s look at what an effective data governance program does.
- Provides the right resources to deliver the right business value at the right time
- Determines what to govern and establishes business value driven by governance
- Ensures that the right people are represented and creates accountabilities
- Articulates decision-making and communication processes for the roles, responsibilities, accountabilities, ownership, and decision rights needed to govern
- Adapts to meet the organization’s changing needs
- Represents the governance framework that isn’t a hierarchical
Those are the why for building a data governance program. But the how of building your data governance program, is where methodologies and frameworks come in. Data governance frameworks can be close-knit or loosely structured, and both can work. The suitability of the different approaches to data governance depends on the culture, maturity, objectives, structure, size, complexity and technology of the organization in question.
To get it right from the get-go, the foundational phase of a data governance initiative must include multiple discovery sessions to identify opportunities, quantify business value, identify stakeholders, assess sentiment, prioritize focus areas, develop goals and define a data governance model roadmap. To implement this foundation, we help customers define strategic objectives of data governance, educate stakeholders, obtain senior management buy-in and designate data stewards.
Once these initial steps are complete, the organization’s data governance council can make an informed decision as to whether to go with a highly or loosely structured framework. A less structured approach would have a loosely knit team of individuals who systematically work the roadmap to accomplish the data governance goals. The highly structured team would be supported by dedicated roles as well as clearly-defined tools, templates, and processes. We recommend the latter for more enterprise-wide efforts and the SME Team can help support open roles.
The establishment of the overarching enterprise-wide data governance best practices framework is the rock upon which your entire data governance program rests upon. It identifies a data governor, the data governance council (headed by the data governor), and a steering committee including people from multiple lines of business. This is key to obtaining buy-in and to ensuring the adoption of standards, processes, and guidelines at the tactical level of your organization.
One common misconception is that an agile governance model eliminates or downplays the need for traditional roles such as data governance lead, data owner, and data steward. Instead, it encourages you to re-imagine those roles and to embrace new ones.
Agile governance thinking may also contribute to more granularity in certain roles. For example, the traditional data steward may be subdivided into business data steward and a technical data steward (traditionally known as a data custodian). These steward roles work hand in hand to ensure business processes are supported by technology infrastructure, with the technology data steward recommending improvements in processes based on better and more creative technology usage.
We recommend identifying explicit stakeholder roles, such as data consumers, data producers, and other participants, even when they may not be formally engaged in governance activities. Their participation is truly important and should be recognized.
Teams should establish a meeting cadence where people regularly come together to identify issues and create and implement solutions, particularly those connected to new and emerging programs. This helps governance remain connected to the real-time needs of the business and isn’t seen as a hurdle.
Applying Agile Methodology to Data Governance
This is where the fun starts, identifying a business problem and gathering stakeholders who know about the problem and are trying (or have tried) to solve it. Go for a quick win, stakeholders should think about questions they’d like to answer with data that work towards solving the business problem. From there, choose a question.
Once the group establishes the question to answer, data producers gather data for data consumers to use. That curation has to happen in a durable place so its fruits can be preserved for others in the future. New knowledge and reusable assets will be created and captured with each iteration.
Within these cycles, data producers will quickly learn what’s working and what’s not about the data sources they’re curating, and they can make improvements in real-time. Doing analytics with a living, evolving data asset focuses stakeholders and provides valuable insights at high frequency. That’s why Agile Data Governance practitioners see ROI in days instead of months. By cataloging the work as it happens, and not only the “finished” analysis, teams continuously learn from each other and elevate their data literacy. That’s because people learn data skills and domain knowledge faster by doing the work and seeing their peers solve real problems.
Transparency and iteration lead to progressively higher quality as teams refine analysis and data sources one step at a time. The completed reproducible output now gives people a jumping-off point. When people document their analysis as part of the workflow—not as an afterthought, which is today’s unfortunate norm—their coworkers can find, understand, reuse, and adapt it. As with software development (and everything else in life), it’s easier to start a data project if you’ve got something to build on.
As teams build data assets together and watch each other solve real business problems, the community of data producers, data consumers, and domain experts within the organization grows. Useful, creative, once-rare data practices will spread from team to team and become true, widespread best practices. Anyone who wants to make data-driven decisions will finally find what they need without friction or fear.
Agile Data Governance in Action
To give you a more practical feel of what agile governance looks like, here are some of the key activities that would be involved.
- Knowing your Stakeholders, Keeping them Engaged
- Include data owners in the program’s steering committee. Data owners are usually representatives of business units who are charged with ensuring the accuracy of data.
- Data owners, data stewards, and users receive training that raises their understanding of the need for governance, how the business will benefit, and what specific role they are expected to play in the entire process. This equips participants with the knowledge needed, communicates expectations, and delivers a consistent message.
- Creating a Single Pane of Glass
- Make the agile work stories visible and readily accessible to the steering committee so they can conveniently keep tabs on what is being tackled at any point in time. The key is to make sure everyone understands what needs to be done and can readily refer back to the work plan if they want clarification.
- You could do this via a simple non-tech technique such as Post-It notes, an Intranet dashboard, a large TV screen, an agile project management software or any combination of these.
- Adopting a Centralized Knowledge Base
- Information is meant to be shared. Consolidating business terminology, data definitions and structures, and governance practices into a data catalog, like data.world, enables trusted access and collaboration throughout the enterprise.
- A data catalog keeps both the data and the business connected and in-context so that you actually understand the data. No matter if you have one database or hundreds of applications, your data catalog will democratize your data ecosystem and streamline data access and self-service.
- Starting Small, Growing Fast
- Data governance is not a one-and-done initiative, it is an ongoing effort. By setting realistic expectations and implementing agile methodology your program is more likely to see greater adoption and will avoid the “Icarus” situation.
- For example, plan to complete and approve work in some format on a weekly or monthly cadence. This could run iteratively until the work’s outcome conforms to those realistic expectations.
- Reflecting on Wins (but don’t stop there)
- Just because the work is considered complete and signed off by the data governance council, does not mean that the learning should stop. There’s always room for improvement. Ergo, once work is complete, team members must regularly come together to identify areas of improvement. This retrospection is integrated into the framework, so it’s triggered automatically.
The Payoff for Agile Data Governance
In today’s big-data-fast reality, many organizations have hundreds of thousands of data sources, potentially millions of different data sets, and a growing number of self-service users consuming that information broadly. Traditional top-down, workflow-driven data governance just can’t keep up. What’s needed is a modern approach to governance that’s agile, scalable, and takes advantage of machine learning to meet the needs of data-driven businesses.
In short, the payoff for agile data governance is substantial. Data users will be much happier to participate in the program and will develop a strong conviction that indeed the strategic end goals of the data governance best practices framework are achievable. As the data governance program matures and the focus starts to shift to strengthening the pillars of an agile data governance framework such as data management, data security, and data privacy, then new standards, policies, procedures, and controls will be required as well as updates to existing ones.
Agile data governance doesn’t mean there isn’t a need for rules and decisions about data such as those provided by a centralized, decentralized, or hybrid approach. Taking an agile approach incorporates the best parts of traditional models but shifts the focus to providing support that empowers individuals to collaborate and get more value from data.