April 16, 2020
A lot has been said about the pace of change in today’s business environment. In the book “It’s not the BIG that eat the SMALL, it’s the FAST that eat the SLOW,” many examples are given of smaller, faster, and more agile companies putting bigger, slower, and rigid companies out of business. This has translated to IT as well. Software vendors and project management methodologies now tout how their software and methods will help your organization become more agile, delivering services and information more quickly and with less effort.
With the understanding that today’s business environment is experiencing an ever increasing rate of change and that the BI ecosystem needs to support this, it is critical to take a fresh look at your BI environment. There are three key, often competing, components of a BI solution that must be considered to create agility:
- Performance: How responsive are reports, visualizations, and dashboards.
- Flexibility: How adaptable is the solution to changing business processes and questions.
- Usability: How intuitive and easy the solution is to use.
Where the Project Management Triangle speaks to the quality of a deliverable, the Agility Triangle speaks to the architecture of a solution. A project manager must balance cost, scope, and time, whereas a solution architect must balance performance, flexibility and usability. In working with companies across many industries, I have seen patterns in BI implementations that cause them to fall short of being agile and effective solutions.
Pattern #1: A sole focus on data base PERFORMANCE
The most common mistake I have seen teams make is a sole focus on data base PERFORMANCE. While this is important, I have seen teams focus so much on performance, that they neglect FLEXIBILITY and USABILITY. This approach implements data structures where everything is pre-calculated and pre-aggregated, leaving the presentation layer very simple. The introduction of a new requirement requires the updating of data structures and ETL processes before the presentation layer can be updated. With the single focus on data base response time performance, the environment becomes much less agile, taking weeks to implement a change to ensure response times are not compromised. In essence, the database design becomes the presentation layer. Most of the BI tools today take advantage of in-memory data cacheing, parallel processes, and clustering that greatly increase response times. The database is not the only component where performance optimization is addressed. A more flexible data structure design can be implemented without compromising performance if all components are optimized.
Pattern #2: A primary focus on FLEXIBILITY
In other environments, teams primarily focus on FLEXIBILITY, where a great deal of effort is put into offering a great deal of flexibility in reporting to support “any question” someone might possibly ask. While the goal is met, the solutions are very difficult to use and leave users confused and frustrated. USABILITY receives a poor grade here and performance can be acceptable but not great. There needs to be a balance between flexibility and usability. The “everything for anyone” requirement is best supported with an Ad Hoc query tool meant for power users. The use of the right tool for the right need is important. When development becomes very complex and time consuming, this may be a sign that the right tool is not being utilized for the need.
Pattern #3: Ease of use above all else
In environments where the focus is entirely on USABILITY, in the quest to be like Steve Jobs, reports and Dashboards are designed to be extremely simple to execute. That is not a bad thing, however, to meet the needs of various departments, user groups, and skill levels, many versions of the same reports and dashboards are created with the only difference of filters and available columns. Making changes to a report meant making changes to A LOT of reports. Agility is almost non-existent and produces a deluge of reports that is overwhelming to users. While the reports themselves are very easy to execute, finding a report with the needed data is difficult. A great deal of time is required in support of the reports, including security and portal management. There is a high Cost of Ownership in these environments.
Balance is critical to have an agile and effective BI ecosystem. I use the term ecosystem purposefully. A BI solution should be comprised of many tools and layers that interact fluidly. It is not simply a single BI Software. Data architecture, ETL processes, Reports, Dashboards, Visualizations, MDM, Program Management, and more, all need to be considered. People, processes, and technology need to be evaluated and graded. When evaluating your own BI environment, how would you grade it? What success factors would you use? In asking “how agile is my environment?” you are asking a great number of questions: “How long does it take to make changes?” “How streamed lined is the change request process?” “How long is the request queue?” “How easy is it to use?” “Can it answer new questions on the same data without changes to the underlying data structures and ETL processes?” “Can users navigate to the right reports to answer their questions” “Do power users have the right tools to perform their tasks?”
One of the reasons I am passionate about data warehousing and BI is the challenge in balancing business requirements with IT capabilities. It’s the process of understanding, evaluating, and collaborating that is both challenging and rewarding. If all parties involved are willing to work together to reach the best possible outcome, great things happen! Great things can happen in your environment, if a balanced approached is taken, the right questions are asked, and everyone works toward the same goal.
Originally Published on October 22, 2018
Scott Keith, ETL/Data Architect, Prodigo Solutions