Prior to founding PingCAP, Max Liu was a software engineer for more than 15 years. He spent many hours designing and trying to fix database scaling issues to make coding faster, including creating Codis open-source project as the distributed cache solution.
Having first-hand experience with the time-consuming and repetitive processes engineers and developers face, Max developed TiDB, an open-source distributed database. TiDB offers a more streamlined database that can handle scalability and petabytes of data so customers can focus on more important areas like data analysis and business development.
M.R. Rangaswami: What is HTAP and why is it important to the enterprise?
Max Liu: Hybrid Transactional and Analytical Processing (HTAP) is a type of database that is able to process both Online Transactional and Analytical Processing workloads within the same system, sharing the single source of truth without data link delay in between. This allows for the simplification of technology stacks and data silos, which helps companies to build actionable data insight right from the real-time update and be able to drive for faster growth. HTAP is important to the enterprise because it allows for more efficient and streamlined data processing, which can improve cost efficiency, and expedite business operations and decision making.
M.R.: What are the biggest challenges for database and analytics management today and how should they be addressed?
Max: The biggest challenges for database and analytics management today can be summarized in three Vs: volume, variety, and velocity. The need to process a growing volume and variety of data, the need for real-time data processing and analysis, and the need to integrate data from multiple sources and systems. These challenges can be addressed through the use of advanced technologies such as in-memory databases, distributed databases, and cloud-based analytics platforms, respectively. In order to satisfy all three needs, a very complex data architecture has become a new norm.
As the side effect, the challenge extends further into balancing between data capability and evolving velocity. Additionally, the lack of data engineering talent for such a complex architecture is a high barrier for most organizations to adopt a data-driven culture, investing in skilled personnel, and implementing effective data governance and security practices.
M.R.: Where do you see the database market evolving in the next 5 years?
Max: In the next 5 years, I expect the database market to continue to grow and evolve, with a focus on cloud-based solutions, the integration of artificial intelligence and machine learning technologies, and the development of distributed and scalable databases to support the growing volume and complexity of data. Simplified data architecture is likely to play a key role in this evolution, as it can help to reduce complexity and improve data accessibility, enabling organizations to gain greater insights from their data and make more informed business decisions.
Additionally, there may be increased emphasis on data security and privacy, as well as the integration of databases with other technologies, which can be beneficial from the single database architecture again. Overall, I expect the database market to grow even faster with recent AI technology boosts, like OpenAI 3.5 or even the coming OpenAI 4. And the beauty of simplicity, for instance HTAP databases and low/no code tools, will become more powerful.
M.R. Rangaswami is the Co-Founder of Sandhill.com