Transforming Material for MkDocs¶

We are working on an amazing set of features which has required some behind-the-scenes work we are now ready to share in a series of posts. Here, we give an overview of our goals, features in the making, things that kept us up at night, and our commitment to Open Source.

We know it's been quite a while since our last update, which is why we're eager to share what's happening in and around Material for MkDocs with you. For the largest part of 2024, we've been focused on transforming the core of Material for MkDocs preparing the ground for new, interconnected features that are amongst the most frequently requested.

This article is the first in a series where we'll explore how we aim to support our users through improved information retrieval, provide better support for multi-lingual sites and versioning, as well as improve the overall authoring experience. We outline our vision for the projects' evolution and describe what we have been working on. In this and the coming blog posts, we will share our progress with you, and we're excited to hear your thoughts.

Please note that this post includes several technical details in the footnotes, specifically on challenges. Feel free to skip them if you're not interested in the specifics.

A success story¶

In 2024, Material for MkDocs has firmly established itself as a leading tool in the documentation framework landscape, with more than 5 million downloads each month as of this writing. What began as @squidfunk's personal project has evolved into a versatile resource for creating comprehensive documentation sites, personal websites, blogs, and notably, for managing knowledge both within and outside of organizations.

With almost 50,000 public GitHub projects depending on Material for MkDocs, it's clear that the framework has made a substantial impact. Tens of thousands of authors rely on us to deliver documentation to millions of users each month.¹ Beyond its adoption by many popular Open Source projects, Material for MkDocs is trusted and financially supported by major corporations such as AWS, Microsoft, and Siemens, among many other companies and individuals. We're very grateful for the continued support we receive, which allows us to devote our time to this project, making writing documentation enjoyable.

Our users particularly appreciate Material for MkDocs for its ease of use and straightforward setup. It simplifies the process by handling the complexities of web technologies for you, so you can build a production-ready static site in minutes without needing to invest years into mastering frontend development or JavaScript. This makes it accessible to a wide range of users, regardless of their technical background. Additionally, Material for MkDocs is MIT-licensed and free to use, which has contributed to its widespread adoption, and allows everybody to build sophisticated documentation sites at no cost.

Our vision is to provide you with the tools that allow you to own your docs, enable you to develop your own processes for producing amazing documentation and to avoid being locked into costly subscription services even for doing simple things. We believe it is important that these tools should be easy to set up and configure to suit your needs but should also give you the freedom to adapt them further, if needed. This is why we are always striving to ensure our architectures are modular and future-proof.²

Challenges¶

Now, let's talk about our journey and the particular challenges we're addressing. With "we", we're referring to the incredible team that @squidfunk was fortunate to build around his original work, thanks to the financial support he receives from his sponsors. This remarkable team includes @alexvoss and @katharinalisalin, whose invaluable contributions in research, development, and community support have been essential to the project's ongoing success.

Together, we've started exploring new technologies, incorporating the feedback we received from our users, and rethinking critical components from first principles, serving our growing community one of the best frameworks to create documentation.

This section highlights the key areas we've been focusing on.

Search and discovery¶

From our perspective, search is a cornerstone of any effective documentation site, enabling users to swiftly locate the information they need. From the start, we've relied on Lunr.js, a popular client-side search library that has streamlined the deployment of documentation sites without needing an external service. Over the years, Lunr.js has served us well by answering millions of queries on sites built with Material for MkDocs. Yet, as our users' needs evolved, we encountered limitations that proved difficult to overcome.

We learned that the BM25 ranking algorithm at the core of Lunr.js (as in many other search implementations) isn't well-suited for effective and stable typeahead. Adding new documents can influence existing rankings, requiring manual adjustments³ that are cumbersome and hard to scale. Furthermore, Lunr.js has performance issues that stem from its design not fully leveraging modern JavaScript engines and optimizations, making it slower and memory-intensive.⁴

Over the last years, we've invested substantially into improving the search experience. For instance, expanding Lunr.js' functionality to add support for rich search previews and tokenizer lookahead demanded substantial engineering effort. Lunr.js allows to customize tasks such as stemming, stopword filtering, and trimming with pipeline functions, but it makes it particularly hard to add extensions like term highlighting or alternative ranking methods. Compounded by the fact that Lunr.js has been unmaintained since 2020, it became clear that we needed to find a more powerful solution. We've looked at alternative JavaScript-based libraries to keep our client-side search, but none met our requirements or lived up to our expectations.

To address these challenges, we've embarked on developing a new search system from first principles that not only matches but already exceeds the capabilities of Lunr.js. Built from the ground up, this system is faster, more compact, and most importantly: modular. It is based on a growing core evolving around two core concepts we isolated to be essential – engines and plugins – allowing for flexible configuration and assembly of components like text indexing, vector embeddings, filtering, ranking, highlighting, and more. Every part of it can be replaced or extended, enabling users to tailor the search system to their specific needs.

Our new search system, which we will release as a separate MIT-licensed Open Source project, is designed to handle a wide range of scenarios — from small blogs to large documentation projects. Of course, it supports offline use and integrates seamlessly with both client and server environments. The advanced ranking system provides fine-tuned control, and integration with third-party services is now more straightforward.

Goal – Empower users to quickly find the information they need while freeing authors from managing external services, by providing a search system that adapts to diverse content types.

You might wonder why it's not yet available. The process of developing this system from scratch and uncovering its potential has led us to re-evaluate core concepts of Material for MkDocs. As a result, we've decided to postpone the release of the new search system to integrate it into the broader update that we've started to work on. If you keep on reading, you'll learn about further reasons why we've decided to follow this approach.

We're excited to share more details about this update in one of the next posts in this series.

Translations and versioning¶

Supporting multi-language sites in MkDocs is the most requested feature on our discussion board and in conversations with users, yet it presents significant challenges, as MkDocs does not natively support it. The same applies to versioning, which also involves synchronisation of multi-project builds. While the MkDocs ecosystem has developed various plugins and tools to address these issues, there is still substantial untapped potential. We began exploring these areas but quickly encountered problems that hindered our progress.

As you may know, our initial effort involved the projects plugin that aims to extend MkDocs to add support for multi-project environments as a solid foundation to support multi-language sites and versioning. Unfortunately, we encountered significant challenges due to MkDocs' internal architecture and design constraints, which we're working actively on resolving.⁵

Goal – Enable scaling documentation to any size or team structure by offering seamless methods for integrating multiple documentation projects, whether they involve different languages, versions, or distinct sections of an overall body of work.

As a result, we are developing a new approach to offer a more comprehensive and robust solution for both multi-language support and versioning. This new approach also intersects with adjacent functionalities like search, as many of our users are interested in federated search capabilities that combine results from multiple documentation sites into a unified search interface. Overcoming this challenge is one of the major hurdles we need to address before releasing the new search system.

Editing and collaboration¶

We had considered developing a live editor in response to MkDocs' performance issues with large projects, which in most cases stem from compute-intensive plugins that don't employ caching. A proof of concept based on Pyodide (= running Python in the browser) generated significant interest among users and prompted many organizations and individuals to share their collaborative workflows for feedback. Sadly, implementing this live editor proved to be very challenging, as it would require rebuilding substantial parts of MkDocs.⁶ After discontinuing work on this approach, our progress with multi-project support has renewed our belief that we can finally solve the sluggish editing experience that was reported several times over the last years.⁷

This brings us to collaboration, which wasn't initially on our list of priorities. However, throughout 2024, conversations with various organizations and maintainers of popular Open Source projects highlighted a frequent request for enhanced collaboration features. Many users expressed a need for capabilities that would allow non-technical team members to suggest and make changes to the documentation. We're genuinely grateful for this feedback, as it has come at a pivotal time. We recognize the need to streamline tracking and discussing changes, as well as to facilitate drive-by contributions.

Goal – Everyone, regardless of their technical skill level, can easily work on and improve the documentation and contribute to a growing corpus of knowledge.⁹

This focus on collaboration aligns with how knowledge is managed in enterprises. In large organizations, documentation often exists in information silos — decentralized yet essential repositories of information. We understand the need to be able to unify these disparate sources into a coherent body of knowledge while preserving decentralized ownership. This also nicely aligns with our previously outlined work on multi-project support, as well as the new search system to implement federated search among multiple projects.

Large Language Models (LLMs)¶

Almost a year ago, we introduced an experimental chatbot on our documentation site. It quickly became one of the most anticipated features, with users requesting the ability to deploy similar functionality on their own sites, underscoring the growing demand for interactive documentation tools. However, we found that adding to the myriad of existing chat solutions and simply building another thin wrapper on top of ChatGPT is nonsense.

Goal – We're envisioning creating a unified interface that seamlessly integrates advanced search, chat, and summarization features, providing an interactive documentation experience.

As we delved into this ambitious project, we gained valuable insights from user feedback. Users began interacting with the chatbot in their native languages, an outcome we hadn't anticipated given that our documentation is in English. Remarkably (or obviously to those that work on LLMs year round), the chatbot responded in the same language. This ability of LLMs is one of the genuinely exciting features of these machine learning models as it has the potential to improve the accessibility of the documentation. However, while we employed state-of-the-art RAG methodologies, the results were mixed, and occasional hallucinations surfaced.

These experiences led us to prioritize enhancing our search capabilities before integrating LLM-based features. Building a search engine from scratch was already a substantial effort, and adding more complexity without a solid search foundation would be premature. By rearchitecting our search functionalities, we aim to create a robust platform that will seamlessly support advanced information retrieval and deliver a cohesive interactive documentation experience.

Team, transparency, and growth¶

While we navigate the challenges and explore the opportunities of this project, we consider it essential to demonstrate how we're building a solid foundation for its continued growth and success. Please consider this an overview rather than a formal roadmap — our detailed plans are in the works. The goals we've highlighted represent the most impactful areas we aim to address.

Thanks to the generous support from our sponsors, we're fortunate to be assembling a team capable of dedicating significant time and expertise to this endeavor. This newfound capacity allows us to delve deeper into core development while also engaging more comprehensively with our user community. A special mention goes to @kamilkrzyskow, one of our invaluable community experts, who has been essential in supporting users and fostering discussions on our platform.

With the team's support, @squidfunk can concentrate on the heart of development, while we have begun investing in user research. This effort is helping us understand how organizations and individuals interact with our tools, guiding the project's future direction based on real feedback from numerous conversations with users and companies.

Looking to expand our team further, we are committed to improving transparency and communication. Our previous work often happened behind the scenes due to time constraints, but we're now focused on making our processes more open and inviting for new contributors. By embracing this collaborative approach, we aim to enhance our tools and ensure they meet the evolving needs of our community.

What's ahead of us¶

As we look into the future, some of the groundwork we're laying now is crucial for the exciting developments on the horizon. Many of the initiatives we've discussed represent foundational work that will set the stage for much more ambitious and innovative features. Once these core elements are in place, we'll deliver a range of exciting new functionalities that align with our vision and goals.

In the coming months, we'll be sharing more details about our plans and how they will contribute to our overarching objectives. While growth and innovation are at the forefront of our plans, we want to assure you that our core values remain unchanged. We are committed to maintaining the principles that have guided us thus far, ensuring that our growth is both healthy and consistent:

• Against recent industry trends with companies moving away from core ideas of Open Source, we are doubling down on our commitment to Open Source because we believe it's at the heart of the value proposition of our work and the docs as code approach.

• Our organic approach to growth is part of this strategy as it keeps us independent of individual funding sources and pressures to provide a return on investment, which is what causes many other projects to turn away from the principles of Open Source.

• Likewise, we are driven by the needs of the community for a rich ecosystem of adaptations of the core framework. Extensibility and modularity are key for this and we are working hard to improve the developer experience by providing clear interfaces.

Stay tuned for updates as we continue to build on our progress and explore new possibilities. We're excited about the future and look forward to sharing more with you as we advance.

Martin, Alex and Kathi

Comments