You have changed your resolution.
Refresh the page
In our continuous effort to bring more convenience and efficiency to our clients, we at Global Concept devised an innovative email management system that allows users to connect their email addresses using IMAP and store all the correspondence within their CRM. This system provides the ability to view conversation history within their client's tab, essentially transforming the CRM into a comprehensive communication hub.
During the development process, we encountered a significant challenge that we had to overcome to ensure the smooth performance of this system. This blog post explores that challenge and our unique solution.
As part of our rigorous testing phase, we duplicated existing content to simulate the database's performance with two million messages. Unfortunately, we found that the system's performance slowed significantly under this volume of data.
The core of the problem was that some messages were as large as 200kb of text size, and MySQL, our chosen database management system, struggles with handling large text data. It became clear that we needed to find a solution to ensure that our system could handle the potential volume of data our clients might need to manage.
To solve this issue, we opted for an innovative approach that involved storing only the essential information in MySQL, such as the subject and the date of the email, while storing the full text of the messages elsewhere.
The text of the messages was stored in JSON files within our file storage system. This approach helped to relieve the burden on MySQL, allowing it to handle the essential information more efficiently.
But, of course, we still needed a way to make the full text of the messages accessible and searchable for our clients. That's where Elasticsearch came into play.
We created an Elasticsearch instance and stored the text of the messages there, with stripped HTML tags. Elasticsearch is a powerful search engine known for its speed and efficiency in handling full-text search operations, even with large datasets.
By storing the text data in Elasticsearch, we ensured that our system still offers full-text search within the content. Not only that, but it also provides typo-tolerance, which improves the user experience by returning relevant results even if the search query contains spelling errors.
Our innovative approach to the email management system illustrates our commitment to overcoming challenges and ensuring that our solutions are efficient, scalable, and user-friendly. By leveraging different technologies like MySQL, JSON files, and Elasticsearch, we were able to create a system that seamlessly handles large data volumes and provides robust, typo-tolerant search functionality. At Global Concept, we're all about pushing the boundaries of what's possible to deliver the best solutions for our clients.
