Improve the performance and flexibility of search queries with OpenSearch
Airbus
Industry
Aerospace
Service
Cloud Migration
Technolgies
RDS, OpenSearch, S3 bucket, Lambda
Customer Challenge
The search engine is critical for the client from a business perspective. The current solution is to perform research queries directly in RDS SQL Server database.
However, this search approach has latency issues, and some limitations related to the searching queries in Json structure with multiple levels of hierarchic and nested objects.
Another challenge is that the usage of some services (e.g., AWS Migration Service) are not approved by the client.
Proposed solution
We compared SQL and OpenSearch solutions in terms of performance and capability of search in complex structure of Json data based on the same datasets and number of records.
Then, we decided with the client to adopt OpenSearch solution to allow more performance and flexibility.
To migrate data from RDS to OpenSearch, we exported the data from existing RDS database to S3 Bucket in csv format and then a Lambda is implemented to read files from S3 bucket, treat and push documents into OpenSearch indices.
Results & mprovements
After the data is migrated to Amazon OpenSearch, the search engine has been improved without managing any servers or database and with easy scalability.
The average execution time of big GET Query is decreased from minutes to seconds in an index with more than 20k documents; An advanced search capacity is enabled with more complex queries on hierarchical structure and dynamic nested objects; Visualization enablement with OpenSearch Dashboards that can be used from external tools (Kibana).
More projects
To ensure the client requirements, this notification solution must be flexible, reliable, traceable, configurable and easily integrated with other clients’ systems or modules.
Managing files and images effectively is crucial for the business client, however, this task brings new challenges related to the integration with existing systems.