Most existing search solutions, such as Elasticsearch, only use the keywords found in the query and the documents in which they’re searching. Because of this limited view, they usually fail to consider the user’s persona, specific characteristics, and history, such as previous search queries and the time spent on each of the search results. This information is vital to finding the most relevant information for our users. Furthermore, once the document is found, where in the document is the answer? If the most relevant document is quite long, it won’t help the user if it doesn’t specify where in the document to find the information they’re looking for. While doing all this, we also need to keep in mind latency. How long after entering the query will the user receive their answer? Our system addresses these vital points. Let’s take a deeper look into how by examining our architecture.
,type=downsize)
- User queries are sometimes misspelled or poorly worded. The query pre-processing, rewriting, understanding component does spell checking, expands the query and rewrites where necessary to improve the recall.
- The query is checked against the cache and, if present, the results are retrieved.
- The document adapter collects different types of documents from multiple sources and ingests them into the database.
- The BM25 algorithm is used for ranking the documents based on the search query. Afterwards, the top documents are sent to the next stage.
- The document re-ranking stage compares documents pairwise and, based on the user interaction features, re-ranks the documents.
- Relevant paragraphs from the top document are sent to the question answering system to select precise answers to the query.
- From the User Interface (UI), the system also logs click-through data from which features are derived to improve the algorithm. A thumbs up/down interactive UI component helps collect explicit feedback from the users.
Feature engineering:
The above system looks great, but it won’t work without some additional data in the form of feature engineering. Feature engineering is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in improved model accuracy in unseen data. In this case, that means predicting the relevance of the query within the context of the user’s persona.
User Features: Role, Name, and Email
Query Features: Query intent, Popular/Frequent Queries
Document Features: Rank, Engagement, Term Frequencies, Document Length, Source, Category, Subcategory, Filename, Document Content
User Interaction Features: Rank, Clicked, Time Spent on Page, Rating (Thumbs Up/Down)
Derived Features: Click Frequency, Click Probability, Is Next Clicked, Is Previous Clicked, Is Clicked Above, Is Clicked Below, Average Rating
These features are engineered throughout the entire lifecycle of the system. User and User Interaction features are created when they create an account and how they interact with the system. Query features are created as users query the system. Document features are created when the documents are crawled or adapted. Derived features are created from analysis of user interactions with specific queries and documents.
Models:
After evaluating multiple algorithms, the following algorithms are picked for this solution framework:
BM25
We chose BM25 to perform our initial document ranking because it scales very well with the number of documents to be searched and powers the most basic search solutions like Elasticsearch. This is because it uses sparse vector representations. We also modified the open source package so that it supports online learning. When the new documents are saved into the database, the term frequencies and document frequencies will be updated accordingly in the database. We have tested a couple of modifications to BM25 and found that BM25+ has the best performance. We also use BM25+ for ranking paragraphs within the top documents. These top paragraphs are sent to the Question Answering stage.
LambdaMART
LambdaMART is used in the re-ranking stage. We chose LambdaMART because it brings the power of gradient boosting trees to the ranking problem, especially when you want to integrate additional features such as user feedback. For training and inference, we collect document, user interactive, derived, query and user features. LambdaMART uses normalized discounted cumulative gain (NDCG) as the metric for the loss function. We can tune the model offline with different hyper-parameters such as the number of trees, learning rate, NDCG, and more. Optimizing these parameters allows us to achieve high relevancy in re-ranking.
DistilBERT
DistilBERT is used in the Question Answering stage. We chose DistilBERT because it provides nearly the same relevancy as BERT at a fraction of the inference time and memory. It efficiently provides a selected answer from the context due to the student and teacher mechanism behind DistilBERT. HuggingFace’s Transformers package provides a pre-trained model so that we can quickly spin it up for inference. We were able to fine tune the model with the help of our subject matter experts (SMEs), providing additional high quality supervised training data. During the tokenizing procedure, we add special tokens, [CLS] and [SEP], to mark the start of the query, to separate the query and context, and to mark the end of the context. Also, we set the max token length to be 512 tokens. Finally, the model will return answer ID and answer tokens. Therefore, we can get the answer by converting token IDs back to strings.
The pre-trained model weights and related API can be found here: https://huggingface.co/transformers/master/
Evaluation metrics:
In an online setting, the success of a search session can be based on the per-query level with metrics such as click-through rate, which can be calculated as the number of clicks/number of impressions.
Another metric could be dwell time, which is the length of time a searcher spends viewing a webpage after they click on the link on a search engine result page (SERP)
The NDCG, which is commonly used offline, is derived as follows: Cumulative Gain is the sum of all the relevance scores in a recommendation set. It does not consider the rank of the document which is included when calculating Discounted Cumulative Gain. The computation involves discounting the relevance score by dividing it with the log of the corresponding position. However, it is still not complete. Depending on various factors, the number of recommendations served may vary for every user. Therefore, the DCG will vary accordingly. We need a score with proper upper and lower bounds to calculate the mean across all the recommendation scores to report a final one. NDCG accounts for this and normalizes the DCG.
Performance
To evaluate the performance of our QA model, we had questions and answers procured by our SMEs. These questions were used as search queries in the system and the results were matched with the answers from humans. We were able to exactly match 65 percent of these answers. With human evaluation, we were able to determine that 90 percent had essentially the same semantic content.
DistilBERT QA Performance:
- Exact match: 65 percent
- Same-info match: 90 percent
Next steps
Application is released as limited availability to Ericsson’s Market Area North America (MANA) organization for evaluation. Upon successful testing it will be released as a general availability (GA). Its flexible framework can be reused for all such similar applications.
Acknowledgements
We would like to thank Thalanayar Muthukumar for providing ample and essential support to this project. We would also like to thank Piyush Mittal, Chenhua Shi, Yiyu Ren, and Subramanian Iyer for their contributions to development.
Learn more
Read Anila’s previous co-authored blog post Facial recognition in security systems: How they work and what they could contribute to the security posture.
Learn more on AI in networks.
Read more on AI operations and optimization
"search" - Google News
October 26, 2021 at 05:03PM
https://ift.tt/3Gln4mj
A deep dive into persona-based search - Ericsson
"search" - Google News
https://ift.tt/2QWB6Sh
Bagikan Berita Ini
0 Response to "A deep dive into persona-based search - Ericsson"
Post a Comment