Using filters is a very common circumstance during the conversation with crystal especially when we want to know something more precise about our data.
That is why we might want to narrow down our generic question (like How are my sales going?) to a more detailed insight, for example by asking info on a geographical area (How are my UK sales going?), on a given period of time (How are my 2021 sales going?) or on a particular item (How are my sneakers sales going?).
By using dynamic filters we can retrieve very precise insights and obtain detailed answers to our questions.
How to ask questions including filters
Filters can be numerical (when they involve numbers) and non-numerical (anything which is not numbers).
Numerical filters can be currency, item codes, measures, while non-numerical filters can be countries, cities, areas, item names, clients names and so on.
crystal is trained to recognize words and phrases even if they are not spelled exactly like they were in the set up process or if they are incomplete.
You might also want to ask crystal about the total sales in the UK, but you are typing instead: “show me total sales for United kigdom”. The advisor will still get what you mean, regardless of your typing error.
The same will happen with incomplete sentences: let’s assume that you want to retrieve some information on the sales of a sneakers model which is called “Super High Quality Summer Range High X3” (your filter) but you wanna be fast, so what you are typing is: “Can you show me X3 shoes sales?”. crystal should still understand what you mean.
crystal will also recognize numbers, regardless of how you write them: 1000 and 1.000 are the same for the advisor.
However, to be 100% sure of getting the right result we’d recommend to admin users to keep the filter value aliasing short and simple (for example, “X3 sneakers” would work better than “Super High Quality Summer Range High X3” or “Milan site” would be easier than “Northern Italy Area - site of Milan”) and to members to try to be as complete and specific as possible when adding filter values to a question.
You can also ask crystal about filters with negative values, for example "list down the products with negative rating" or "what are the countries with a negative score?".
How to ask questions excluding filters
If you want, you can exclude a specific filter when asking for a topic.
For example, you may want to see the sales distribution save for a specific country. In that case, you can ask crystal a question like "show me the breakdown of sales excluding Spain".
Or, if you're monitoring the energy production in a plant, you may ask crystal about the “comparison of actual vs. budget not considering energy losses.”
To ask such questions, you can use formulas like "different from, without, not, not in, not for, excluding, save for, not considering" to exclude a specific value.
Data preparation guidelines for effective filter classification
As explained here, crystal needs some data preparation activity to guarantee the best experience possible. In the case of filter use, however, our NLP stack already performs data cleaning to make it easier to ask for complex filter values.
The pre-processing phase requires only two precautions to obtain effective filter classification. It is advisable to:
Remove punctuation and redundant whitespaces.
Make values case insensitive.
Filter classification logics
Generally, when a user requires to apply a filter in the conversation with the advisor, there are two scenarios possible:
The system recognizes the filter value by exact match, for example when a user asks Show me the total sales for France, and the filter is France.
The system recognizes the filter value by partial match, for example when a user asks Show me the total sales for Italy, and the filter is Northern Italy.
Let’s see in detail how these filter logics work.
The text included in the question to be recognized is exactly the same as in the data source (unless any modification is included in the pre-processing, like punctuation or special characters). This is the simplest case for the system to recognize: when the filter is “Italy”, the user will type “Italy” in the question and the system will immediately recognize it.
Sometimes it can happen, though, that a user does not type the filter value exactly as it is in the data source, or that some parts of the filter value are missing in the advisor question, but we still want the system to recognize it: that’s when the partial match procedure gets involved.
crystal will retrieve the right filter, even if parts are missing or misspelled, thanks to a Full Text engine, a tool able to scan a lot of documents very quickly and to find the most similar values to the one given as input.
This engine can search across all the values thanks to the Named-entity recognition (NER), a machine learning model that learns to identify filter-related words inside a sentence, and it will return a ranking list of the most relevant filter values. The highest-scored words will be retrieved as filter values.
Let’s see a couple of examples.
In the first case our filter is “Northern Italy”, but the user typed instead “Nrthern Italy” (and this would be a misspelling) in the question.
In this case, the Full Text might find as possible filter values:
The values would be ranked and “Northern Italy” would be selected as the highest-scored word, our filter value.
Another case can occur when the user types a partial value in the question: let’s assume that we want to know the sales results for “Australia”, but we just type “Austr” instead.
The Full Text will find:
Both words will have the same score and, not knowing what we want to see in our answer, the system will trigger the disambiguation function.
One thing that is important to mention is that the NER model relies both on word meaning and on sentence structure: for this reason introducing the filter value with particles such as “for”, “in” etc. would surely help the classification. For example typing “Show me total sales for France” would be more helpful than “Show me total sales France”.
Entity aliasing and filter aliasing
Both entities and filter values can be associated with aliases to be used in the conversation. Check out this article to know more!
Hope this article was helpful! Get some more tips on the advisor here!
If you have any questions about crystal’s features, you have encountered a problem or you would like to share your feedback, contact us using this form.