I really loved how you have broken the semantic layer. This is one of the most detailed articles I have read on the semantic layer. I would like to understand if you have used the open source models also for the same, and what the efficiency is if you are using the open source model as well. Is it possible to increase the efficiency to maybe 95 to 99%?
Insightful experiment! With a semantic layer, I feel like we will be able to reach really high accuracies when it comes to questions that include already defined metrics. Because as you have done in your experiment, evaluating the agents with the "unit tests" equivalent questions can show the pitfalls of your workspace for your agent clearly, and within a couple of iterations, you can apply what you have learned, which helps the agent perform better.
I am quite curious about the scenarios where users ask reasonable and unambiguous ad hoc questions that have not been defined in the semantic layer. Since it is impossible to test our agent & workspace for such questions, given the wide variety, I wonder how we can get an idea of the expected performance of agents. Thinking loudly, a way for this can be just like evaluating an ML model on test data; we can also have a second set of unit test questions that are not defined in the semantic layer and have non-negotiable ground truth values, which let us observe how an agent performs on "unseen data". Have you tried a similar approach?
You're right! In my case, my 40 unit tests already include questions that a basic semantic layer can't answer - like queries requiring sub-CTEs or pre-aggregation joins. For those, you either change your data model to make them computable in one pass, or you allow a fallback to SQL (that's what the dbt natural language querying skill does in my setup)
I really loved how you have broken the semantic layer. This is one of the most detailed articles I have read on the semantic layer. I would like to understand if you have used the open source models also for the same, and what the efficiency is if you are using the open source model as well. Is it possible to increase the efficiency to maybe 95 to 99%?
thanks for the kind comment!
I havent' used the open source model yet - but you're right I should test several models.
I'm not sure we need to reach a higher theoric reliability - the reliability gap might be solved by just clarifications with human interactions
Hi! Super interesting.
Do you have some sort of YouTube tutorial to make it more concrete for me?
I am looking for a simple 101 just to see how to go from "zero to hero".
Or do you have some sort of tutorial to setup nao?
Thanks
Max
Hey Max, yes we have one here! https://www.youtube.com/watch?v=LBHD3WSXnrU&list=PL-oT4gDum20rZQ_X29MjpSj-ggOUQgssN
Insightful experiment! With a semantic layer, I feel like we will be able to reach really high accuracies when it comes to questions that include already defined metrics. Because as you have done in your experiment, evaluating the agents with the "unit tests" equivalent questions can show the pitfalls of your workspace for your agent clearly, and within a couple of iterations, you can apply what you have learned, which helps the agent perform better.
I am quite curious about the scenarios where users ask reasonable and unambiguous ad hoc questions that have not been defined in the semantic layer. Since it is impossible to test our agent & workspace for such questions, given the wide variety, I wonder how we can get an idea of the expected performance of agents. Thinking loudly, a way for this can be just like evaluating an ML model on test data; we can also have a second set of unit test questions that are not defined in the semantic layer and have non-negotiable ground truth values, which let us observe how an agent performs on "unseen data". Have you tried a similar approach?
You're right! In my case, my 40 unit tests already include questions that a basic semantic layer can't answer - like queries requiring sub-CTEs or pre-aggregation joins. For those, you either change your data model to make them computable in one pass, or you allow a fallback to SQL (that's what the dbt natural language querying skill does in my setup)