Got it. If I understand the graph correctly, Rules only was about 44% correct. Rules + sampling only reached 45% (1% improvement). Seems sampling impact is pretty small, right?
I tried something similar, I wonder what you think of the results. Dbt vs ORM. I will try incorporate some of your tests when I do the next steps. It’s a theory that dbt / SQL is way too sparse in terms of useful info for agent to be effective
Really interesting article, and great that you’re investigating this as well. I think you’re probably right that more typed / ORM-style setups like Drizzle can be easier for agents to reason over than plain dbt/SQL, since there’s more explicit structure and less guesswork for the model.
The tradeoff is that it can get much harder for humans to read, audit, and verify compared to simple SQL. So it feels like a balance between agent-friendly and human-friendly so not sure what data teams will choose 🙂
Thank you for taking a look, yes that is exactly the crux. I’m convinced based on what I’ve seen from SWE that humans will not need to read as much code. Where the balance is is maybe less extreme than what we are currently seeing.
I’ll keep you posted as I run more experiments, and look forward to reading more of yours
Would be really interested to see the rules.md. can you share the file - maybe anonymize some things?
can you share more details about sampling? were you including 10 rows per table?
in my context, i share some helper views that joins tables together and that seems to help the AI understand the relationships
Yes for sample I add the 10 latest rows
And about the rules.md, global structure looks like this:
## Business Overview
**Product**: Nao is an AI code editor for data teams (forked from VS Code), integrated with databases and dbt projects.
etc.
## Data Architecture
Info about our medalion architecture
## Core Data Models
Example:
#### `fct_stripe_mrr` (Monthly Recurring Revenue)
**Purpose**: MRR by subscription and month
**Key Columns**:
- `stripe_subscription_id`, `stripe_customer_id`, `month`
- `mrr_usd`: Monthly recurring revenue in USD
- `n_paying_users`: Number of users in the subscription
- `n_churned_users`: Number of users who churned this month
- `is_active`: Whether subscription was active this month
- `is_churned`: Whether subscription churned this month
- `billing_interval`: 'month' or 'year'
**Use For**: MRR trends, revenue forecasting, growth metrics, churn analysis, team vs individual revenue
## Analysis Process
Info on Select the Right Table(s)
- **User questions** → Start with `dim_users`
- **Activity/engagement questions** → Use `fct_nao_activity` or `fct_nao_posthog_events`
- **Revenue questions** → Use `fct_stripe_mrr` or `fct_stripe_subscriptions`
- **Feature usage questions** → Use `fct_nao_posthog_events`
- **Error/stability questions** → Use `fct_nao_posthog_exceptions`
## Key Metrics Reference
Metrics are defined like this:
- **Paying Users Count**: Count of users with `is_paying = true` in `dim_users`
Got it. If I understand the graph correctly, Rules only was about 44% correct. Rules + sampling only reached 45% (1% improvement). Seems sampling impact is pretty small, right?
Yes if you have good rules it seems like it. But the % stays quite low so my next step is to investigate how to make this % improve!
Pretty crazy to me that this is one of the few analytics agent eval "benchmarks" that I can find given how hot the topic is rn
Yes! I think very few people do this ground work
Hey! Do you use the sql query, the output or both for eval?
Hey! We only check the data output
I tried something similar, I wonder what you think of the results. Dbt vs ORM. I will try incorporate some of your tests when I do the next steps. It’s a theory that dbt / SQL is way too sparse in terms of useful info for agent to be effective
https://open.substack.com/pub/groupby1/p/a-modern-data-benchmark
Really interesting article, and great that you’re investigating this as well. I think you’re probably right that more typed / ORM-style setups like Drizzle can be easier for agents to reason over than plain dbt/SQL, since there’s more explicit structure and less guesswork for the model.
The tradeoff is that it can get much harder for humans to read, audit, and verify compared to simple SQL. So it feels like a balance between agent-friendly and human-friendly so not sure what data teams will choose 🙂
Thank you for taking a look, yes that is exactly the crux. I’m convinced based on what I’ve seen from SWE that humans will not need to read as much code. Where the balance is is maybe less extreme than what we are currently seeing.
I’ll keep you posted as I run more experiments, and look forward to reading more of yours