I think there's an additional important difference and that's optimization and query cost. Warehouse-native apps owned by other teams run queries directly on the warehouse, owned the by the data team. With additional queries comes a price, absorbed by the data team. If this doesn't have oversight, it's easy to rack up a pretty hefty Snowflake bill. The optimization of query efficiency and cost is up to the vendor, but most warehouse-native tools are very early and aren't talking about it yet. It remains to be seen if there is an additional cost, and if so, if there's a savings that will make an organization come out net-positive.
Jun 24, 2022·edited Jun 24, 2022Liked by Arpit Choudhury
> With additional queries comes a price, absorbed by the data team. If this doesn't have oversight, it's easy to rack up a pretty hefty Snowflake bill. The optimization of query efficiency and cost is up to the vendor, but most warehouse-native tools are very early and aren't talking about it yet.
Great point - and that's why when you implement a warehouse-native app, you need a partnership between the business team and the data team. You effectively need shared vision, budget, goals, and sometimes even leadership. If you don't have that, then you're not going to have a great time.
I'm biased because I work for a warehouse-native saas vendor, but our clients have seen up to 60% contractual decrease in cost using our solution vs. a competitor's - many times, they're even able to consolidate their tech stack. Does that mean increased Snowflake compute costs? Absolutely, but the question is - how much? (To be honest, I don't know - we're almost never given any insight from clients). These queries aren't typically super heavy, even with Enterprise orgs, I've never seen anyone go above a Medium warehouse with our tool. At the end of the day, even with the increased compute cost, we're confident that the solution is much cheaper.
Edit: Re-reading your initial comment again, optimization of query efficiency and cost is NOT up to the vendor here, it's up to the client in most cases, which is one of the best things about this solution. Now, if you're supplying/specifying a specific schema that the data should be transformed to, that's where it's more on the vendor.
When we first moved from on-premise to SaaS, the analogy was it's like plugging your appliance into a socket vs running your own electricity generator at home. SaaS was compelling
Today, indeed, it does feel like each appliance is powered it's own fully integrated electric companies. And those electric companies are sharing data (ELT - RETL) to get 360 view of your house. Now that Warehouses are SaaS, it feels like warehouse native apps are compelling with a singular backend (also in the cloud)
Which are the top Warehouse native apps you're tracking?
I'm primarily keeping an eye on warehouse-native enagagement tools (MessageGears, Vero) as well as traditional SaaS tools that are building native support for data warehouses.
I think users will never be satisfied with a R-ETL setup. Mainly two reasons
1. Because wherever there is data sync, there is a chance of data loss / discrepancy. E.g. Companies can sync data in their marketing tool even now without R-ETL using Segment. But still they can’t trust the data in their marketing tool because of data loss issues.
2. No one would like to work back and forth between two tools - reverse ETL for data sync and mapping, and their main tool for everything else.
The other advantages like unlimited data retention, data security, low setup time, etc are added benefits.
In the long run, it is very likely. But it takes time to build a mature warehouse-native app which is as good as the ones people are using currently. So R-ETL can be a good solution in the short term.
I think there's an additional important difference and that's optimization and query cost. Warehouse-native apps owned by other teams run queries directly on the warehouse, owned the by the data team. With additional queries comes a price, absorbed by the data team. If this doesn't have oversight, it's easy to rack up a pretty hefty Snowflake bill. The optimization of query efficiency and cost is up to the vendor, but most warehouse-native tools are very early and aren't talking about it yet. It remains to be seen if there is an additional cost, and if so, if there's a savings that will make an organization come out net-positive.
Even with Reverse ETL, you will be querying your warehouse at a fixed interval. Optimization of queries is required there as well.
> With additional queries comes a price, absorbed by the data team. If this doesn't have oversight, it's easy to rack up a pretty hefty Snowflake bill. The optimization of query efficiency and cost is up to the vendor, but most warehouse-native tools are very early and aren't talking about it yet.
Great point - and that's why when you implement a warehouse-native app, you need a partnership between the business team and the data team. You effectively need shared vision, budget, goals, and sometimes even leadership. If you don't have that, then you're not going to have a great time.
I'm biased because I work for a warehouse-native saas vendor, but our clients have seen up to 60% contractual decrease in cost using our solution vs. a competitor's - many times, they're even able to consolidate their tech stack. Does that mean increased Snowflake compute costs? Absolutely, but the question is - how much? (To be honest, I don't know - we're almost never given any insight from clients). These queries aren't typically super heavy, even with Enterprise orgs, I've never seen anyone go above a Medium warehouse with our tool. At the end of the day, even with the increased compute cost, we're confident that the solution is much cheaper.
Edit: Re-reading your initial comment again, optimization of query efficiency and cost is NOT up to the vendor here, it's up to the client in most cases, which is one of the best things about this solution. Now, if you're supplying/specifying a specific schema that the data should be transformed to, that's where it's more on the vendor.
This is so insightful, thanks Luke!
Ah yeah, that's good to know, thanks Sarah! Vendors should definitely address this in their communication IMO.
Thanks for this post.
When we first moved from on-premise to SaaS, the analogy was it's like plugging your appliance into a socket vs running your own electricity generator at home. SaaS was compelling
Today, indeed, it does feel like each appliance is powered it's own fully integrated electric companies. And those electric companies are sharing data (ELT - RETL) to get 360 view of your house. Now that Warehouses are SaaS, it feels like warehouse native apps are compelling with a singular backend (also in the cloud)
Which are the top Warehouse native apps you're tracking?
Thanks for your comment Gaurav.
I'm primarily keeping an eye on warehouse-native enagagement tools (MessageGears, Vero) as well as traditional SaaS tools that are building native support for data warehouses.
I think users will never be satisfied with a R-ETL setup. Mainly two reasons
1. Because wherever there is data sync, there is a chance of data loss / discrepancy. E.g. Companies can sync data in their marketing tool even now without R-ETL using Segment. But still they can’t trust the data in their marketing tool because of data loss issues.
2. No one would like to work back and forth between two tools - reverse ETL for data sync and mapping, and their main tool for everything else.
The other advantages like unlimited data retention, data security, low setup time, etc are added benefits.
Thanks Manish for sharing your thoughts. Are you suggesting that rETL will completely be replaced by warehouse-native apps?
In the long run, it is very likely. But it takes time to build a mature warehouse-native app which is as good as the ones people are using currently. So R-ETL can be a good solution in the short term.