Rendered at 09:08:31 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
sgarland 9 hours ago [-]
It is hilarious to me when companies act like they've made some huge discovery that's been well-known to anyone working in DBs for the past decade, if not more.
JSON is slow as hell, both for serdes, and especially querying it. Use it sparingly, if at all. Normalization is not that hard, and it's trivial now with LLMs. Tell Claude you want your schema in BCNF, or 5NF if you're feeling froggy. Denormalize if and only if you have performance issues.
nomel 2 hours ago [-]
In my group, the person who helped the vendor set up our new DB has convinced management to convert all of our metadata tables (which we use for every filter operation) to JSON. As you can guess, he's never benchmarked a DB in his life.
Since our visualization tool went from 2 seconds per click (when I set it up the original DB) to over 40 now, even before implementing this change, I'm watching in amazement, waiting for someone to complain. It's truly incredible how tolerant users are. It's one of the reasons I think AI will take our jobs much sooner than most think: the average user has no useful perspective on silly things like performance or determinism, "that's just how it is".
tomaskafka 2 hours ago [-]
They do hate it; they have been taught that no matter the complaints, the bad experience will never change, so why waste time complaining.
edwcross 2 hours ago [-]
Whenever I call some client support service to complain about their website being broken, incorrect, malfunctioning, or something similar, and I explicitly ask them to notify their IT system about it, I feel like some weird alien or raging boomer asking for something impossible. They don't even understand it, and my complaints just become useless. There are no more e-mails to contact, nothing that can be done to improve the situation. It's enshittification all the way down, and too bad for those of us who had known a better state of affairs, we just have to become blind to it.
guites 9 hours ago [-]
The point they make is that user data is dynamic, meaning you can't just blindly normalize every json attribute into its own column. The challenge is determining which attributes deserve "shredding" and doing it automatically iiuc
Neywiny 8 hours ago [-]
Not even companies. I've seen this with colleagues. They wanted JSON parsing in a moderately embedded device. No thank you. Parse it elsewhere, give me the binary result
boricj 3 hours ago [-]
Meh, it's not ideal but it's not that bad either.
I'm working on an embedded system with a MCU and I've benchmarked JSON serialization at over 500 KiB/s (through a HTTP GET request) using my own library. Deserialization is slower, but that's partly because I have super-linear algorithms downstream of the parsing.
CBOR is much nicer and faster to work with (smaller documents, definite length strings directly consumable) and I do support it in my library, but I derive most of my performance gains by ditching DOM-style APIs and using stream processing instead. Allocating nodes and chasing pointers all over the heap sucks for performance, especially for large documents, regardless of the format (and that's assuming it even fits within available RAM).
weird-eye-issue 5 hours ago [-]
If you have proper indexes into the JSON columns I wonder what the performance difference is, if any, for Postgres
lofaszvanitt 5 hours ago [-]
Everyone needs their aha moment :}.
galkk 2 hours ago [-]
Use better databases? Learn prior art?
Dremel paper explained shredding in like 2010? Parquet had it in a while?
Yes, things like nested repeated fields (or structs) will mess you up hard time. When I was playing with Duckdb querying parquet several years ago I had issues with column names and even was digging some source code to improve statistics use, but it was doable and reasonably efficient on my datasets.
Although yes. If you have resources of large companies, both computational and manpower, you can do very nice things with your data. All that determination of “what would be nice to shred” based on real query statistics, compaction, regular optimization and improvements of data at rest by maintenance jobs… again, use better databases/engines and there’s no free lunch, you get what you paid for (hopefully).
japgolly 7 hours ago [-]
> Rather than storing all attributes in a single JSON column, we automatically extract the most frequently accessed attributes into separate, strongly typed columns.
I'm glad they made the discovery but this is DB 101
Jean-Papoulos 3 hours ago [-]
Re-discovering databases from first principles.
stuaxo 1 hours ago [-]
Feels insane to not do this.
MrThoughtful 3 hours ago [-]
Why not store everything in a single json column but dynamically generate index files, so finding rows is fast?
Then you never have to change the data. Only the index files.
JSON is slow as hell, both for serdes, and especially querying it. Use it sparingly, if at all. Normalization is not that hard, and it's trivial now with LLMs. Tell Claude you want your schema in BCNF, or 5NF if you're feeling froggy. Denormalize if and only if you have performance issues.
Since our visualization tool went from 2 seconds per click (when I set it up the original DB) to over 40 now, even before implementing this change, I'm watching in amazement, waiting for someone to complain. It's truly incredible how tolerant users are. It's one of the reasons I think AI will take our jobs much sooner than most think: the average user has no useful perspective on silly things like performance or determinism, "that's just how it is".
I'm working on an embedded system with a MCU and I've benchmarked JSON serialization at over 500 KiB/s (through a HTTP GET request) using my own library. Deserialization is slower, but that's partly because I have super-linear algorithms downstream of the parsing.
CBOR is much nicer and faster to work with (smaller documents, definite length strings directly consumable) and I do support it in my library, but I derive most of my performance gains by ditching DOM-style APIs and using stream processing instead. Allocating nodes and chasing pointers all over the heap sucks for performance, especially for large documents, regardless of the format (and that's assuming it even fits within available RAM).
Dremel paper explained shredding in like 2010? Parquet had it in a while?
https://blog.x.com/engineering/en_us/a/2013/dremel-made-simp...
Yes, things like nested repeated fields (or structs) will mess you up hard time. When I was playing with Duckdb querying parquet several years ago I had issues with column names and even was digging some source code to improve statistics use, but it was doable and reasonably efficient on my datasets.
Although yes. If you have resources of large companies, both computational and manpower, you can do very nice things with your data. All that determination of “what would be nice to shred” based on real query statistics, compaction, regular optimization and improvements of data at rest by maintenance jobs… again, use better databases/engines and there’s no free lunch, you get what you paid for (hopefully).
I'm glad they made the discovery but this is DB 101
Then you never have to change the data. Only the index files.