Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
co0lster
33 days ago
|
parent
|
context
|
favorite
| on:
650GB of Data (Delta Lake on S3). Polars vs. DuckD...
650GB relates to size of parquet files which are compressed in reality it’s way more.
32 GB of parquet cannot fit in 32GB of RAM
m00x
33 days ago
|
next
[–]
You don't need it to if you just need specific columns. This is the advantage of columnar storage.
barrkel
33 days ago
|
prev
[–]
This would speed things up since it looks like the bottleneck here is I/O.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
32 GB of parquet cannot fit in 32GB of RAM