Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

650GB relates to size of parquet files which are compressed in reality it’s way more.

32 GB of parquet cannot fit in 32GB of RAM



You don't need it to if you just need specific columns. This is the advantage of columnar storage.


This would speed things up since it looks like the bottleneck here is I/O.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: