I am on 8GB RAM plan, yet facing difficulties in reading 4GB file into the python pandas dataframe. Getting out of memory error. Already allocated 4GB swap file, yet issue is not solved.
Could you pls. support?
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
[64152.869711] Out of memory: Kill process 7499 (python3) score 972 or sacrifice child
[64152.874659] Killed process 7499 (python3) total -vm: 12671396kB, anon-rss:79017
free - m
Mem:
total: 7.8G
used: 4.3M
free: 3.5G
shared: 2.7M
buffers: 8.1M
cached: 38M
Swap:
total: 4.0G
used: 2.9G
free: 1.1G
shared: -
buffers:-
cached: -
Wow, 4GB single file reading directly in python on 8GB system? I would extend the swap space to at least 6GB (mkswap I mean) then try to ask kernel to use swap efficiently by modifying swappiness:
Read it here: https://askubuntu.com/questions/103915/how-do-i-configure-swappiness