I have a spring boot application that uses jni to load open source maps data(pbf file). I do not load the whole world map at once just one country and the file size is about 1GB.
Loading the map and creating contraction hierarchy from pdf file is a one time task(but a heavy task), so I chose the $24 droplet. It has 2 cpu’s and 4 gb ram and should be enough to handle the task.
But I my task keep getting killed with the following log in the syslog file
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524312] java invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524320] CPU: 0 PID: 2483 Comm: java Not tainted 5.19.0-23-generic #24-Ubuntu
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524323] Hardware name: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524325] Call Trace:
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524327] <TASK>
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524330] show_stack+0x4e/0x61
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524339] dump_stack_lvl+0x4a/0x6d
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524345] dump_stack+0x10/0x18
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524347] dump_header+0x53/0x246
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524350] oom_kill_process.cold+0xb/0x10
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524352] out_of_memory+0x101/0x2f0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524358] __alloc_pages_may_oom+0x112/0x1e0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524362] __alloc_pages_slowpath.constprop.0+0x4ac/0x9b0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524365] __alloc_pages+0x31d/0x350
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524367] alloc_pages+0x90/0x1c0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524370] folio_alloc+0x1d/0x60
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524372] filemap_alloc_folio+0x8e/0xb0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524374] __filemap_get_folio+0x1c7/0x3c0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524377] filemap_fault+0x144/0x910
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524379] __do_fault+0x39/0x120
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524383] do_read_fault+0xf5/0x170
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524386] do_fault+0xa6/0x300
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524388] handle_pte_fault+0x117/0x240
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524390] __handle_mm_fault+0x696/0x740
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524394] handle_mm_fault+0xba/0x2a0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524396] do_user_addr_fault+0x1c1/0x680
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524401] exc_page_fault+0x80/0x1b0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524405] asm_exc_page_fault+0x27/0x30
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524408] RIP: 0033:0x7f381928d610
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524414] Code: Unable to access opcode bytes at RIP 0x7f381928d5e6.
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524415] RSP: 002b:00007f387e7dc2c0 EFLAGS: 00010287
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524417] RAX: 00007f3785200000 RBX: 00007f387e7dc3f0 RCX: 00007f37a425a040
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524418] RDX: 00007f372425a040 RSI: 0000000080001000 RDI: 0000000000000000
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524429] RBP: 0000000080000000 R08: 00000000ffffffff R09: 0000000000000000
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524430] R10: 0000000000000022 R11: 0000000000000246 R12: 00007f387e7dcdb8
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524431] R13: 00007f38798ab8b0 R14: 0000000000000000 R15: 0000000400000000
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524434] </TASK>
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524435] Mem-Info:
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524437] active_anon:234 inactive_anon:948731 isolated_anon:0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524437] active_file:9 inactive_file:0 isolated_file:0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524437] unevictable:6870 dirty:0 writeback:0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524437] slab_reclaimable:8128 slab_unreclaimable:10801
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524437] mapped:2238 shmem:246 pagetables:2435 bounce:0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524437] kernel_misc_reclaimable:0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524437] free:21504 free_pcp:62 free_cma:0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524441] Node 0 active_anon:936kB inactive_anon:3794924kB active_file:36kB inactive_file:0kB unevictable:27480kB isolated(anon):0kB isolated(file):0kB mapped:8952kB dirty:0kB writeback:0kB shmem:984kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:2752kB pagetables:9740kB all_unreclaimable? yes
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524444] Node 0 DMA free:14848kB boost:0kB min:260kB low:324kB high:388kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524452] lowmem_reserve[]: 0 2917 3869 3869 3869
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524455] Node 0 DMA32 free:54696kB boost:2048kB min:52796kB low:65480kB high:78164kB reserved_highatomic:0KB active_anon:8kB inactive_anon:2954876kB active_file:56kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129184kB managed:3026832kB mlocked:0kB bounce:0kB free_pcp:248kB local_pcp:248kB free_cma:0kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524459] lowmem_reserve[]: 0 0 952 952 952
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524462] Node 0 Normal free:16472kB boost:0kB min:16568kB low:20708kB high:24848kB reserved_highatomic:0KB active_anon:928kB inactive_anon:839660kB active_file:80kB inactive_file:0kB unevictable:27480kB writepending:0kB present:1048576kB managed:975408kB mlocked:27480kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524465] lowmem_reserve[]: 0 0 0 0 0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524468] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14848kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524477] Node 0 DMA32: 24*4kB (E) 146*8kB (UME) 124*16kB (UME) 113*32kB (E) 122*64kB (UME) 97*128kB (UME) 58*256kB (UME) 19*512kB (UE) 3*1024kB (UM) 0*2048kB 0*4096kB = 54736kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524489] Node 0 Normal: 563*4kB (UME) 529*8kB (UE) 254*16kB (UE) 86*32kB (UE) 36*64kB (UME) 9*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16756kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524499] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524501] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524502] 2500 total pagecache pages
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524503] 0 pages in swap cache
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524503] Swap cache stats: add 0, delete 0, find 0/0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524505] Free swap = 0kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524505] Total swap = 0kB
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524506] 1048438 pages RAM
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524507] 0 pages HighMem/MovableOnly
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524507] 44038 pages reserved
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524508] 0 pages hwpoisoned
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524509] Tasks state (memory values in pages):
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524509] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524514] [ 355] 0 355 8117 1011 77824 0 -250 systemd-journal
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524518] [ 394] 0 394 72289 6740 114688 0 -1000 multipathd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524520] [ 486] 101 486 22368 693 77824 0 0 systemd-timesyn
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524522] [ 552] 100 552 4343 899 65536 0 0 systemd-network
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524524] [ 571] 104 571 4769 864 69632 0 0 systemd-resolve
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524525] [ 595] 0 595 2792 926 57344 0 -1000 systemd-udevd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524527] [ 704] 0 704 1789 574 49152 0 0 cron
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524529] [ 705] 102 705 2354 1019 57344 0 -900 dbus-daemon
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524531] [ 711] 0 711 20690 640 61440 0 0 irqbalance
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524532] [ 714] 103 714 55559 1023 77824 0 0 rsyslogd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524534] [ 715] 0 715 368610 2520 262144 0 -900 snapd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524535] [ 716] 0 716 4138 909 73728 0 0 systemd-logind
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524537] [ 776] 0 776 1516 192 49152 0 0 agetty
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524539] [ 778] 0 778 27388 2808 114688 0 0 unattended-upgr
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524541] [ 789] 0 789 1505 204 45056 0 0 agetty
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524542] [ 1251] 0 1251 3712 1067 65536 0 -1000 sshd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524544] [ 1354] 0 1354 74095 1439 163840 0 0 packagekitd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524546] [ 1358] 0 1358 58460 798 86016 0 0 polkitd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524549] [ 1415] 0 1415 4395 1097 86016 0 0 sshd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524551] [ 1529] 0 1529 4575 1129 77824 0 100 systemd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524553] [ 1530] 0 1530 26106 1228 90112 0 100 (sd-pam)
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524557] [ 1638] 0 1638 2206 1016 61440 0 0 bash
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524559] [ 1717] 0 1717 343290 218 172032 0 -900 droplet-agent
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524561] [ 1817] 0 1817 4407 1383 77824 0 0 sshd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524562] [ 1859] 0 1859 2242 948 61440 0 0 bash
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524564] [ 1961] 0 1961 3712 977 69632 0 0 sshd
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524566] [ 2482] 0 2482 2716524 934410 7901184 0 0 java
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524567] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-1.scope,task=java,pid=2482,uid=0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [ 612.524623] Out of memory: Killed process 2482 (java) total-vm:10866096kB, anon-rss:3736748kB, file-rss:892kB, shmem-rss:0kB, UID:0 pgtables:7716kB oom_score_adj:0
Nov 4 18:56:11 ubuntu-c-2-4gib-blr1-01 systemd[1]: session-1.scope: A process of this unit has been killed by the OOM killer.
I have already tried the swap file tutorial - https://www.digitalocean.com/community/articles/how-to-add-swap-on-ubuntu-12-04
I also tried to work with a dedicated cpu optimised droplet but still the same thing.
The application works well on a docker image on my local system so I tried to boot a droplet with docker and use the same image but I still get the same OOM killed error
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Hi @giantlightseagreenwhale-2432612431302459356342492f53744c365233326851504a58306d6565,
Apart from what has already been mentioned, I’ll suggest adding SWAP as well. 2GBs of Swap should be enough to help you out. Here is a tutorial on how to add it to your Droplet;
https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04
Hello @giantlightseagreenwhale-2432612431302459356342492f53744c365233326851504a58306d6565
You might want to check out the memory optimized droplets and spin up one in order to resolve the oom issues.
Another thing to do is to monitor the server’s resource usage, as there might be another process that consumes big chunks of the RAM.
You can also use tools like CSF and add the java process to the ignore list in order to prevent the kernel killer to stop the process.
Regards
Want to learn more? Join the DigitalOcean Community!
Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.
Sign up now