Question

Java jni out of memory killed

I have a spring boot application that uses jni to load open source maps data(pbf file). I do not load the whole world map at once just one country and the file size is about 1GB.

Loading the map and creating contraction hierarchy from pdf file is a one time task(but a heavy task), so I chose the $24 droplet. It has 2 cpu’s and 4 gb ram and should be enough to handle the task.

But I my task keep getting killed with the following log in the syslog file

Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524312] java invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524320] CPU: 0 PID: 2483 Comm: java Not tainted 5.19.0-23-generic #24-Ubuntu
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524323] Hardware name: DigitalOcean Droplet/Droplet, BIOS 20171212 12/12/2017
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524325] Call Trace:
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524327]  <TASK>
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524330]  show_stack+0x4e/0x61
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524339]  dump_stack_lvl+0x4a/0x6d
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524345]  dump_stack+0x10/0x18
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524347]  dump_header+0x53/0x246
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524350]  oom_kill_process.cold+0xb/0x10
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524352]  out_of_memory+0x101/0x2f0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524358]  __alloc_pages_may_oom+0x112/0x1e0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524362]  __alloc_pages_slowpath.constprop.0+0x4ac/0x9b0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524365]  __alloc_pages+0x31d/0x350
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524367]  alloc_pages+0x90/0x1c0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524370]  folio_alloc+0x1d/0x60
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524372]  filemap_alloc_folio+0x8e/0xb0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524374]  __filemap_get_folio+0x1c7/0x3c0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524377]  filemap_fault+0x144/0x910
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524379]  __do_fault+0x39/0x120
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524383]  do_read_fault+0xf5/0x170
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524386]  do_fault+0xa6/0x300
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524388]  handle_pte_fault+0x117/0x240
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524390]  __handle_mm_fault+0x696/0x740
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524394]  handle_mm_fault+0xba/0x2a0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524396]  do_user_addr_fault+0x1c1/0x680
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524401]  exc_page_fault+0x80/0x1b0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524405]  asm_exc_page_fault+0x27/0x30
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524408] RIP: 0033:0x7f381928d610
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524414] Code: Unable to access opcode bytes at RIP 0x7f381928d5e6.
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524415] RSP: 002b:00007f387e7dc2c0 EFLAGS: 00010287
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524417] RAX: 00007f3785200000 RBX: 00007f387e7dc3f0 RCX: 00007f37a425a040
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524418] RDX: 00007f372425a040 RSI: 0000000080001000 RDI: 0000000000000000
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524429] RBP: 0000000080000000 R08: 00000000ffffffff R09: 0000000000000000
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524430] R10: 0000000000000022 R11: 0000000000000246 R12: 00007f387e7dcdb8
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524431] R13: 00007f38798ab8b0 R14: 0000000000000000 R15: 0000000400000000
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524434]  </TASK>
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524435] Mem-Info:
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524437] active_anon:234 inactive_anon:948731 isolated_anon:0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524437]  active_file:9 inactive_file:0 isolated_file:0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524437]  unevictable:6870 dirty:0 writeback:0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524437]  slab_reclaimable:8128 slab_unreclaimable:10801
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524437]  mapped:2238 shmem:246 pagetables:2435 bounce:0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524437]  kernel_misc_reclaimable:0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524437]  free:21504 free_pcp:62 free_cma:0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524441] Node 0 active_anon:936kB inactive_anon:3794924kB active_file:36kB inactive_file:0kB unevictable:27480kB isolated(anon):0kB isolated(file):0kB mapped:8952kB dirty:0kB writeback:0kB shmem:984kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:2752kB pagetables:9740kB all_unreclaimable? yes
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524444] Node 0 DMA free:14848kB boost:0kB min:260kB low:324kB high:388kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524452] lowmem_reserve[]: 0 2917 3869 3869 3869
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524455] Node 0 DMA32 free:54696kB boost:2048kB min:52796kB low:65480kB high:78164kB reserved_highatomic:0KB active_anon:8kB inactive_anon:2954876kB active_file:56kB inactive_file:0kB unevictable:0kB writepending:0kB present:3129184kB managed:3026832kB mlocked:0kB bounce:0kB free_pcp:248kB local_pcp:248kB free_cma:0kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524459] lowmem_reserve[]: 0 0 952 952 952
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524462] Node 0 Normal free:16472kB boost:0kB min:16568kB low:20708kB high:24848kB reserved_highatomic:0KB active_anon:928kB inactive_anon:839660kB active_file:80kB inactive_file:0kB unevictable:27480kB writepending:0kB present:1048576kB managed:975408kB mlocked:27480kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524465] lowmem_reserve[]: 0 0 0 0 0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524468] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14848kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524477] Node 0 DMA32: 24*4kB (E) 146*8kB (UME) 124*16kB (UME) 113*32kB (E) 122*64kB (UME) 97*128kB (UME) 58*256kB (UME) 19*512kB (UE) 3*1024kB (UM) 0*2048kB 0*4096kB = 54736kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524489] Node 0 Normal: 563*4kB (UME) 529*8kB (UE) 254*16kB (UE) 86*32kB (UE) 36*64kB (UME) 9*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16756kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524499] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524501] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524502] 2500 total pagecache pages
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524503] 0 pages in swap cache
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524503] Swap cache stats: add 0, delete 0, find 0/0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524505] Free swap  = 0kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524505] Total swap = 0kB
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524506] 1048438 pages RAM
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524507] 0 pages HighMem/MovableOnly
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524507] 44038 pages reserved
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524508] 0 pages hwpoisoned
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524509] Tasks state (memory values in pages):
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524509] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524514] [    355]     0   355     8117     1011    77824        0          -250 systemd-journal
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524518] [    394]     0   394    72289     6740   114688        0         -1000 multipathd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524520] [    486]   101   486    22368      693    77824        0             0 systemd-timesyn
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524522] [    552]   100   552     4343      899    65536        0             0 systemd-network
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524524] [    571]   104   571     4769      864    69632        0             0 systemd-resolve
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524525] [    595]     0   595     2792      926    57344        0         -1000 systemd-udevd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524527] [    704]     0   704     1789      574    49152        0             0 cron
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524529] [    705]   102   705     2354     1019    57344        0          -900 dbus-daemon
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524531] [    711]     0   711    20690      640    61440        0             0 irqbalance
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524532] [    714]   103   714    55559     1023    77824        0             0 rsyslogd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524534] [    715]     0   715   368610     2520   262144        0          -900 snapd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524535] [    716]     0   716     4138      909    73728        0             0 systemd-logind
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524537] [    776]     0   776     1516      192    49152        0             0 agetty
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524539] [    778]     0   778    27388     2808   114688        0             0 unattended-upgr
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524541] [    789]     0   789     1505      204    45056        0             0 agetty
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524542] [   1251]     0  1251     3712     1067    65536        0         -1000 sshd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524544] [   1354]     0  1354    74095     1439   163840        0             0 packagekitd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524546] [   1358]     0  1358    58460      798    86016        0             0 polkitd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524549] [   1415]     0  1415     4395     1097    86016        0             0 sshd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524551] [   1529]     0  1529     4575     1129    77824        0           100 systemd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524553] [   1530]     0  1530    26106     1228    90112        0           100 (sd-pam)
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524557] [   1638]     0  1638     2206     1016    61440        0             0 bash
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524559] [   1717]     0  1717   343290      218   172032        0          -900 droplet-agent
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524561] [   1817]     0  1817     4407     1383    77824        0             0 sshd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524562] [   1859]     0  1859     2242      948    61440        0             0 bash
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524564] [   1961]     0  1961     3712      977    69632        0             0 sshd
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524566] [   2482]     0  2482  2716524   934410  7901184        0             0 java
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524567] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-1.scope,task=java,pid=2482,uid=0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 kernel: [  612.524623] Out of memory: Killed process 2482 (java) total-vm:10866096kB, anon-rss:3736748kB, file-rss:892kB, shmem-rss:0kB, UID:0 pgtables:7716kB oom_score_adj:0
Nov  4 18:56:11 ubuntu-c-2-4gib-blr1-01 systemd[1]: session-1.scope: A process of this unit has been killed by the OOM killer.

I have already tried the swap file tutorial - https://www.digitalocean.com/community/articles/how-to-add-swap-on-ubuntu-12-04

I also tried to work with a dedicated cpu optimised droplet but still the same thing.

The application works well on a docker image on my local system so I tried to boot a droplet with docker and use the same image but I still get the same OOM killed error


Submit an answer
Answer a question...

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

KFSys
Site Moderator
Site Moderator badge
November 5, 2022

Hi @giantlightseagreenwhale-2432612431302459356342492f53744c365233326851504a58306d6565,

Apart from what has already been mentioned, I’ll suggest adding SWAP as well. 2GBs of Swap should be enough to help you out. Here is a tutorial on how to add it to your Droplet;

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04

alexdo
Site Moderator
Site Moderator badge
November 5, 2022

Hello @giantlightseagreenwhale-2432612431302459356342492f53744c365233326851504a58306d6565

You might want to check out the memory optimized droplets and spin up one in order to resolve the oom issues.

Another thing to do is to monitor the server’s resource usage, as there might be another process that consumes big chunks of the RAM.

You can also use tools like CSF and add the java process to the ignore list in order to prevent the kernel killer to stop the process.

Regards

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.