csselo
By:
csselo

Out of Memory Killed

August 16, 2017 811 views
Python Ubuntu 16.04

Hi

Syslog


Aug 16 13:01:23 ubuntu-512mb-nyc3-01 kernel: [12176.988105] Out of memory: Kill process 22112 (python) score 587 or sacrifice child
Aug 16 13:01:23 ubuntu-512mb-nyc3-01 kernel: [12176.991195] Killed process 22112 (python) total-vm:1299692kB, anon-rss:68292kB, file-rss:1728kB
Au

All Syslog

Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204461] python invoked oom-killer: gfp_mask=0x24200ca, order=0, oom_score_adj=0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204492] python cpuset=/ mems_allowed=0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204604] CPU: 0 PID: 22563 Comm: python Not tainted 4.4.0-87-generic #110-Ubuntu
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204607] Hardware name: DigitalOcean Droplet, BIOS 20161103 11/03/2016
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204636]  0000000000000286 5d5238e1f22738b7 ffff88001cb739b8 ffffffff813f9903
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204670]  ffff88001cb73b70 ffff88001ab5f000 ffff88001cb73a28 ffffffff8120b75e
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204676]  ffffffff81cd6fe7 0000000000000000 ffffffff81e67720 0000000000000206
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204682] Call Trace:
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.204974]  [<ffffffff813f9903>] dump_stack+0x63/0x90
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205075]  [<ffffffff8120b75e>] dump_header+0x5a/0x1c5
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205139]  [<ffffffff81192ce2>] oom_kill_process+0x202/0x3c0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205213]  [<ffffffff81193109>] out_of_memory+0x219/0x460
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205253]  [<ffffffff811990f8>] __alloc_pages_slowpath.constprop.88+0x938/0xad0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205260]  [<ffffffff81199516>] __alloc_pages_nodemask+0x286/0x2a0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205279]  [<ffffffff811e489d>] alloc_pages_vma+0xad/0x250
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205299]  [<ffffffff811d52fe>] __read_swap_cache_async+0xee/0x140
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205304]  [<ffffffff811d5376>] read_swap_cache_async+0x26/0x60
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205307]  [<ffffffff811d54b5>] swapin_readahead+0x105/0x1b0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205359]  [<ffffffff810cb061>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205393]  [<ffffffff811c2667>] handle_mm_fault+0x1317/0x1820
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205415]  [<ffffffff810a9c76>] ? finish_task_switch+0x76/0x220
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205477]  [<ffffffff8106b577>] __do_page_fault+0x197/0x400
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205483]  [<ffffffff8106b847>] trace_do_page_fault+0x37/0xe0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205502]  [<ffffffff81063f29>] do_async_page_fault+0x19/0x70
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205597]  [<ffffffff81844068>] async_page_fault+0x28/0x30
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205679] Mem-Info:
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205689] active_anon:22875 inactive_anon:87452 isolated_anon:32
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205689]  active_file:63 inactive_file:59 isolated_file:0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205689]  unevictable:915 dirty:0 writeback:0 unstable:0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205689]  slab_reclaimable:3276 slab_unreclaimable:3287
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205689]  mapped:709 shmem:50 pagetables:1997 bounce:0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205689]  free:1645 free_pcp:142 free_cma:0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205702] Node 0 DMA free:1920kB min:88kB low:108kB high:132kB active_anon:6144kB inactive_anon:6332kB active_file:0kB inactive_file:0kB unevictable:100kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:100kB dirty:0kB writeback:0kB mapped:48kB shmem:0kB slab_reclaimable:420kB slab_unreclaimable:524kB kernel_stack:16kB pagetables:172kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:4 all_unreclaimable? yes
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205739] lowmem_reserve[]: 0 458 458 458 458
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205745] Node 0 DMA32 free:4660kB min:2696kB low:3368kB high:4044kB active_anon:85356kB inactive_anon:343476kB active_file:252kB inactive_file:236kB unevictable:3560kB isolated(anon):128kB isolated(file):0kB present:507884kB managed:484152kB mlocked:3560kB dirty:0kB writeback:0kB mapped:2788kB shmem:200kB slab_reclaimable:12684kB slab_unreclaimable:12624kB kernel_stack:2688kB pagetables:7816kB unstable:0kB bounce:0kB free_pcp:568kB local_pcp:568kB free_cma:0kB writeback_tmp:0kB pages_scanned:18484 all_unreclaimable? yes
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205762] lowmem_reserve[]: 0 0 0 0 0
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205770] Node 0 DMA: 20*4kB (ME) 11*8kB (UE) 10*16kB (UME) 24*32kB (UME) 9*64kB (UM) 2*128kB (ME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1928kB
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205839] Node 0 DMA32: 41*4kB (UMEH) 158*8kB (UMEH) 68*16kB (UMH) 25*32kB (UME) 3*64kB (UME) 1*128kB (M) 0*256kB 0*512kB 1*1024kB (H) 0*2048kB 0*4096kB = 4660kB
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205890] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205896] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205899] 65433 total pagecache pages
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205903] 64650 pages in swap cache
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205907] Swap cache stats: add 10583356, delete 10518706, find 4609366/8073610
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205909] Free swap  = 0kB
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205912] Total swap = 1048572kB
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205916] 130969 pages RAM
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205918] 0 pages HighMem/MovableOnly
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205921] 5954 pages reserved
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205923] 0 pages cma reserved
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205926] 0 pages hwpoisoned
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205941] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205975] [  619]     0   619     9077      413      20       4      585             0 systemd-journal
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205984] [  661]     0   661    23693      102      17       3       53             0 lvmetad
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205991] [  690]     0   690    10637      421      23       3      205         -1000 systemd-udevd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.205997] [  867]   100   867    25081      360      19       4       70             0 systemd-timesyn
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206003] [ 1258]     0  1258     1306        8       9       3       23             0 iscsid
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206008] [ 1259]     0  1259     1431      881       9       3        0           -17 iscsid
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206014] [ 1267]   104  1267    64099      127      27       3      292             0 rsyslogd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206020] [ 1280]   107  1280    10725      365      26       3      126          -900 dbus-daemon
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206026] [ 1292]     0  1292     6932      387      18       3       87             0 cron
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206030] [ 1294]     0  1294     1100      296       8       3       37             0 acpid
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206035] [ 1299]     0  1299    68622      194      36       3      181             0 accounts-daemon
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206040] [ 1302]     0  1302     7137      247      19       3       87             0 systemd-logind
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206045] [ 1305]     0  1305     6511      336      18       3       59             0 atd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206051] [ 1308]     0  1308    40226      146      15       3       81             0 lxcfs
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206057] [ 1314]     0  1314    16380      442      36       3      183         -1000 sshd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206063] [ 1317]     0  1317    69629      978      31       6     1434             0 snapd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206068] [ 1349]     0  1349     3344       56      11       3       48             0 mdadm
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206074] [ 1354]     0  1354    69295        5      40       3      174             0 polkitd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206079] [ 1411]     0  1411     3619      361      11       3       39             0 agetty
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206084] [ 1576]     0  1576    16458      442      37       3      127             0 login
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206089] [ 1615]  1000  1615    11312      117      26       3      204             0 systemd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206094] [ 1621]  1000  1621    15324        8      33       3      496             0 (sd-pam)
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206100] [ 1624]  1000  1624     5333      443      15       3      479             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206106] [19541]   112 19541   278986        0     140       4    33718             0 mysqld
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206112] [19871]     0 19871     9200       80      22       3      193             0 systemd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206118] [19873]     0 19873    15318        6      33       3      492             0 (sd-pam)
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206124] [20069]     0 20069    23842      347      48       3      250             0 sshd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206129] [20101]     0 20101     5325      417      14       3      470             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206134] [20119]     0 20119    12744      405      30       3      106             0 su
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206140] [20120]  1000 20120     5354      443      15       3      500             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206146] [21206]  1000 21206    12751      409      30       3      106             0 su
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206151] [21209]     0 21209     4980      414      14       3      125             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206156] [21430]     0 21430    12751      398      29       3      106             0 su
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206162] [21431]  1000 21431     5332      384      14       3      478             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206167] [21525]  1000 21525    12751      403      28       3      106             0 su
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206173] [21530]     0 21530     4980      410      15       3      125             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206179] [22093]     0 22093    12751      405      30       3      106             0 su
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206184] [22094]  1000 22094     5330      443      15       3      479             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206190] [22209]     0 22209    23842      284      49       3      245             0 sshd
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206209] [22265]     0 22265     5325      437      15       3      470             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206216] [22314]     0 22314    12744      399      30       3      106             0 su
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206236] [22315]  1000 22315     5331      430      15       4      477             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206242] [22533]  1000 22533    12751      405      30       3      106             0 su
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206247] [22534]     0 22534     4981      402      14       3      126             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206253] [22544]     0 22544    12751      405      30       3      104             0 su
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206259] [22545]     0 22545     4981      402      14       3      127             0 bash
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206265] [22563]  1000 22563   352586    44632     637       3   209941             0 python
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206270] [22670]     0 22670     1511       25       8       3       34             0 tail
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206284] [22736]     0 22736    12235      410      28       3      106             0 cron
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206290] [22737]  1000 22737     1127      128       8       3       25             0 sh
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206296] [22738]  1000 22738     1127      124       7       3       24             0 sh
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206301] [22740]  1000 22740    79634      467     107       4     7568             0 python
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.206340] Out of memory: Kill process 22563 (python) score 659 or sacrifice child
Aug 16 13:31:18 ubuntu-512mb-nyc3-01 kernel: [13972.211658] Killed process 22563 (python) total-vm:1410344kB, anon-rss:176748kB, file-rss:1780kB

swap


              total        used        free      shared  buff/cache   available
Mem:           488M        167M        163M        332K        157M        292M
Swap:          1.0G        164M        859M

I only install mysql server and python lib
It always crashes.

what can I do?

2 Answers

You need to allocate the memory to mysql in the configuration file.

I resized swapfile 1G to 4G
set

vm.vfscachepressure = 80
vm.swappiness=80

solved

link
https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04

One of the easiest way of increasing the responsiveness of your server and guarding against out of memory errors in your applications is to add some swap space. In this guide, we will cover how to add a swap file to an Ubuntu 16.04 server. &lt;$&gt;[warning] [label...
Have another answer? Share your knowledge.