服务器内存不足导致LINUX KILL MYSQLD 故障

OS操作系统：SUSE Linux 11

数据库版 本：MYSQL 5.6

 

客户反映MYSQL数据库因为内存不足被重启。

1、根据MYSQL报警日志确认问题：

2016-10-24 12:28:49 29226 [Note] Plugin 'FEDERATED' is disabled.

2016-10-24 12:28:49 29226 [Note] InnoDB: Using atomics to ref count buffer pool pages

2016-10-24 12:28:49 29226 [Note] InnoDB: The InnoDB memory heap is disabled

2016-10-24 12:28:49 29226 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2016-10-24 12:28:49 29226 [Note] InnoDB: Compressed tables use zlib 1.2.3

2016-10-24 12:28:49 29226 [Note] InnoDB: Using Linux native AIO

2016-10-24 12:28:49 29226 [Note] InnoDB: Using CPU crc32 instructions

2016-10-24 12:28:49 29226 [Note] InnoDB: Initializing buffer pool, size = 2.0G

2016-10-24 12:28:50 29226 [Note] InnoDB: Completed initialization of buffer pool

2016-10-24 12:28:50 29226 [Note] InnoDB: Highest supported file format is Barracuda.

2016-10-24 12:28:50 29226 [Note] InnoDB: Log scan progressed past the checkpoint lsn 27852365485

2016-10-24 12:28:50 29226 [Note] InnoDB: Database was not shutdown normally!

2016-10-24 12:28:50 29226 [Note] InnoDB: Starting crash recovery.

2016-10-24 12:28:50 29226 [Note] InnoDB: Reading tablespace information from the .ibd files...

2016-10-24 12:28:50 29226 [Note] InnoDB: Restoring possible half-written data pages

2016-10-24 12:28:50 29226 [Note] InnoDB: from the doublewrite buffer...

InnoDB: Doing recovery: scanned up to log sequence number 27852370215

2016-10-24 12:28:51 29226 [Note] InnoDB: Starting an apply batch of log records to the database...

InnoDB: Progress in percent: 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

InnoDB: Apply batch completed

InnoDB: Last MySQL binlog file position 0 401322767, file name TSPTONLDB01-it_mgmad-bin.000023

2016-10-24 12:28:52 29226 [Note] InnoDB: 128 rollback segment(s) are active.

2016-10-24 12:28:52 29226 [Note] InnoDB: 5.6.19 started; log sequence number 27852370215

2016-10-24 12:28:52 29226 [Note] Recovering after a crash using /dbdata/it_mgmad/binlog/it_mgmad/TSPTONLDB01-it_mgmad-bin

2016-10-24 12:29:00 29226 [Note] Starting crash recovery...

2016-10-24 12:29:00 29226 [Note] Crash recovery finished.

2016-10-24 12:29:06 29226 [Note] Server hostname (bind-address): '*'; port: 60002

2016-10-24 12:29:06 29226 [Note] IPv6 is available.

2016-10-24 12:29:06 29226 [Note]   - '::' resolves to '::';

2016-10-24 12:29:06 29226 [Note] Server socket created on IP: '::'.

2016-10-24 12:29:06 29226 [Warning] 'proxies_priv' entry '@ root@tsptonldb01' ignored in --skip-name-resolve mode.

2016-10-24 12:29:06 29226 [Note] Event Scheduler: Loaded 0 events

2016-10-24 12:29:06 29226 [Note] /usr/local/mysql/bin/mysqld: ready for connections.

Version: '5.6.19-upsql-0.1.1-log'  socket: '/etc/mysock/it_mgmad.sock'  port: 60002  UpSQL Server

2016-10-24 12:30:21 29226 [ERROR] /usr/local/mysql/bin/mysqld: Out of memory (Needed 8160 bytes)

2016-10-24 12:30:21 29226 [ERROR] Out of memory; check if mysqld or some other process uses all available memory; if not, you may have to use 'ulimit' to allow mysqld to use more memory or you can add more swap space

161024 12:31:35 mysqld_safe Number of processes running now: 0

161024 12:31:35 mysqld_safe mysqld restarted

2016-10-24 12:31:37 29659 [Warning] Buffered warning: Could not increase number of max_open_files to more than 10240 (request: 25000)

 

 

从上面的信息，我们看到， MSYQL报内存不足的错误无法启动。

2. 操作系统日志提示错误

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996482] Mem-Info:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996483] Node 0 DMA per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996485] CPU    0: hi:    0, btch:   1 usd:   0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996486] CPU    1: hi:    0, btch:   1 usd:   0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996487] Node 0 DMA32 per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996488] CPU    0: hi:  186, btch:  31 usd:   0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996489] CPU    1: hi:  186, btch:  31 usd:   5

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996490] Node 0 Normal per-cpu:

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996491] CPU    0: hi:  186, btch:  31 usd:  30

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996492] CPU    1: hi:  186, btch:  31 usd: 144

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996495] active_anon:658762 inactive_anon:260562 isolated_anon:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996496]  active_file:353 inactive_file:1716 isolated_file:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996496]  unevictable:0 dirty:6 writeback:5 unstable:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996497]  free:21800 slab_reclaimable:2214 slab_unreclaimable:10742

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996498]  mapped:765 shmem:456 pagetables:6664 bounce:0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996499] Node 0 DMA free:15880kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15688kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996505] lowmem_reserve[]: 0 3000 4010 4010

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996507] Node 0 DMA32 free:54356kB min:50368kB low:62960kB high:75552kB active_anon:2275764kB inactive_anon:569032kB active_file:12kB inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072160kB mlocked:0kB dirty:0kB writeback:0kB mapped:32kB shmem:12kB slab_reclaimable:1896kB slab_unreclaimable:7060kB kernel_stack:2904kB pagetables:6588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:611 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996513] lowmem_reserve[]: 0 0 1010 1010

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996515] Node 0 Normal free:16964kB min:16956kB low:21192kB high:25432kB active_anon:359284kB inactive_anon:473216kB active_file:1400kB inactive_file:6776kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1034240kB mlocked:0kB dirty:24kB writeback:20kB mapped:3028kB shmem:1812kB slab_reclaimable:6960kB slab_unreclaimable:35908kB kernel_stack:3624kB pagetables:20068kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:12878 all_unreclaimable? yes

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996520] lowmem_reserve[]: 0 0 0 0

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996522] Node 0 DMA: 0*4kB 1*8kB 0*16kB 0*32kB 2*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15880kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996544] Node 0 DMA32: 1308*4kB 973*8kB 448*16kB 181*32kB 91*64kB 43*128kB 11*256kB 8*512kB 8*1024kB 1*2048kB 0*4096kB = 54456kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996549] Node 0 Normal: 439*4kB 303*8kB 169*16kB 83*32kB 18*64kB 5*128kB 4*256kB 3*512kB 3*1024kB 0*2048kB 0*4096kB = 16964kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996553] 38343 total pagecache pages

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996554] 35726 pages in swap cache

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996556] Swap cache stats: add 5506519, delete 5470793, find 22920240/23085658

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996557] Free swap  = 0kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317509.996557] Total swap = 2097148kB

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006955] 1048560 pages RAM

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006957] 67010 pages reserved

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006958] 1978 pages shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006959] 953768 pages non-shared

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006960] [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006974] [  571]     0   571     4223        1   1     -17         -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006981] [ 1689]     0  1689      998        1   0       0             0 acpid

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006983] [ 1704]   100  1704     5444      117   0       0             0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006986] [ 1722]     0  1722     4824       62   0       0             0 syslog-ng

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006988] [ 1725]     0  1725     1065        0   0       0             0 klogd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006991] [10038]     0 10038    31835      168   1       0             0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006994] [10096]     0 10096    15193        1   0       0             0 VGAuthService

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006996] [10185]   101 10185     8608      146   0       0             0 hald

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.006999] [10188]     0 10188    27278        0   0       0             0 console-kit-dae

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007001] [10189]     0 10189     4505        2   0       0             0 hald-runner

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007004] [10284]     0 10284     8690        0   0       0             0 hald-addon-inpu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007007] [10308]     0 10308     8690       17   0       0             0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007009] [10312]   101 10312     9129        0   1       0             0 hald-addon-acpi

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007012] [10313]     0 10313     8690       16   0       0             0 hald-addon-stor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007014] [12046]     0 12046     7409        8   0     -17         -1000 auditd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007017] [12048]     0 12048     3606        0   0       0             0 audispd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007020] [12081]     0 12081     4723       16   0       0             0 rpcbind

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007022] [12083]     0 12083     2088       17   0       0             0 haveged

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007025] [12140]     0 12140     4216        1   1     -17         -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007027] [12141]     0 12141     4216        1   0     -17         -1000 udevd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007029] [12224]     0 12224     6759        1   1       0             0 mcelog

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007031] [12701]     0 12701    17296       28   0     -17         -1000 sshd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007034] [12715]     0 12715    17933        0   1       0             0 cupsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007036] [12730]     0 12730    59557       71   0       0             0 nscd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007038] [12798]     0 12798     9552       28   0       0             0 master

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007041] [12825]    51 12825    10052       31   1       0             0 qmgr

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007043] [12864]     0 12864    16711       47   1       0             0 gdm

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007046] [12878]     0 12878    27377      121   0       0             0 X

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007048] [12885]     0 12885     3697       15   0       0             0 cron

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007051] [12924]     0 12924     1121        2   1       0             0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007055] [12925]     0 12925     1121        2   1       0             0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007058] [12926]     0 12926     1121        2   0       0             0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007061] [12927]     0 12927     1121        2   1       0             0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007064] [12928]     0 12928     1121        2   1       0             0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007067] [12929]     0 12929     1121        2   0       0             0 mingetty

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007071] [13064]   107 13064     5548        0   1       0             0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007074] [13065]   107 13065     3362        0   1       0             0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007077] [13066]   107 13066    39551       82   0       0             0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007080] [13079]   107 13079     9863       67   0       0             0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007083] [13083]   107 13083    31861        0   1       0             0 at-spi-registry

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007086] [13084]   107 13084    70831     2912   0       0             0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007090] [13086]   107 13086    20207        0   1       0             0 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007092] [13094]   107 13094    57536      135   0       0             0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007095] [13096]   107 13096    10854        0   0       0             0 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007098] [13097]   107 13097    54246        0   0       0             0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007101] [21349]  1018 21349     2857        2   1       0             0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007105] [12777]  1016 12777     2857        2   0       0             0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007109] [18450]     0 18450    13384       49   1       0             0 gdm-simple-slav

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007112] [18452]     0 18452    66809    13462   1       0             0 X

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007115] [18489]   107 18489     5548        0   1       0             0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007118] [18518]     0 18518    24613        0   1       0             0 gdm-session-wor

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007121] [18540]     0 18540    39584       89   1       0             0 gnome-session

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007124] [18613]     0 18613     4178       19   0       0             0 gpg-agent

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007126] [18619]     0 18619     5548        0   1       0             0 dbus-launch

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007129] [18620]     0 18620     3429        0   0       0             0 dbus-daemon

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007132] [18623]     0 18623     9887      123   1       0             0 gconfd-2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007135] [18627]     0 18627    91696      332   0       0             0 gnome-settings-

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007138] [18631]     0 18631    23814        1   1       0             0 gnome-keyring-d

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007141] [18635]     0 18635    10908        0   1       0             0 gvfsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007144] [18639]     0 18639    16822       19   0       0             0 gvfs-fuse-daemo

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007147] [18645]     0 18645    51782      400   1       0             0 metacity

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007149] [18650]     0 18650    93659      621   1       0             0 gnome-panel

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007152] [18651]     0 18651   135905    13563   1       0             0 nautilus

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007155] [18653]     0 18653    36624        0   0       0             0 bonobo-activati

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007160] [18659]     0 18659    82105      455   1       0             0 main-menu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007171] [18662]     0 18662    12083        0   1       0             0 gvfsd-trash

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007174] [18666]     0 18666    13609        1   0       0             0 gvfs-hal-volume

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007178] [18668]     0 18668    15774        1   1       0             0 gvfs-gphoto2-vo

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007181] [18677]     0 18677    75456        1   1       0             0 gnome-volume-co

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007184] [18681]     0 18681    56947        1   0       0             0 python

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007187] [18686]     0 18686    39298      113   0       0             0 vmtoolsd

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007189] [18688]     0 18688    57564      125   0       0             0 gnome-power-man

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007192] [18692]     0 18692    38020      114   1       0             0 gnome-screensav

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007195] [18693]     0 18693    48186        0   1       0             0 pulseaudio

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007198] [18718]     0 18718    10962        0   0       0             0 gvfsd-burn

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007201] [21417]     0 21417    59168      142   1       0             0 application-bro

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007204] [21420]     0 21420    57129      199   0       0             0 baobab

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007208] [21463]     0 21463    35883        1   0       0             0 gnomesu

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007211] [21464]     0 21464    13445        4   0       0             0 gnomesu-pam-bac

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007214] [21471]     0 21471     3221        1   0       0             0 yast2

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007217] [21501]     0 21501   118060      114   0       0             0 y2base

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007221] [21621]     0 21621    55974      123   0       0             0 gnome-terminal

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007224] [21622]     0 21622     1616        0   0       0             0 gnome-pty-helpe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007227] [21623]     0 21623     3505        1   0       0             0 bash

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007230] [21649]     0 21649     9244        1   0       0             0 gvfsd-metadata

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007233] [31142]  1016 31142  1100222   219744   1       0             0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007236] [31181]  1018 31181  1039509   163992   1       0             0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007239] [  699]  1017   699     2856       98   0       0             0 mysqld_safe

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007242] [ 1625]  1017  1625  1076677   414298   1       0             0 mysqld

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007246] [ 1861]  1019  1861   641372    48223   1       0             0 java

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007284] [14971]    51 14971    10014      128   0       0             0 pickup

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007287] Out of memory: Kill process 31142 (mysqld) score 283 or sacrifice child

Oct 25 13:41:07 TSPTONLDB01 kernel: [14317510.007358] Killed process 31142 (mysqld) total-vm:4400888kB, anon-rss:878976kB, file-rss:0kB

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351765] vmtoolsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351772] vmtoolsd cpuset=/ mems_allowed=0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351777] Pid: 10038, comm: vmtoolsd Tainted: G            X 3.0.13-0.27-default #1

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351780] Call Trace:

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351803]  [<ffffffff810048b5>] dump_trace+0x75/0x300

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351814]  [<ffffffff8143ea0f>] dump_stack+0x69/0x6f

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351825]  [<ffffffff810f2b6f>] dump_header+0x8f/0x1f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351834]  [<ffffffff810f3235>] oom_kill_process+0x95/0x180

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351841]  [<ffffffff810f346c>] out_of_memory+0x14c/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351848]  [<ffffffff810f8efb>] __alloc_pages_slowpath+0x7ab/0x7f0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351854]  [<ffffffff810f90f1>] __alloc_pages_nodemask+0x1b1/0x1c0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351863]  [<ffffffff811312b5>] alloc_pages_current+0xa5/0x120

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351871]  [<ffffffff810f08f2>] filemap_fault+0x172/0x3d0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351878]  [<ffffffff8111916a>] __do_fault+0x7a/0x5d0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351884]  [<ffffffff8111975d>] handle_pte_fault+0x9d/0x230

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351893]  [<ffffffff81445327>] do_page_fault+0x207/0x4c0

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.351900]  [<ffffffff81442065>] page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.353040] DWARF2 unwinder stuck at page_fault+0x25/0x30

Oct 25 13:45:45 TSPTONLDB01 kernel: [14317788.353042]

 

 

通过日志发现发现在10月24日到11月份出现多次内存不足，操作系统 KILL 掉MYSQLD进程的情况。

 

在该服务器上有3个MYSQL实例，每个MYSQL实例的 INNODB 缓冲池设置为2GB，但操作系统的物理内存

只有4GB，另外在服务器上还有一个数据同步的 JAVA进程。结合日志分析，MYSQL 内存分配不合理，建议把服务器内存扩展到10GB。

 

建议MYSQL INNODB池计算方法： INNODB 缓冲池内存= 操作系统物理内存 - （预留给操作系统的内存1-2GB）

- （JAVA进程的内存）-（MYSQL最大连接数*2MB）