Couchbase memory issue

Hi All,

I have setup single node cluster on my RedHat 7 machine. after some days my Couchbase service is not able to start.

$ systemctl status couchbase-server.service
â couchbase-server.service - Couchbase Server
   Loaded: loaded (/usr/lib/systemd/system/couchbase-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2022-02-18 03:14:19 EST; 56min ago
     Docs: https://docs.couchbase.com
 Main PID: 2356 (code=exited, status=1/FAILURE)

So i check what is a problem to start my Couchbase so I found below,

journalctl -q _PID=2356

Jan 29 12:20:01 server CROND[24807]: (root) CMD (/usr/openv/pdde/vpfs/bin/vpfs_monitor_s.sh > /dev/null 2>&1)

Feb 01 14:18:01 server CROND[24807]: (root) CMD (/opt/SAtools/process-stats/bin/process-stats > /dev/null 2>&1)

Feb 18 02:04:51 server couchbase[24807]: {"init terminating in do_boot",{{badmatch,{error,{bad_return,{{ns_babysitter,start,[normal,[]]},{'EXIT',{{badmatch,{error,enospc}},[{ns_babysitter,do_write_file,3,[{file,"src/ns_babysitter.erl"},{line,88}]},{ns_babysitter,start,2,[{file,"src/ns_babysitter.erl"},{line,57}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,277}]}]}}}}}},[{ns_babysitter_bootstrap,start,0,[{file,"src/ns_babysitter_bootstrap.erl"},{line,23}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}

Feb 18 02:04:51 server couchbase[24807]: init terminating in do_boot ({{badmatch,{error,{bad_return,{_}}}},[{ns_babysitter_bootstrap,start,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Feb 18 02:04:52 server couchbase[24807]: Crash dump is being written to: erl_crash.dump.1645167889.24807.babysitter...done

Couchbase /opt/couchbase/var/lib/couchbase/logs/error.log file showing below logs,

[stats:error,2022-02-17T04:24:34.675-05:00,ns_1@127.0.0.1:<0.27687.1376>:stats_reader:log_bad_responses:208]Some nodes didn't respond: ['ns_1@127.0.0.1']
[stats:error,2022-02-17T04:24:34.675-05:00,ns_1@127.0.0.1:<0.24284.1376>:stats_reader:log_bad_responses:208]Some nodes didn't respond: ['ns_1@127.0.0.1']
[stats:error,2022-02-17T04:24:34.953-05:00,ns_1@127.0.0.1:<0.25736.1376>:stats_reader:log_bad_responses:208]Some nodes didn't respond: ['ns_1@127.0.0.1']
[stats:error,2022-02-17T04:24:34.953-05:00,ns_1@127.0.0.1:<0.25335.1376>:stats_reader:log_bad_responses:208]Some nodes didn't respond: ['ns_1@127.0.0.1']
[stats:error,2022-02-17T04:24:34.954-05:00,ns_1@127.0.0.1:<0.22320.1376>:stats_reader:log_bad_responses:208]Some nodes didn't respond: ['ns_1@127.0.0.1']
[user:error,2022-02-17T15:21:15.223-05:00,ns_1@127.0.0.1:<0.1920.1458>:compaction_daemon:ensure_can_db_compact:1028]Cannot compact database `EVENTING_DATA/1`: the estimated necessary disk space is about 7008 bytes but the currently available disk space is 0 bytes.
[user:error,2022-02-17T15:21:15.226-05:00,ns_1@127.0.0.1:<0.495.1458>:compaction_daemon:ensure_can_db_compact:1028]Cannot compact database `EVENTING_DATA/3`: the estimated necessary disk space is about 5220 bytes but the currently available disk space is 0 bytes.
[user:error,2022-02-17T15:21:15.230-05:00,ns_1@127.0.0.1:<0.343.1458>:compaction_daemon:ensure_can_db_compact:1028]Cannot compact database `EVENTING_DATA/7`: the estimated necessary disk space is about 7026 bytes but the currently available disk space is 0 bytes.
[user:error,2022-02-17T15:21:15.233-05:00,ns_1@127.0.0.1:<0.1377.1458>:compaction_daemon:ensure_can_db_compact:1028]Cannot compact database `EVENTING_DATA/8`: the estimated necessary disk space is about 7016 bytes but the currently available disk space is 0 bytes.

As per my understanding my Couchbase node not responding because of memory issue ?

Please help me … I am also not able to start my CB service.
How to change memory setting without starting ?

1 Like

Might this not be at least in part caused by your disk being full? (The compaction messages indicate you have no free space on at least one filesystem.) I’d suggest starting there as it would be the simplest thing to address.

(Check df -h for filesystem usage information, and du -sh <path>/* to summarise usage under a given path.)

HTH.

But question is how i can start … Couchbase service … as its giving issue … as mention above error …
from filesystem can i able to modify memory details … ?

I think it won’t start because your disk is full. Free up some space on the filesystem (or enlarge it if you can) then try restart it. To be clear, you do not need to change memory settings - you need to free up space on the disk.

Specifically I note: “but the currently available disk space is 0 bytes” in your errors. This is not memory but disk. I also note that the start-up failure was in “do_write_file” which implies writing to disk - and this is failing because there is no free space on disk (error “enospc”).

I suggested you run “df -h” (etc.) from a terminal window so you can see what space exists on your system as this may help you clean-up.

Memory available …

$ cat /proc/meminfo
MemTotal: 196974444 kB
MemFree: 168086472 kB
MemAvailable: 173936292 kB
Buffers: 269944 kB
Cached: 6980284 kB
SwapCached: 0 kB
Active: 22634648 kB
Inactive: 2992604 kB
Active(anon): 19014888 kB
Inactive(anon): 324800 kB
Active(file): 3619760 kB
Inactive(file): 2667804 kB
Unevictable: 25748 kB
Mlocked: 25748 kB
SwapTotal: 33554428 kB
SwapFree: 33554428 kB
Dirty: 4092 kB
Writeback: 0 kB
AnonPages: 18402852 kB
Mapped: 431172 kB
Shmem: 953476 kB
Slab: 599916 kB
SReclaimable: 313276 kB
SUnreclaim: 286640 kB
KernelStack: 71216 kB
PageTables: 114864 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 132041648 kB
Committed_AS: 23866432 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 724396 kB
VmallocChunk: 34256678772 kB
Percpu: 18176 kB
HardwareCorrupted: 0 kB
AnonHugePages: 22528 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 240440 kB
DirectMap2M: 9914368 kB
DirectMap1G: 190840832 kB

It is your disk not memory that you need to look at.

2 Likes

ohh i got your point …
checking

I have increase the CPU and memory. And freshly install couchbase. Now, the issue is not there. I am not sure whether this is the right method.