Thursday, June 21, 2012

Troubleshooting slapd error: too many open files

Debian testing comes with openldap (slapd package) version 2.4.28. I noticed that ldap clients start receiving error and cannot contact ldap service any more. While the slapd daemon was running I found a number of errors in syslog (file /var/log/syslog):
... ldap1 slapd[4894]: SASL [conn=1611] Failure: GSSAPI 
  Error: Unspecified GSS failure.  Minor code may 
  provide more information (Too many open files)
... ldap1 slapd[4894]: warning: cannot open 
  /etc/hosts.allow: Too many open files
... ldap1 slapd[4894]: warning: cannot open 
  /etc/hosts.deny: Too many open files
... ldap1 slapd[4894]: SASL [conn=1611] Failure: GSSAPI
  Error: Unspecified GSS failure.  Minor code may
  provide more information (Too many open files)
Take a look at slapd process max open files soft limit:
cat /proc/`pidof slapd`/limits
Check number of files open by slapd process:
pidof slapd | xargs lsof -a -p | wc -l
The slapd process must not exceed max open files soft limit. Take a look at files opened by slapd process:
pidof slapd | xargs lsof -a -p | tail
Here is a sample output that shows there are a number of deleted files... actually one file /var/tmp/ldap_103.
slapd   4894 openldap  117u   REG  0,197 3371 705975 
  /var/tmp/ldap_103 (deleted)
slapd   4894 openldap  118u   REG  0,197 3371 705975 
  /var/tmp/ldap_103 (deleted)
slapd   4894 openldap  119u   REG  0,197 3371 705975 
  /var/tmp/ldap_103 (deleted)
There is definitely a bug in slapd that cause max open files limit exceed.

Recycle Process

We can write a script that does a check and restart daemon if it reaches certain limit (file /usr/local/sbin/slapd-restart).
#!/bin/sh

# Restart slapd daemon if it has open more than 512 files
if [ `pidof slapd | xargs lsof -a -p | wc -l` -gt 512 ]
then
    /etc/init.d/slapd restart
fi
Let cron run this script hourly:
ln -s /usr/local/sbin/slapd-restart \
  /etc/cron.hourly/slapd-restart
If slapd process exceed max open files soft limit too quickly consider schedule cron job more frequently.

Mount /var/tmp in tmpfs

While slapd process creates small files in /var/tmp quite quickly I found reasonable to mount it with tmpfs (file /etc/fstab):
tmpfs /var/tmp tmpfs noatime,nodev,noexec,nosuid,size=1M 0 0
Restart slapd daemon so changes take place:
/etc/init.d/slapd stop
# Ensure the /var/tmp is empty
rm /var/tmp/ldap_103
mount /var/tmp
/etc/init.d/slapd start
# Ensure the /var/tmp mounted
df -h | grep /var/tmp
Ensure the /var/tmp mounted:
tmpfs           1.0M  4.0K 1020K   1% /var/tmp
Regularly take a look a /var/log/syslog if there are any errors reported.