Vmware Server 2 Web Access Connection Loss (vmware-hostd crash) Workarounds
Summery of Issue
With upgrading to RHEL 5.4, CentOS 5.4 and Ubuntu 9.10, the latest 2.x.x versions of VMware Server are having serious Web Access GUI connection failures, specifically vmware-hostd crashing repeatedly. This has been found with VMware Server 2.0.0, Vmware Server 2.0.1 and VMware Server 2.0.2. VMware Server 2.x.x was stable in the previous revisions of the mentioned OS’s. Below are two solutions that “appear” to make for a stable vmware-hostd process. You are advised strongly to satisfy your own assuredness of the stability of vmware-hostd using these solutions before deployment to a mission critical environment.
Both solutions do not require you to stop all vmware related processes on the host server. The following steps assume vmware-hostd has crashed and left VMware clients still running.
Verify vmware-host Process has Failed
Note: If you get the below from the ps command you have another issue and this document is not for you.
root 10858 1 0 16:47 ? 00:00:02 /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml
root 11055 11026 0 17:02 pts/3 00:00:00 grep vmware-hostd
Regaining VMware Server 2 Web Access GUI Control
If you want to start the vmware-hostd process to manage your VMware Server 2 guest operating systems again you may do so with the following commands.
# /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml &
[1] 11139
# <hit return/enter>
[1]+ Done /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml
# ps -ef | grep hostd
root 11140 1 22 17:13 ? 00:00:01 /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml
root 11155 11026 0 17:13 pts/3 00:00:00 grep hostd&
nohup is not needed in this instance as vmware-hostd runs as a daemon but the ampersand “&” is. Otherwise you’ll get logged output to the screen and when you exit your session vmware-hostd will stop too.
Solving the VMware Server 2 Web Access GUI Connection Failure
I recommend looking at both solutions. I’m currently employing solution #2 but I’ll leave that decision up to you. Both allow you to use the start/stop /etc/init.d/vmware script as you normally would and are permanent unlike the quick fix above to get the vmware-hostd process up and running again. Again with both solutions you need to determine if they, in fact, produce a stable VMware Server 2 environment before deployment to a mission critical environment.
SOLUTION #1 (libc-2.5.so reversion – RHEL 5.4 & CentOS 5.4)
Download and copy libc-2.5.so into place:
# rpm -Uvh –root=/tmp/ –nodeps ./glibc-2.5-34.x86_64.rpm
# mkdir /usr/lib/vmware/lib/libc.so.6
# cp /tmp/lib64/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6
Edit /usr/sbin/vmware-hostd adding the following export command just before the last line in the script as follows:
export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH
eval exec “$DEBUG_CMD” “$binary” “$@”
SOLUTION #2 (Circumventing vmware-hostd library wrapping script – RHEL 5.4, CentOS 5.4 & Ubuntu 9.10 )
Here is another method not requiring reverting to an older version of libc-2.5.so. The downside in this solution is it circumvents the dynamic library path building of the /usr/sbin/vmware-hostd script and executes the /usr/lib/vmware/bin/vmware-hostd binary directly. I do not know if this will present problems in the future or not.
Below is the snippet from the modified /etc/init.d/vmware. You can see I added a LD_LIBRARY_PATH statement, commented out the old exec call and added a new one.
vmware_start_hostd() {
export LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8
vmware_bg_exec “`vmware_product_name` Host Agent” \
“$vmdb_answer_LIBDIR/bin/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
#”$vmdb_answer_SBINDIR/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
}
Restart VMware Server 2
If you don’t have critical guest OS’s running you can stop the guests via the VMware Server 2 Web Access GUI and restart VMware:
Stopping VMware autostart virtual machines:
Virtual machines [ OK ]
Stopping VMware management services:
VMware Virtual Infrastructure Web Access
VMware Server Host Agent [ OK ]
Stopping VMware services:
VMware Authentication Daemon [ OK ]
VM communication interface socket family: [ OK ]
Virtual machine communication interface [ OK ]
Virtual machine monitor [ OK ]
Bridged networking on /dev/vmnet0 [ OK ]
Host network detection [ OK ]
DHCP server on /dev/vmnet1 [ OK ]
Host-only networking on /dev/vmnet1 [ OK ]
DHCP server on /dev/vmnet8 [ OK ]
NAT service on /dev/vmnet8 [ OK ]
Host-only networking on /dev/vmnet8 [ OK ]
Virtual ethernet [ OK ]
Starting VMware services:
Virtual machine monitor [ OK ]
Virtual machine communication interface [ OK ]
VM communication interface socket family: [ OK ]
Virtual ethernet [ OK ]
Bridged networking on /dev/vmnet0 [ OK ]
Host-only networking on /dev/vmnet1 (background) [ OK ]
DHCP server on /dev/vmnet1 [ OK ]
Host-only networking on /dev/vmnet8 (background) [ OK ]
DHCP server on /dev/vmnet8 [ OK ]
NAT service on /dev/vmnet8 [ OK ]
VMware Server Authentication Daemon (background) [ OK ]
Shared Memory Available [ OK ]
Starting VMware management services:
VMware Server Host Agent (background) [ OK ]
VMware Virtual Infrastructure Web Access
Starting VMware autostart virtual machines:
Virtual machines [ OK ]
As more information on this issue becomes available this post will be updated. Please post your findings too.
This information was generated by my experimentation and the helpful posts of the VMware Community, reference: http://communities.vmware.com/thread/229957?tstart=0
Thanks or the great blog post. This issue has been driving me nuts!
I am so happy Google found this post…I had been fighting this issue for 3 days to no avail.
You rock!
Thanks !!! I almost lost my hair because of that stupid issue! That was a really helpful post.
Thanks man. I was ready to move to a different virtualization solution when I couldn’t find the problem with VMWare.
Thank you for info!
Downloaded and used:
yum downgrade glibc-2.5-34.el5_3.1.i686.rpm glibc-common-2.5-34.el5_3.1.i686.rpm glibc-devel-2.5-34.el5_3.1.i686.rpm glibc-headers-2.5-34.el5_3.1.i686.rpm
Then reboot.
Works good!
This was greatly helpful, thank you! I had to do a little search for find a CentOS 5.3 mirror that was still up.
Thank you for this important help.
On a 64 bits RHEL 5.5, I’m testing the #2 solution since hours and no crash since.
I like the second solution best as I don’t have to look for this RPM.
Please note that for the time being, this RPM is quite hard to find on the classical repos…
I tried the second solution, but I’m having problems. Once editing the startup script (/etc/init.d/vmware), I start the service back up and get this:
Starting VMware services:
Virtual machine monitor[ OK ]
Virtual machine communication interface[ OK ]
VM communication interface socket family:[ OK ]
Virtual ethernet[ OK ]
Bridged networking on /dev/vmnet0[ OK ]
Host-only networking on /dev/vmnet1 (background)[ OK ]
DHCP server on /dev/vmnet1[ OK ]
Host-only networking on /dev/vmnet8 (background)[ OK ]
DHCP server on /dev/vmnet8[ OK ]
NAT service on /dev/vmnet8[ OK ]
VMware Server Authentication Daemon (background)[ OK ]
Shared Memory Available[ OK ]
Starting VMware management services:
declare -x G_BROKEN_FILENAMES=”1″
declare -x HISTSIZE=”1000″
declare -x HOME=”/root”
declare -x HOSTNAME=”crono.phys.uh.edu”
declare -x INPUTRC=”/etc/inputrc”
declare -x LANG=”en_US.UTF-8″
declare -x LESSOPEN=”|/usr/bin/lesspipe.sh %s”
declare -x LOGNAME=”root”
declare -x LS_COLORS=”no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:”
declare -x MAIL=”/var/spool/mail/root”
declare -x OLDPWD
declare -x PATH=”/sbin:/usr/sbin:/bin:/usr/bin”
declare -x PWD=”/root”
declare -x SHELL=”/bin/bash”
declare -x SHLVL=”2″
declare -x TERM=”vt100″
declare -x USER=”root”
VMware Server Host Agent (background)[ OK ]
VMware Virtual Infrastructure Web Access
Starting VMware autostart virtual machines:
Virtual machines[ OK ]
Then, when I try to access the Web interface, I’m unable to connect. Any suggestions? Thanks!
P.S. Here’s what I’ve edited:
# Start host agent
vmware_start_hostd() {
export
LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8
vmware_bg_exec “`vmware_product_name` Host Agent” \
“$vmdb_answer_LIBDIR/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
# “$vmdb_answer_SBINDIR/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
}
Hello George!
Thank you for this full-covered solution! I try to use variant #2 on fresh CentOS 5.5 x64 2.6.18-194.8.1.el5, and can’t understand – this fix work, when i manually run
export LD_LIBRARY_PATH and /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml &. But then i put it in /etc/init.d/vmware script , hostd can’t load. I try to debug it with bash -x /etc/init.d/vmware start , and see this:
+ return 0
+ service_vmware_mgmt start
+ case “$1″ in
++ vmware_product
++ echo wgs
++ exit 0
+ ‘[' wgs = wgs ']‘
+ echo ‘Starting VMware management services:’
Starting VMware management services:
+ vmware_start_hostd
+ export LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8
+ LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8
++ vmware_product_name
++ echo ‘VMware Server’
++ exit 0
+ vmware_bg_exec ‘VMware Server Host Agent’ /usr/lib/vmware/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml
+ local ‘msg=VMware Server Host Agent’
+ local func=/usr/lib/vmware/vmware-hostd
+ shift 2
+ ‘[' '' = yes ']‘
+ echo -n ‘ VMware Server Host Agent (background)’
VMware Server Host Agent (background)+ logger -t ‘VMware[init]‘ -p daemon.err
+ vmware_success
++ type -t echo_success
+ ‘[' function = function ']‘
+ echo_success
+ ‘[' color = color ']‘
+ echo -en ’33[60G'
+ echo -n '['
[+ '[' color = color ']‘
+ echo -en ’33[0;32m'
+ echo -n ' OK '
OK + '[' color = color ']‘
+ echo -en ’33[0;39m'
+ echo -n ']‘
]+ echo -ne ‘\r’
+ return 0
+ echo
+ return 0
e.g. all OK. But ps -aux | grep host
Warning: bad syntax, perhaps a bogus ‘-’? See /usr/share/doc/procps-3.2.7/FAQ
root 12371 0.0 0.0 61184 732 pts/2 S+ 12:36 0:00 grep host
– hostd is absent in running processes after vmware start script =\ May be you can help me with any ideas , why hostd run manually, but can’t run while running vmware start script?
Hi Epiq,
If it works manually from the command line but not from the startup script I would suspect a syntax error. If you were to copy the change from this blog for /etc/init.d/vmware and paste into a pUTTY session with vi open, quotes (“) have a tendency to translate into periods (.). Also check the hostd log /var/log/vmware/hostd.log and see if it tells you anything.
I ran bash -x /etc/init.d/vmware start and compared to your output and found the following difference:
Yours:
+ vmware_bg_exec ‘VMware Server Host Agent’ /usr/lib/vmware/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml
Mine:
+ vmware_bg_exec ‘VMware Server Host Agent’ /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml
Unless you have a different directory structure than my Vmware Server you have a bad file path to vmware-hostd.
Let me know how you do.
George
@Pamela Lloyd
Glad you got it working Pamela!
@George Knerr
I’am change
“$vmdb_answer_LIBDIR/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
to
“$vmdb_answer_LIBDIR/bin/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
and this solve my problem – it was my stupid syntax error =\
Thanx a lot for help with it !
This is great to find!
Just converted a server from Ubuntu 10.04 to 64bit RHEL 5.5 and this seems to have solved the problems connecting with the with web client as well as using the VMware Infrastructure Client as well.
Thanks for sharing..
@Pamela Lloyd
Pamela I have the same problem, did you solve it?
Hi George,
solution #1 – functional!!
with a fresh
CentOS 5.5 2.6.18-194.11.4.el5.x86_64 and VMware-server-2.0.2-203138.x86_64
glibc @ http://vault.centos.org/5.3/os/x86_64/CentOS/glibc-2.5-34.x86_64.rpm
Thanks for that idea.