Home > Linux, Vmware > Vmware Server 2 Web Access Connection Loss (vmware-hostd crash) Workarounds

Vmware Server 2 Web Access Connection Loss (vmware-hostd crash) Workarounds

Summery of Issue

With upgrading to RHEL 5.4, CentOS 5.4 and Ubuntu 9.10, the latest 2.x.x versions of VMware Server are having serious Web Access GUI connection failures, specifically vmware-hostd crashing repeatedly. This has been found with VMware Server 2.0.0, Vmware Server 2.0.1 and VMware Server 2.0.2. VMware Server 2.x.x was stable in the previous revisions of the mentioned OS’s. Below are two solutions that “appear” to make for a stable vmware-hostd process. You are advised strongly to satisfy your own assuredness of the stability of vmware-hostd using these solutions before deployment to a mission critical environment.

Both solutions do not require you to stop all vmware related processes on the host server. The following steps assume vmware-hostd has crashed and left VMware clients still running.

Verify vmware-host Process has Failed

Note: If you get the below from the ps command you have another issue and this document is not for you.

# ps -ef |grep vmware-hostd

root 10858 1 0 16:47 ? 00:00:02 /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml

root 11055 11026 0 17:02 pts/3 00:00:00 grep vmware-hostd

Regaining VMware Server 2 Web Access GUI Control

If you want to start the vmware-hostd process to manage your VMware Server 2 guest operating systems again you may do so with the following commands.

# export LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8

# /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml &

[1] 11139

# <hit return/enter>

[1]+ Done /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml

# ps -ef | grep hostd

root 11140 1 22 17:13 ? 00:00:01 /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml

root 11155 11026 0 17:13 pts/3 00:00:00 grep hostd&

nohup is not needed in this instance as vmware-hostd runs as a daemon but the ampersand “&” is. Otherwise you’ll get logged output to the screen and when you exit your session vmware-hostd will stop too.

Solving the VMware Server 2 Web Access GUI Connection Failure

I recommend looking at both solutions. I’m currently employing solution #2 but I’ll leave that decision up to you. Both allow you to use the start/stop /etc/init.d/vmware script as you normally would and are permanent unlike the quick fix above to get the vmware-hostd process up and running again. Again with both solutions you need to determine if they, in fact, produce a stable VMware Server 2 environment before deployment to a mission critical environment.

SOLUTION #1 (libc-2.5.so reversion – RHEL 5.4 & CentOS 5.4)

Download and copy libc-2.5.so into place:

# lynx http://mirror.centos.org/centos/5.3/os/x86_64/CentOS/glibc-2.5-34.x86_64.rpm

# rpm -Uvh –root=/tmp/ –nodeps ./glibc-2.5-34.x86_64.rpm

# mkdir /usr/lib/vmware/lib/libc.so.6

# cp /tmp/lib64/libc-2.5.so /usr/lib/vmware/lib/libc.so.6/libc.so.6

Edit /usr/sbin/vmware-hostd adding the following export command just before the last line in the script as follows:

# tail -3 /usr/sbin/vmware-hostd

export LD_LIBRARY_PATH=/usr/lib/vmware/lib/libc.so.6:$LD_LIBRARY_PATH

eval exec “$DEBUG_CMD” “$binary” “$@”

SOLUTION #2 (Circumventing vmware-hostd library wrapping script – RHEL 5.4, CentOS 5.4 & Ubuntu 9.10 )

Here is another method not requiring reverting to an older version of libc-2.5.so. The downside in this solution is it circumvents the dynamic library path building of the /usr/sbin/vmware-hostd script and executes the /usr/lib/vmware/bin/vmware-hostd binary directly. I do not know if this will present problems in the future or not.

Below is the snippet from the modified /etc/init.d/vmware. You can see I added a LD_LIBRARY_PATH statement, commented out the old exec call and added a new one.

# Start host agent

vmware_start_hostd() {

export LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8

vmware_bg_exec “`vmware_product_name` Host Agent” \

“$vmdb_answer_LIBDIR/bin/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”

#”$vmdb_answer_SBINDIR/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”

}

Restart VMware Server 2

If you don’t have critical guest OS’s running you can stop the guests via the VMware Server 2 Web Access GUI and restart VMware:

# /etc/init.d/vmware restart

Stopping VMware autostart virtual machines:
Virtual machines [ OK ]
Stopping VMware management services:
VMware Virtual Infrastructure Web Access
VMware Server Host Agent [ OK ]
Stopping VMware services:
VMware Authentication Daemon [ OK ]
VM communication interface socket family: [ OK ]
Virtual machine communication interface [ OK ]
Virtual machine monitor [ OK ]
Bridged networking on /dev/vmnet0 [ OK ]
Host network detection [ OK ]
DHCP server on /dev/vmnet1 [ OK ]
Host-only networking on /dev/vmnet1 [ OK ]
DHCP server on /dev/vmnet8 [ OK ]
NAT service on /dev/vmnet8 [ OK ]
Host-only networking on /dev/vmnet8 [ OK ]
Virtual ethernet [ OK ]
Starting VMware services:
Virtual machine monitor [ OK ]
Virtual machine communication interface [ OK ]
VM communication interface socket family: [ OK ]
Virtual ethernet [ OK ]
Bridged networking on /dev/vmnet0 [ OK ]
Host-only networking on /dev/vmnet1 (background) [ OK ]
DHCP server on /dev/vmnet1 [ OK ]
Host-only networking on /dev/vmnet8 (background) [ OK ]
DHCP server on /dev/vmnet8 [ OK ]
NAT service on /dev/vmnet8 [ OK ]
VMware Server Authentication Daemon (background) [ OK ]
Shared Memory Available [ OK ]
Starting VMware management services:
VMware Server Host Agent (background) [ OK ]
VMware Virtual Infrastructure Web Access
Starting VMware autostart virtual machines:
Virtual machines [ OK ]

As more information on this issue becomes available this post will be updated. Please post your findings too.

This information was generated by my experimentation and the helpful posts of the VMware Community, reference: http://communities.vmware.com/thread/229957?tstart=0

Categories: Linux, Vmware Tags:
  1. November 26th, 2009 at 07:31 | #1

    Thanks or the great blog post. This issue has been driving me nuts!

  2. Shawn
    December 9th, 2009 at 22:53 | #2

    I am so happy Google found this post…I had been fighting this issue for 3 days to no avail.

    You rock!

  3. December 28th, 2009 at 17:58 | #3

    Thanks !!! I almost lost my hair because of that stupid issue! That was a really helpful post.

  4. Jouni Renfors
    January 7th, 2010 at 06:35 | #4

    Thanks man. I was ready to move to a different virtualization solution when I couldn’t find the problem with VMWare. :)

  5. Pander
    January 28th, 2010 at 03:29 | #5

    Thank you for info!

    Downloaded and used:

    yum downgrade glibc-2.5-34.el5_3.1.i686.rpm glibc-common-2.5-34.el5_3.1.i686.rpm glibc-devel-2.5-34.el5_3.1.i686.rpm glibc-headers-2.5-34.el5_3.1.i686.rpm

    Then reboot.

    Works good!

  6. April 14th, 2010 at 16:24 | #6

    This was greatly helpful, thank you! I had to do a little search for find a CentOS 5.3 mirror that was still up.

  7. Nicolas Ecarnot
    May 7th, 2010 at 04:16 | #7

    Thank you for this important help.

    On a 64 bits RHEL 5.5, I’m testing the #2 solution since hours and no crash since.

    I like the second solution best as I don’t have to look for this RPM.
    Please note that for the time being, this RPM is quite hard to find on the classical repos…

  8. Pamela Lloyd
    June 16th, 2010 at 07:57 | #8

    I tried the second solution, but I’m having problems. Once editing the startup script (/etc/init.d/vmware), I start the service back up and get this:

    Starting VMware services:
    Virtual machine monitor[ OK ]
    Virtual machine communication interface[ OK ]
    VM communication interface socket family:[ OK ]
    Virtual ethernet[ OK ]
    Bridged networking on /dev/vmnet0[ OK ]
    Host-only networking on /dev/vmnet1 (background)[ OK ]
    DHCP server on /dev/vmnet1[ OK ]
    Host-only networking on /dev/vmnet8 (background)[ OK ]
    DHCP server on /dev/vmnet8[ OK ]
    NAT service on /dev/vmnet8[ OK ]
    VMware Server Authentication Daemon (background)[ OK ]
    Shared Memory Available[ OK ]
    Starting VMware management services:
    declare -x G_BROKEN_FILENAMES=”1″
    declare -x HISTSIZE=”1000″
    declare -x HOME=”/root”
    declare -x HOSTNAME=”crono.phys.uh.edu”
    declare -x INPUTRC=”/etc/inputrc”
    declare -x LANG=”en_US.UTF-8″
    declare -x LESSOPEN=”|/usr/bin/lesspipe.sh %s”
    declare -x LOGNAME=”root”
    declare -x LS_COLORS=”no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:”
    declare -x MAIL=”/var/spool/mail/root”
    declare -x OLDPWD
    declare -x PATH=”/sbin:/usr/sbin:/bin:/usr/bin”
    declare -x PWD=”/root”
    declare -x SHELL=”/bin/bash”
    declare -x SHLVL=”2″
    declare -x TERM=”vt100″
    declare -x USER=”root”
    VMware Server Host Agent (background)[ OK ]
    VMware Virtual Infrastructure Web Access
    Starting VMware autostart virtual machines:
    Virtual machines[ OK ]

    Then, when I try to access the Web interface, I’m unable to connect. Any suggestions? Thanks!

    P.S. Here’s what I’ve edited:

    # Start host agent
    vmware_start_hostd() {
    export
    LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8
    vmware_bg_exec “`vmware_product_name` Host Agent” \
    “$vmdb_answer_LIBDIR/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
    # “$vmdb_answer_SBINDIR/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
    }

  9. Epiq
    July 23rd, 2010 at 01:45 | #9

    Hello George!

    Thank you for this full-covered solution! I try to use variant #2 on fresh CentOS 5.5 x64 2.6.18-194.8.1.el5, and can’t understand – this fix work, when i manually run
    export LD_LIBRARY_PATH and /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml &. But then i put it in /etc/init.d/vmware script , hostd can’t load. I try to debug it with bash -x /etc/init.d/vmware start , and see this:

    + return 0
    + service_vmware_mgmt start
    + case “$1″ in
    ++ vmware_product
    ++ echo wgs
    ++ exit 0
    + ‘[' wgs = wgs ']‘
    + echo ‘Starting VMware management services:’
    Starting VMware management services:
    + vmware_start_hostd
    + export LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8
    + LD_LIBRARY_PATH=/usr/lib/vmware/vmacore:/usr/lib/vmware/hostd:/usr/lib/vmware/lib/libxml2.so.2:/usr/lib/vmware/lib/libexpat.so.0:/usr/lib/vmware/lib/libstdc++.so.6:/usr/lib/vmware/lib/libgcc_s.so.1:/usr/lib/vmware/lib/libcrypto.so.0.9.8:/usr/lib/vmware/lib/libssl.so.0.9.8
    ++ vmware_product_name
    ++ echo ‘VMware Server’
    ++ exit 0
    + vmware_bg_exec ‘VMware Server Host Agent’ /usr/lib/vmware/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml
    + local ‘msg=VMware Server Host Agent’
    + local func=/usr/lib/vmware/vmware-hostd
    + shift 2
    + ‘[' '' = yes ']‘
    + echo -n ‘ VMware Server Host Agent (background)’
    VMware Server Host Agent (background)+ logger -t ‘VMware[init]‘ -p daemon.err
    + vmware_success
    ++ type -t echo_success
    + ‘[' function = function ']‘
    + echo_success
    + ‘[' color = color ']‘
    + echo -en ’33[60G'
    + echo -n '['
    [+ '[' color = color ']‘
    + echo -en ’33[0;32m'
    + echo -n ' OK '
    OK + '[' color = color ']‘
    + echo -en ’33[0;39m'
    + echo -n ']‘
    ]+ echo -ne ‘\r’
    + return 0
    + echo

    + return 0

    e.g. all OK. But ps -aux | grep host
    Warning: bad syntax, perhaps a bogus ‘-’? See /usr/share/doc/procps-3.2.7/FAQ
    root 12371 0.0 0.0 61184 732 pts/2 S+ 12:36 0:00 grep host
    – hostd is absent in running processes after vmware start script =\ May be you can help me with any ideas , why hostd run manually, but can’t run while running vmware start script?

  10. George Knerr
    July 23rd, 2010 at 08:12 | #10

    Hi Epiq,

    If it works manually from the command line but not from the startup script I would suspect a syntax error. If you were to copy the change from this blog for /etc/init.d/vmware and paste into a pUTTY session with vi open, quotes (“) have a tendency to translate into periods (.). Also check the hostd log /var/log/vmware/hostd.log and see if it tells you anything.

    I ran bash -x /etc/init.d/vmware start and compared to your output and found the following difference:

    Yours:
    + vmware_bg_exec ‘VMware Server Host Agent’ /usr/lib/vmware/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml

    Mine:
    + vmware_bg_exec ‘VMware Server Host Agent’ /usr/lib/vmware/bin/vmware-hostd -a -d -u /etc/vmware/hostd/config.xml

    Unless you have a different directory structure than my Vmware Server you have a bad file path to vmware-hostd.

    Let me know how you do.

    George

  11. George Knerr
    July 23rd, 2010 at 08:16 | #11

    @Pamela Lloyd
    Glad you got it working Pamela!

  12. Epiq
    July 25th, 2010 at 09:49 | #12

    @George Knerr
    I’am change
    “$vmdb_answer_LIBDIR/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
    to
    “$vmdb_answer_LIBDIR/bin/vmware-hostd” -a -d -u “$vmware_etc_dir/hostd/config.xml”
    and this solve my problem – it was my stupid syntax error =\
    Thanx a lot for help with it !

  13. Jeff Musick
    July 30th, 2010 at 20:25 | #13

    This is great to find!

    Just converted a server from Ubuntu 10.04 to 64bit RHEL 5.5 and this seems to have solved the problems connecting with the with web client as well as using the VMware Infrastructure Client as well.

    Thanks for sharing..

  14. Carlo
    September 24th, 2010 at 08:36 | #14

    @Pamela Lloyd

    Pamela I have the same problem, did you solve it?

  15. Daniel Baerthel
    September 24th, 2010 at 13:17 | #15

    Hi George,

    solution #1 – functional!!

    with a fresh
    CentOS 5.5 2.6.18-194.11.4.el5.x86_64 and VMware-server-2.0.2-203138.x86_64
    glibc @ http://vault.centos.org/5.3/os/x86_64/CentOS/glibc-2.5-34.x86_64.rpm

    Thanks for that idea.

  16. sapan
    September 28th, 2010 at 02:06 | #16

    I had the Web Access Problem on Centos 5.5 64-bit with vmware server 2.0.2, and I used the solution #2 (changinng the /etc/init.d/vmware script) to fix it.
    Thanks a lot.

  17. stefan
    November 4th, 2010 at 12:55 | #17

    when i restart of the vmware server i get only “ok”s so it looks like there’s no problem, but although i can’t access the webinterface…

    i used the 2nd way!

    any good ideas ;) ??

  1. May 17th, 2010 at 04:47 | #1
  2. June 15th, 2010 at 05:33 | #2
  3. July 29th, 2010 at 23:26 | #3
  4. February 9th, 2012 at 01:58 | #4