Troubleshooting HMON

Note: HMON commands provide output specific to the device on which the command is executed.
Note: Refer to the Ruckus FastIron Command Reference Guide for more information on the commands.

When an HMON client is marked as faulty, a syslog message similar to the following is issued: 

Application webserver failed recovery and functionality provided by it may not be available until FastIron acts on it

Work with RUCKUS technical support if this type of failure occurs. Before contacting technical support, perform the following steps to gather diagnostic information.

  1. In Privileged EXEC mode, enter the show hmon status command and check the client list to confirm that the failed process is indeed being monitored by HMON. Capture the output. 
    device# show hmon status 
----------------------
Health Monitor Status: 
----------------------
Hmon's Stack Role is : Standalone
Number of Clients    : 18

Client Names (ID) : 
        nginx (4)
        uwsgi (5)
        PySzAgtSrv.py (6)
        dhcpd (3)
        dhcpdv4 (20)
        snmpagentd (7)
        logmgr.py (13)
        poed (8)
        netconfd_svc (9)
        fcgiwrap (11)
        netconfd-pro (10)
        plugin_proc.py (15)
        cfg_sync (12)
        sshd (16)
        fi_syslog (17)
        slam_server.py (18)
        PyWebFlaskAgent (19)
        socat (28)
    The previous example of the show hmon status command displays information for eighteen registered HMON processes identified by their Client name and ID. The output also indicates that the device is not part of a stack.
  2. Enter the show hmon client status all-clients command to check the administrative and operational state of the application. Capture the output. 
    device# show hmon client status all-clients
-----------------------------
Health Monitor Client Status: 
-----------------------------

Status for client ID 4:
Process Name              : nginx
Valid                     : Yes
Admin. State              : Enabled, Started, HA Enabled
Oper. State               : Up

Status for client ID 5:
Process Name              : uwsgi
Valid                     : Yes
Admin. State              : Disabled, HA Disabled
Oper. State               : Down

Status for client ID 6:
Process Name              : PySzAgtSrv.py
Valid                     : Yes
Admin. State              : Disabled, HA Disabled
Oper. State               : Down

Status for client ID 3:
Process Name              : dhcpd
Valid                     : Yes
Admin. State              : Disabled, HA Disabled                 
Oper. State               : Down
    The previous example of the show hmon client status all-clients shows that the dhcpd process is disabled.
  3. Enter the show log command and check for hmond entries similar to those in the following example. 
    Dynamic Log Buffer (4000 lines):
Mar  4 22:22:19:E:hmond[392]: Client uwsgi-2.7 has reached/exceeded max funcmntr fail count: 2, initiating recovery from state UtilReportedFail  
Mar  4 22:22:19:E:hmond[392]: Client uwsgi-2.7 is not functional, fail count is: 2, funcmntr fail count limit is: 2  
Mar  4 22:22:09:E:hmond[392]: Client uwsgi-2.7 is not functional, fail count is: 1, funcmntr fail count limit is: 2
  4. Enter the show hmon client statistics all-clients command and capture the output. 
    device# show hmon client statistics all-clients 
---------------------------------
Health Monitor Client Statistics: 
---------------------------------

Statistics for client ID 4:
Process Name                             : nginx
Most recent PID                          : 1328
Func. Monitor fail counts                : 0
Total number of admin stops              : 0
Total number of disallowed admin stops   : 0
Total number of admin starts             : 1
Total number of disallowed admin starts  : 1
Total number of admin restarts           : 0
Total number of restarts for recovery    : 0
Code from latest func. fail recovery     : 0x0
Status from latest Func. Monitor check   : Invalid

Statistics for client ID 5:
Process Name                             : uwsgi-2.7
Most recent PID                          : 1343
Func. Monitor fail counts                : 0
Total number of admin stops              : 0
Total number of disallowed admin stops   : 0
Total number of admin starts             : 1
Total number of disallowed admin starts  : 1
Total number of admin restarts           : 0
Total number of restarts for recovery    : 0
Code from latest func. fail recovery     : 0x0
Status from latest Func. Monitor check   : Invalid

Statistics for client ID 6:
Process Name                             : PySzAgtSrv.py
Most recent PID                          : 1356
Func. Monitor fail counts                : 0
Total number of admin stops              : 0
Total number of disallowed admin stops   : 0
Total number of admin starts             : 1
Total number of disallowed admin starts  : 1
Total number of admin restarts           : 0
Total number of restarts for recovery    : 0
Code from latest func. fail recovery     : 0x0
Status from latest Func. Monitor check   : Access Issue

Statistics for client ID 3:
Process Name                             : dhcpd
Most recent PID                          : Not Available
Func. Monitor fail counts                : 0
Total number of admin stops              : 0
Total number of disallowed admin stops   : 0
Total number of admin starts             : 0
Total number of disallowed admin starts  : 0
Total number of admin restarts           : 0
Total number of restarts for recovery    : 0
Code from latest func. fail recovery     : 0x0
Status from latest Func. Monitor check   : Not Invoked
  5. Enter the show hmon client configuration all-clients command and capture the output. 
    device# show hmon client configuration all-clients 
------------------------------------
Health Monitor Client Configuration: 
------------------------------------

Configuration attributes for client ID 4:
Process Name                  : nginx
Startup Script                : nginx-service.sh
Stackrole mask                : 0x3
Starts on bootup              : No
Process restartable           : Yes
Criticality of the process    : Non-Critical
Process restart count limit   : 5
Heart-Beat monitoring reqd.   : No
Functionality monitoring reqd.: Yes
Func. monitoring interval     : 10 Secs
Func. fail count limit        : 2

Configuration attributes for client ID 5:
Process Name                  : uwsgi-2.7
Startup Script                : uwsgi-service.sh
Stackrole mask                : 0x3
Starts on bootup              : No
Process restartable           : Yes
Criticality of the process    : Non-Critical
Process restart count limit   : 5
Heart-Beat monitoring reqd.   : No
Functionality monitoring reqd.: Yes
Func. monitoring interval     : 10 Secs
Func. fail count limit        : 2

Configuration attributes for client ID 6:
Process Name                  : PySzAgtSrv.py
Startup Script                : pySzagent-service.sh
Stackrole mask                : 0x3
Starts on bootup              : No
Process restartable           : Yes
Criticality of the process    : Non-Critical
Process restart count limit   : 5
Heart-Beat monitoring reqd.   : No
Functionality monitoring reqd.: Yes
Func. monitoring interval     : 10 Secs
Func. fail count limit        : 2

Configuration attributes for client ID 3:
Process Name                  : dhcpd
Startup Script                : dhcpd-script.sh
Stackrole mask                : 0x3
Starts on bootup              : No
Process restartable           : Yes
Criticality of the process    : Non-Critical
Process restart count limit   : 5
Heart-Beat monitoring reqd.   : No
Functionality monitoring reqd.: Yes
Func. monitoring interval     : 10 Secs
Func. fail count limit        : 2
    The previous example of the show hmon client configuration all-clients command shows that all HMON clients running on the device are non-critical; that is, the processes become unavailable when marked faulty, and no unit reboot is attempted to initiate switchover to the standby controller.
  6. Note: Beginning with FastIron release 10.0.20, supportsave supports IPv6 also.
    Enter the supportsave all followed by the IP address of the TFTP server where supportsave logs are to be uploaded as shown in the following example. The supportsave collects logs from different modules and uploads the logs into a remote SCP or TFTP server. For more supportsave command options, refer to the RUCKUS FastIron Command Reference.
    Note: In FastIron release 10.0.10d, the maximum number of core files has been limited to 2 for all the ICX platforms.
    The following example sends all information to the default IPv4 remote TFTP server.
    device# supportsave all tftp 10.176.198.2 / CustTagNam
Gathering information. SupportSave has started. 
It will take sometime for completion, depending on the size of the SupportSave file.

Uploading file: 20241028_163534-fm-tmp-2.tgz
Parameter Validation Successful
File Upload Started
Time taken to upload file: 22 seconds
File Upload Complete
 
Uploading file: 20241028_163534-fm-tmp-1.tgz
Parameter Validation Successful
File Upload Started
Time taken to upload file: 12 seconds
File Upload Complete
 
Uploading file: supportsave_ramacOct28_609c9f523a26_20241028_163534.tgz
Parameter Validation Successful
File Upload Started
Time taken to upload file: 3 seconds
File Upload Complete

Uploading file: 20241028_163534-fm-tmp-3.tgz
Parameter Validation Successful
File Upload Started
Time taken to upload file: 45 seconds
File Upload Complete
 
Supportsave took 158 seconds
    The following example sends all information to the default IPv6 remote SCP server.
    device# supportsave all scp ipv6 2001:df3:80d:8221::147 /

Gathering information....SupportSave has started. It will take sometime for completion, depending on the size of the SupportSave file.
Login:
SYSLOG: <14> Nov 27 06:41:37 ICX8200 Router SupportSave collection started  
fiuser
Password:
Uploading file: 20241127_064158-fm-tmp-1.tgz
Parameter Validation Successful
File Upload Started
SYSLOG: <14> Nov 27 06:43:59 ICX8200-48PF Router Download: FILE UPLOAD START  
SYSLOG: <14> Nov 27 06:44:42 ICX8200-48PF Router Download: FILE UPLOAD COMPLETED  
Time taken to upload file: 43 seconds
File Upload Complete
Uploading file: 20241127_064158-fm-tmp-3.tgz
Parameter Validation Successful
File Upload Started
SYSLOG: <14> Nov 27 06:44:42 ICX8200-48PF Router Download: FILE UPLOAD START  
SYSLOG: <14> Nov 27 06:45:49 ICX8200-48PF Router Download: FILE UPLOAD COMPLETED  
Time taken to upload file: 67 seconds
File Upload Complete
Uploading file: supportsave_c0c5207e9c66_20241127_064158.tgz
Parameter Validation Successful
File Upload Started
SYSLOG: <14> Nov 27 06:45:49 ICX8200-48PF Router Download: FILE UPLOAD START  
SYSLOG: <14> Nov 27 06:45:52 ICX8200-48PF Router Download: FILE UPLOAD COMPLETED  
Time taken to upload file: 3 seconds
File Upload Complete
Uploading file: 20241127_064158-fm-tmp-2.tgz
Parameter Validation Successful
File Upload Started
SYSLOG: <14> Nov 27 06:45:53 ICX8200-48PF Router Download: FILE UPLOAD START   
SYSLOG: <14> Nov 27 06:47:02 ICX8200-48PF Router Download: FILE UPLOAD COMPLETED  
Time taken to upload file: 69 seconds
File Upload Complete
Supportsave took 326 seconds
Note: If supportsave was cancelled through Ctrl-C, please run 'supportsave cancel' command before collecting Supportsave again.
    Note: The supportsave command supports Management VRF from FastIron 10.0.10f.
  7. Contact RUCKUS technical support.
Parent topic: HMON - Health Monitor Service