CISCO Troubleshooting Guide for Unified Communications Manager Release 12.5(1) User Guide

Troubleshooting Guide for Unified Communications Manager Release 12.5(1)

Product Information

The Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) provides detailed information on troubleshooting and resolving issues related to the Cisco Unified Communications Manager.

Published Date: 2017-12-07
Last Modified: 2023-11-24

Americas Headquarters
Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706
USA
Website: www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883

Specifications

• Product: Cisco Unified Communications Manager
• Release: 12.5(1)
• Published Date: 2017-12-07
• Last Modified: 2023-11-24

Product Usage Instructions

Chapter 1: Preface
The preface section provides an overview of the purpose, audience, organization, related documentation, conventions, and obtaining support and security guidelines for the Troubleshooting Guide.

Chapter 2: Troubleshooting Overview
In this chapter, you will find information on Cisco Unified Serviceability, Cisco Unified Communications Operating System Administration, general model of problem solving, network failure preparation, and where to find more information.

Cisco Unified Serviceability

This section explains the features and functionalities of Cisco Unified Serviceability, which is a web-based application used for managing and troubleshooting Cisco Unified Communications Manager.

Cisco Unified Communications Operating System Administration
This section provides information on Cisco Unified Communications Operating System Administration, which allows administrators to configure and manage the underlying operating system of Cisco Unified Communications Manager.

General Model of Problem Solving
This section outlines a general model of problem solving that can be followed when troubleshooting issues in Cisco Unified Communications Manager. It covers the steps involved in problem identification, analysis, resolution, and verification.

Network Failure Preparation

This section discusses the importance of network failure preparation and provides guidance on how to prepare for network failures to ensure uninterrupted communication services.

Where to Find More Information
This section provides references to additional sources of information, such as Cisco documentation, community forums, and support resources, where users can find more detailed information on troubleshooting specific issues.

FAQ (Frequently Asked Questions)

Q: Where can I find the latest version of the Troubleshooting Guide?
A: The latest version of the Troubleshooting Guide can be found on the Cisco website at www.cisco.com. Please refer to the website for the most up-to-date information.

Q: How can I obtain support for Cisco Unified Communications Manager?
A: To obtain support for Cisco Unified Communications Manager, you can contact your Cisco representative or visit the Cisco website at www.cisco.com. The website provides information on support options and resources available for troubleshooting and resolving issues.

Q: Are there any known limitations or known issues with Cisco Unified Communications Manager?
A: For information on known limitations or issues with Cisco Unified Communications Manager, please refer to the documentation provided with the product or visit the Cisco website for the latest release notes and bug fixes.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1)
First Published: 2017-12-07 Last Modified: 2023-11-24
Americas Headquarters
Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000
800 553-NETS (6387) Fax: 408 527-0883

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE ABOVE- NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.
All printed copies and duplicate soft copies of this document are considered uncontrolled. See the current online version for the latest version.
Cisco has more than 200 offices worldwide. Addresses and phone numbers are listed on the Cisco website at www.cisco.com/go/offices.
Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: https://www.cisco.com/c/en/us/about/legal/trademarks.html. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R)
© 2017 Cisco Systems, Inc. All rights reserved.

CONTENTS

PREFACE CHAPTER 1 CHAPTER 2

Preface xiii Purpose xiii Audience xiii Organization xiv Related Documentation xv Conventions xv Obtaining Documentation, Obtaining Support, and Security Guidelines xvi Cisco Product Security Overview xvi
Troubleshooting Overview 1 Cisco Unified Serviceability 1 Cisco Unified Communications Operating System Administration 2 General Model of Problem Solving 2 Network Failure Preparation 3 Where to Find More Information 3
Troubleshooting Tools 5 Cisco Unified Serviceability Troubleshooting Tools 5 Command Line Interface 6 kerneldump Utility 7 Enable the Kerneldump Utility 8 Enable Email Alert for Core Dump 8 Network Management 9 System Log Management 9 Cisco Discovery Protocol Support 9 Simple Network Management Protocol Support 10

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) iii

Contents

CHAPTER 3

Sniffer Traces 10 Debugs 10 Cisco Secure Telnet 11 Packet Capture 11
Packet Capturing Overview 11 Configuration Checklist for Packet Capturing 12 Adding an End User to the Standard Packet Sniffer Access Control Group 13 Configuring Packet-Capturing Service Parameters 13 Configuring Packet Capturing in the Phone Configuration Window 14 Configuring Packet Capturing in Gateway and Trunk Configuration Windows 14 Packet-Capturing Configuration Settings 16 Analyzing Captured Packets 17 Common Troubleshooting Tasks, Tools, and Commands 17 Troubleshooting Tips 20 System History Log 21 System History Log Overview 21 System History Log Fields 22 Accessing the System History Log 23 Audit Logging 24 Verify Cisco Unified Communications Manager Services Are Running 28
Cisco Unified Communications Manager System Issues 31 Cisco Unified Communications Manager System Not Responding 31 Cisco Unified Communications Manager System Stops Responding 32 Cisco Unified Communications Manager Administration Does Not Display 33 Error When Attempting to Access Cisco Unified Communications Manager Administration 33 Error When Attempting to Access Cisco Unified Communications Manager Administration on a Subsequent Node 33 You Are Not Authorized to View 34 Problems Displaying or Adding Users with Cisco Unified Communications Manager 34 Name to Address Resolution Failing 35 Port 80 Blocked Between Your Browser and the Cisco Unified Communications Manager Server 36 Improper Network Setting Exists in the Remote Machine 36 Manage Impact of Cisco RAID Operations 37

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) iv

Contents
Database Replication 37 Replication Fails Between the Publisher and the Subscriber Server 38 Database Replication Does Not Occur When Connectivity Is Restored on Lost Node 41 Database Tables Out of Sync Do Not Trigger Alert 41 Resetting Database Replication When You Are Reverting to an Older Product Release 42 utils dbreplication clusterreset 43 utils dbreplication dropadmindb 43
LDAP Authentication Fails 43 Issues with LDAP Over SSL 44 Open LDAP Cannot Verify the Certificate to Connect to the LDAP Server 45 JTAPI Subsystem Startup Problems 46
JTAPI Subsystem is OUT_OF_SERVICE 46 MIVR-SS_TEL-4-ModuleRunTimeFailure 47 MIVR-SS_TEL-1-ModuleRunTimeFailure 49
JTAPI Subsystem is in PARTIAL_SERVICE 50 Security Issues 50
Security Alarms 51 Security Performance Monitor Counters 51 Reviewing Security Log and Trace Files 52 Troubleshooting Certificates 53 Troubleshooting Ciphers 53
Troubleshooting DRS and CDR Functionality 53 Troubleshooting CTL Security Tokens 54
Troubleshooting a Locked Security Token After You Consecutively Enter an Incorrect Security Token Password 54
Troubleshooting If You Lose One Security Token (Etoken) 54 Troubleshooting If You Lose All Security Tokens (Etoken) 55 Troubleshooting ITL Files 55 Troubleshooting CAPF 56 Troubleshooting the Authentication String on the Phone 56 Troubleshooting If the Locally Significant Certificate Validation Fails 56 Verifying That the CAPF Certificate Is Installed on All Servers in the Cluster 56 Verifying That a Locally Significant Certificate Exists on the Phone 57 Verifying That a Manufacture-Installed Certificate (MIC) Exists in the Phone 57
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) v

Contents

CHAPTER 4
CHAPTER 5 CHAPTER 6

Troubleshooting Encryption for Phones and Cisco IOS MGCP Gateways 57 Using Packet Capturing 57
CAPF Error Codes 58
Device Issues 61 Voice Quality 61 Lost or Distorted Audio 62 Correcting Audio Problems From the Cisco Unified IP Phone 63 Echo 64 One-Way Audio or No Audio 65 Codec and Region Mismatches 69 Location and Bandwidth 70 Phone Issues 70 Phone Resets 70 Dropped Calls 71 Phones Not Registering 72 Gateway Issues 72 Gateway Reorder Tone 72 Gateway Registration Failure 73 Gatekeeper Issues 78 Admission Rejects 78 Registration Rejects 79 B-Channel Remains Locked When Restart_Ack Does Not Contain Channel IE 79 Incorrect Device Registration Status Displays 80
Dial Plans and Routing Issues 81 Route Partitions and Calling Search Spaces 81 Group Pickup Configuration 83 Dial Plan Issues 83 Problem When Dialing a Number 84 Secure Dial Plan 85 Automated Alternate Routing (AAR) Limitation with Remote Gateways 85
Cisco Unified Communications Manager Services Issues 87

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) vi

Contents

CHAPTER 7 CHAPTER 8

No Available Conference Bridge 87 Hardware Transcoder Not Working As Expected 89 No Supplementary Services Are Available on an Established Call 90
Voice Messaging Issues 93 Voice Messaging Stops After 30 Seconds 93 Cisco Unity System Does Not Roll Over: Receive Busy Tone 94 Calls That Are Forwarded to Voice Messaging System Get Treated as a Direct Call to Cisco Unity System 94 Administrator Account Is Not Associated with Cisco Unity Subscriber 95
Troubleshooting Features and Services 97 Troubleshooting Barge 97 Troubleshooting Call Back 98 Problems Using Call Back 98 User presses Callback softkey before phone rings 98 User unplugs or resets phone after pressing the CallBack softkey but before Call Back occurs 98 Caller misses availability notification before phone reset. Replace/retain screen does not explicitly state that availability notification occurred. 99 Error Messages for Call Back 100 Locating the Call Back Log Files 100 Troubleshooting Call Control Discovery 100 Troubleshooting Call Park 102 Troubleshooting Ciphers 103 Troubleshooting DRS and CDR Functionality 103 Troubleshooting Cisco Extension Mobility 103 Troubleshooting General Problems with Cisco Extension Mobility 104 Troubleshooting Cisco Extension Mobility Error Messages 104 Troubleshooting Cisco Unified Communications Manager Assistant 106 IPMAConsoleInstall.jsp Displays Error: HTTP Status 503-This Application is Not Currently Available 107 IPMAConsoleInstall.jsp Displays Error: No Page Found Error 107 Exception: java.lang.ClassNotFoundException: InstallerApplet.class 108 Automatic Installation of MS Virtual Machine Is No Longer Provided for Download 108

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) vii

Contents

User Authentication Fails 109 Assistant Console Displays Error: System Error – Contact System Administrator 109 Assistant Console Displays Error: Cisco IP Manager Assistant Service Unreachable 110 Calls Do Not Get Routed When Filtering Is On or Off 111 Cisco IP Manager Assistant Service Cannot Initialize 112 Calling Party Gets a Reorder Tone 113 Manager Is Logged Out While the Service Is Still Running 113 Manager Cannot Intercept Calls That Are Ringing on the Assistant Proxy Line 114 Not Able to Call the Manager Phone When Cisco IP Manager Assistant Service is Down 114 Troubleshooting Cisco Unified Mobility 115 Cisco Unified Mobility User Hangs Up Mobile Phone But Cannot Resume Call on Desktop Phone 115 Dial-via-Office-Related SIP Error Codes 116 Troubleshooting Cisco Web Dialer 117 Authentication Error 117 Service Temporarily Unavailable 117 Directory Service Down 118 Cisco CTIManager Down 118 Session Expired, Please Login Again 118 User Not Logged in on Any Device 119 Failed to Open Device/Line 119 Destination Not Reachable 119 Troubleshooting Directed Call Park 120 Troubleshooting External Call Control 121 Troubleshooting Hotline 124 Troubleshooting Immediate Divert 125 Key Is Not Active 125 Temporary Failure 126 Busy 126 Troubleshooting Intercom 126 Getting Busy Tone When Dialing Out of Intercom Line 127 Intercom Calls Do Not Go to Connected State When Going Off Hook by Using Speaker, Handset,
or Headset 127 Troubleshooting SCCP 127
Intercom Lines Not Showing Up on Phone When Button Template Has Them 127

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) viii

Contents
Intercom Lines Not Showing Up When Phone Falls Back to SRST 128 Troubleshooting SIP 128
Debugging Phones That Are Running SIP 128 Configuration of Phones That Are Running SIP 128 Cisco Extension Mobility User Is Logged In But Intercom Line Does Not Display 128 Where to Find More Information 129 Troubleshooting IPv6 129 Phones Do Not Register with Cisco Unified Communications Manager 129 Calls Over SIP Trunks Fail 130 Calls Between Devices Fail 130 Music On Hold Does Not Play on Phone 130 Troubleshooting Logical Partitioning 131 Logical Partitioning Does Not Function As Expected 131 Logical Partitioning Policies Require Adjustment 132 Troubleshooting SIP with DNS Caching Enabled 133 Logging 133 Log file 133 Packet Capture 134 A/AAAA record caching is not working 134 Hostname resolution returning wrong IP address 135 Cannot find log 135 Set nscd attributes through CLI 136 CLI command to set TTL 136 A/AAAA Record Queries before TTL expires 136 Clearing the cache 136 Content of AAAA record cache 137 Troubleshooting SAML Single Sign On 137 Redirection to IdP fails 137 IdP Authentication Fails 137 Redirection to Unified Communications Manager fails 138 Run Test Fails 138 SAML Single Sign On Page Shows Incorrect Status on Cluster 138 General Tips 139
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) ix

Contents

CHAPTER 9 CHAPTER 10

SNMP Troubleshooting 141 Troubleshooting Tips 141 CISCO-CCM-MIB Tips 142 General Tips 142 Limitations 145 Frequently Asked Questions 146 HOST- RESOURCES-MIB Tips 151 Logs for Collection 151 Disk Space and RTMT 151 Frequently Asked Questions 152 CISCO-CDP-MIB Tips 154 General Tips 154 Frequently Asked Questions 154 SYSAPP-MIB Tips 154 Collecting Logs 155 Using Servlets in Cisco Unified Communications Manager 8.0 155 SNMP Developer Tips 156 Where to Find More Information 158
Opening a Case With TAC 159 Information You Will Need 160 Required Preliminary Information 160 Network Layout 160 Problem Description 161 General Information 161 Online Cases 162 Serviceability Connector 162 Serviceability Connector Overview 162 Benefits of Using Serviceability Service 162 TAC Support for Serviceability Connector 163 Cisco Live! 163 Remote Access 163 Cisco Secure Telnet 164

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) x

Contents

CHAPTER 11 CHAPTER 12

Firewall Protection 164 Cisco Secure Telnet Design 164 Cisco Secure Telnet Structure 165 Set up a Remote Account 165
Case Study: Troubleshooting Cisco Unified IP Phone Calls 167 Troubleshooting Intracluster Cisco Unified IP Phone Calls 167 Sample Topology 167 Cisco Unified IP Phone Initialization Process 168 Cisco Unified Communications Manager Initialization Process 169 Self-Starting Processes 169 Cisco Unified Communications Manager Registration Process 170 Cisco Unified Communications Manager KeepAlive Process 171 Cisco Unified Communications Manager Intracluster Call Flow Traces 171 Troubleshooting Intercluster Cisco Unified IP Phone Calls 175 Sample Topology 176 Intercluster H.323 Communication 176 Call Flow Traces 176 Failed Call Flow 177
Case Study: Troubleshooting Cisco Unified IP Phone-to-Cisco IOS Gateway Calls 179 Call Flow Traces 179 Debug Messages and Show Commands on the Cisco IOS Gatekeeper 182 Debug Messages and Show Commands on the Cisco IOS Gateway 184 Cisco IOS Gateway with T1/PRI Interface 187 Cisco IOS Gateway with T1/CAS Interface 188

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) xi

Contents
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) xii

Preface

This preface describes the purpose, audience, organization, and conventions of this guide and provides information on how to obtain related documentation.
· Purpose, on page xiii · Audience, on page xiii · Organization, on page xiv · Related Documentation, on page xv · Conventions, on page xv · Obtaining Documentation, Obtaining Support, and Security Guidelines, on page xvi · Cisco Product Security Overview, on page xvi

Purpose

The Troubleshooting Guide for Cisco Unified Communications Manager provides troubleshooting procedures for this release of Unified Communications Manager.

Note The information in this version of the Troubleshooting Guide for Unified Communications Manager may not apply to earlier releases of the Unified Communications Manager software.
This document does not cover every possible trouble event that might occur on a Unified Communications Manager system but instead focuses on those events that are frequently seen by the Cisco Technical Assistance Center (TAC) or frequently asked questions from newsgroups.
Audience
The Troubleshooting Guide for Unified Communications Manager provides guidance for network administrators who are responsible for managing the Unified Communications Manager system, for enterprise managers, and for employees. This guide requires knowledge of telephony and IP networking technology.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) xiii

Organization

Preface

Organization

The following table shows how this guide is organized.
Table 1: How This Document Is Organized

Chapter and Title

Description

Troubleshooting Overview, on page 1

Provides an overview of the tools and resources that are available for troubleshooting the Unified Communications Manager.

Troubleshooting Tools, on page 5

Addresses the tools and utilities that you can use to configure, monitor, and troubleshoot Unified Communications Manager and provides general guidelines for collecting information to avoid repetitive testing and re-collection of identical data.

Cisco Unified Communications Manager Describes solutions for the most common issues that relate to

System Issues, on page 31

a Unified Communications Manager system.

Device Issues, on page 61

Describes solutions for the most common issues that relate to IP phones and gateways.

Dial Plans and Routing Issues, on page 81 Describes solutions for the most common issues that relate to dial plans, route partitions, and calling search spaces.

Cisco Unified Communications Manager Services Issues, on page 87

Describes solutions for the most common issues related to services, such as conference bridges and media termination points.

Voice Messaging Issues, on page 93

Describes solutions for the most common voice-messaging issues.

Troubleshooting Features and Services, on Provides information to help you resolve common issues with

page 97

Unified Communications Manager features and services.

SNMP Troubleshooting, on page 141

Provides information on how to troubleshoot with SNMP

Opening a Case With TAC, on page 159 Describes what information is needed to open a case for TAC.

Case Study: Troubleshooting Cisco Unified Describes in detail the call flow between two Cisco Unified IP

IP Phone Calls, on page 167

Phones within a cluster.

Case Study: Troubleshooting Cisco Unified Describes a Cisco Unified IP Phone calling through a Cisco

IP Phone-to-Cisco IOS Gateway Calls, on IOS Gateway to a phone that is connected through a local PBX

page 179

or on the Public Switched Telephone Network (PSTN).

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) xiv

Preface

Related Documentation

Related Documentation
Refer to the Cisco Unified Communications Manager Documentation Guide for further information about related Cisco IP telephony applications and products. The following URL shows an example of the path to the documentation guide: http://www.cisco.com/en/US/products/sw/voicesw/ps556/products_documentation_roadmaps_list.html For documentation that relates to Cisco Unity, refer to the following URL:
https://www.cisco.com/c/en/us/support/unified-communications/index.html

Conventions

This document uses the following conventions:

Convention

Description

boldface font

Commands and keywords are in boldface.

italic font

Arguments for which you supply values are in italics.

[]

Elements in square brackets are optional.

{x|y|z}

Alternative keywords are grouped in braces and separated by vertical bars.

[x|y|z]

Optional alternative keywords are grouped in brackets and separated by vertical bars.

string

A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

screen font

Terminal sessions and information the system displays are in screen font.

boldface screen font Information you must enter is in boldface screen font.

italic screen font

Arguments for which you supply values are in italic screen font.

Nonprinting characters, such as passwords, are in angle

brackets.

Notes use the following conventions:

Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the publication.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) xv

Obtaining Documentation, Obtaining Support, and Security Guidelines

Preface

Timesavers use the following conventions:
Timesaver Means the described action saves time. You can save time by performing the action described in the paragraph. Tips use the following conventions:
Tip Means the information contains useful tips. Cautions use the following conventions:
Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data.

Warnings use the following conventions:
Warning This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, you must be aware of the hazards involved with electrical circuitry and familiar with standard practices for preventing accidents.
Obtaining Documentation, Obtaining Support, and Security Guidelines
For information on obtaining documentation, obtaining support, providing documentation feedback, security guidelines, and also recommended aliases and general Cisco documents, see the monthly What”s New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at: http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Cisco Product Security Overview
This product contains cryptographic features and is subject to United States and local country laws governing import, export, transfer and use. Delivery of Cisco cryptographic products does not imply third-party authority to import, export, distribute or use encryption. Importers, exporters, distributors and users are responsible for compliance with U.S. and local country laws. By using this product you agree to comply with applicable laws and regulations. If you are unable to comply with U.S. and local laws, return this product immediately. Further information regarding U.S. export regulations may be found at http://www.access.gpo.gov/bis/ear/ear_data.html.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) xvi

1 C H A P T E R
Troubleshooting Overview
This section provides the necessary background information and available resources to troubleshoot the Unified Communications Manager.
· Cisco Unified Serviceability, on page 1 · Cisco Unified Communications Operating System Administration, on page 2 · General Model of Problem Solving, on page 2 · Network Failure Preparation, on page 3 · Where to Find More Information, on page 3
Cisco Unified Serviceability
Cisco Unified Serviceability, a web-based troubleshooting tool for Unified Communications Manager, provides the following functionality to assist administrators troubleshoot system problems:
· Saves Unified Communications Manager services alarms and events for troubleshooting and provides alarm message definitions.
· Saves Unified Communications Manager services trace information to various log files for troubleshooting. Administrators can configure, collect, and view trace information.
· Monitors real-time behavior of the components in a Unified Communications Manager cluster through the real-time monitoring tool (RTMT).
· Generates reports for Quality of Service, traffic, and billing information through Unified Communications Manager CDR Analysis and Reporting (CAR).
· Provides feature services that you can activate, deactivate, and view through the Service Activation window.
· Provides an interface for starting and stopping feature and network services. · Archives reports that are associated with Cisco Unified Serviceability tools. · Allows Unified Communications Manager to work as a managed device for SNMP remote management
and troubleshooting. · Monitors the disk usage of the log partition on a server (or all servers in the cluster).
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 1

Cisco Unified Communications Operating System Administration

Troubleshooting Overview

Access Cisco Unified Serviceability from the Cisco Unified Communications Manager Administration window by choosing Cisco Unified Serviceability from the Navigation drop-down list box. Installing the Unified Communications Manager software automatically installs Cisco Unified Serviceability and makes it available. See Cisco Unified Serviceability Administration Guide for detailed information and configuration procedures on the serviceability tools.
Cisco Unified Communications Operating System Administration
Cisco Unified Communications Operating System Administration allows you to perform the following tasks to configure and manage the Cisco Unified Communications Operating System:
· Check software and hardware status. · Check and update IP addresses. · Ping other network devices. · Manage Network Time Protocol servers. · Upgrade system software and options. · Restart the system.
Refer to the Administration Guide for Cisco Unified Communications Manager for detailed information and configuration procedures on the serviceability tools.
General Model of Problem Solving
When troubleshooting a telephony or IP network environment, define the specific symptoms, identify all potential problems that could be causing the symptoms, and then systematically eliminate each potential problem (from most likely to least likely) until the symptoms disappear. The following steps provide guidelines to use in the problem-solving process.
Procedure 1. Analyze the network problem and create a clear problem statement. Define symptoms and potential causes. 2. Gather the facts that you need to help isolate possible causes. 3. Consider possible causes based on the facts that you gathered. 4. Create an action plan based on those causes. Begin with the most likely problem and devise a plan in
which you manipulate only one variable. 5. Implement the action plan; perform each step carefully while testing to see whether the symptom disappears. 6. Analyze the results to determine whether the problem has been resolved. If the problem was resolved,
consider the process complete. 7. If the problem has not been resolved, create an action plan based on the next most probable cause on your
list. Return to 4, on page 2 and repeat the process until the problem is solved.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 2

Troubleshooting Overview

Network Failure Preparation

Make sure that you undo anything that you changed while implementing your action plan. Remember that you want to change only one variable at a time.
Note If you exhaust all the common causes and actions (either those outlined in this document or others that you have identified in your environment), contact Cisco TAC.
Network Failure Preparation
You can always recover more easily from a network failure if you are prepared ahead of time. To determine if you are prepared for a network failure, answer the following questions:
· Do you have an accurate physical and logical map of your internetwork that outlines the physical location of all of the devices on the network and how they are connected as well as a logical map of network addresses, network numbers, and subnetworks?
· Do you have a list of all network protocols that are implemented in your network for each of the protocols implemented and a list of the network numbers, subnetworks, zones, and areas that are associated with them?
· Do you know which protocols are being routed and the correct, up-to-date configuration information for each protocol?
· Do you know which protocols are being bridged? Are any filters configured in any of these bridges, and do you have a copy of these configurations? Is this applicable to Unified Communications Manager?
· Do you know all the points of contact to external networks, including any connections to the Internet? For each external network connection, do you know what routing protocol is being used?
· Has your organization documented normal network behavior and performance, so you can compare current problems with a baseline?
If you can answer yes to these questions, faster recovery from a failure results.
Where to Find More Information
Use the following links for information on various IP telephony topics: · For further information about related Cisco IP telephony applications and products, see the Cisco Unified Communications Manager Documentation Guide. The following URL shows an example of the path to the documentation guide: https://www.cisco.com/en/US/products/sw/voicesw/ps556/products_documentation_roadmaps_list.html · For documentation related to Cisco Unity, see the following URL: https://www.cisco.com/en/US/products/sw/voicesw/ps2237/tsd_products_support_series_home.html
· For documentation related to Cisco Emergency Responder, see the following URL: https://www.cisco.com/en/US/products/sw/voicesw/ps842/tsd_products_support_series_home.html

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 3

Where to Find More Information

Troubleshooting Overview

· For documentation related to Cisco Unified IP Phone, see the following URL: https://www.cisco.com/en/US/products/hw/phones/ps379/tsd_products_support_series_home.html
· For information on designing and troubleshooting IP telephony networks, see the Cisco IP Telephony Solution Reference Network Design Guides that are available at: https://www.cisco.com/go/srnd

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 4

2 C H A P T E R
Troubleshooting Tools
This section addresses the tools and utilities that you use to configure, monitor, and troubleshoot Unified Communications Manager and provides general guidelines for collecting information to avoid repetitive testing and recollection of identical data.
Note To access some of the URL sites that are listed in this document, you must be a registered user, and you must be logged in.
· Cisco Unified Serviceability Troubleshooting Tools, on page 5 · Command Line Interface, on page 6 · kerneldump Utility, on page 7 · Network Management, on page 9 · Sniffer Traces, on page 10 · Debugs, on page 10 · Cisco Secure Telnet, on page 11 · Packet Capture, on page 11 · Common Troubleshooting Tasks, Tools, and Commands, on page 17 · Troubleshooting Tips, on page 20 · System History Log, on page 21 · Audit Logging, on page 24 · Verify Cisco Unified Communications Manager Services Are Running, on page 28
Cisco Unified Serviceability Troubleshooting Tools
Refer to the Cisco Unified Serviceability Administration Guide for detailed information of the following different types of tools that Cisco Unified Serviceability provides to monitor and analyze the various Unified Communications Manager systems.
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 5

Command Line Interface

Troubleshooting Tools

Table 2: Serviceability Tools

Term

Definition

Cisco Unified Real-Time This tool provides real-time information about Unified Communications Manager Monitoring Tool (RTMT) devices and performance counters and enables you to collect traces.
Performance counters can be system-specific or Unified Communications Manager specific. Objects comprise the logical groupings of like counters for a specific device or feature, such as Cisco Unified IP Phones or Unified Communications Manager System Performance. Counters measure various aspects of system performance. Counters measure statistics such as the number of registered phones, calls that are attempted and calls in progress.

Alarms

Administrators use alarms to obtain the run-time status and state of the Unified Communications Manager system. Alarms contain information about system problems such as explanation and recommended action.
Administrators search the alarm definitions database for alarm information. The alarm definition contains a description of the alarm and recommended actions.

Trace

Administrators and Cisco engineers use trace files to obtain specific information about Unified Communications Manager service problems. Cisco Unified Serviceability sends configured trace information to the trace log file. Two types of trace log files exist: SDI and SDL.

Every service includes a default trace log file. The system traces system diagnostic interface (SDI) information from the services and logs run-time events and traces to a log file.

The SDL trace log file contains call-processing information from services such as Cisco CallManager and Cisco CTIManager. The system traces the signal distribution layer (SDL) of the call and logs state transitions into a log file.

Note

In most cases, you will only gather SDL traces when Cisco Technical

Assistance Center (TAC) requests you to do so.

Quality Report Tool

This term designates voice quality and general problem-reporting utility in Cisco Unified Serviceability.

Serviceability Connector The Cisco Webex Serviceability service increases the speed with which Cisco technical assistance staff can diagnose issues with your infrastructure. It automates the tasks of finding, retrieving, and storing diagnostic logs and information into an SR case. The service also triggers analysis against diagnostic signatures so that TAC can more efficiently identify and resolve issues with your on-premises equipment.

Command Line Interface
Use the command line interface (CLI) to access the Unified Communications Manager system for basic maintenance and failure recovery. Obtain access to the system by either a hard-wired terminal (a system monitor and keyboard) or by performing a SSH session.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 6

Troubleshooting Tools

kerneldump Utility

The account name and password get created at install time. You can change the password after install, but you never can change the account name. A command represents a text instruction that caused the system to perform some function. Commands may be stand alone, or they can have mandatory or optional arguments or options. A level comprises a collection of commands; for example, show designates a level, whereas show status specifies a command. Each level and command also includes an associated privilege level. You can execute a command only if you have sufficient privilege level. For complete information on the Unified Communications Manager CLI command set, see the Command Line Interface Reference Guide for Cisco Unified Solutions.
kerneldump Utility
The kerneldump utility allows you to collect crash dump logs locally on the affected machine without requiring a secondary server. In a Unified Communications Manager cluster, you only need to ensure the kerneldump utility is enabled on the server before you can collect the crash dump information.
Note Cisco recommends that you verify the kerneldump utility is enabled after you install Unified Communications Manager to allow for more efficient troubleshooting. If you have not already done so, enable the kerneldump utility before you upgrade the Unified Communications Manager from supported appliance releases.
Important Enabling or disabling the kerneldump utility will require a reboot of the node. Do not execute the enable command unless you are within a window where a reboot would be acceptable.
The command line interface (CLI) for the Cisco Unified Communications Operating System can be used to enable, disable, or check the status of the kerneldump utility. Use the following procedure to enable the kernel dump utility:
Working with Files That Are Collected by the Utility To view the crash information from the kerneldump utility, use the Cisco Unified Real-Time Monitoring Tool or the Command Line Interface (CLI). To collect the kerneldump logs by using the Cisco Unified Real-Time Monitoring Tool, choose the Collect Files option from Trace & Log Central. From the Select System Services/Applications tab, choose the Kerneldump logs check box. For more information on collecting files using Cisco Unified Real-Time Monitoring Tool, see the Cisco Unified Real-Time Monitoring Tool Administration Guide. To use the CLI to collect the kerneldump logs, use the “file” CLI commands on the files in the crash directory. These are found under the “activelog” partition. The log filenames begin with the IP address of the kerneldump client and end with the date that the file is created. For more information on the file commands, refer to the Command Line Interface Reference Guide for Cisco Unified Solutions.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 7

Enable the Kerneldump Utility

Troubleshooting Tools

Enable the Kerneldump Utility
Use this procedure to enable the kerneldump utility. In the event of a kernel crash, the utility provides a mechanism for collecting and dumping the crash. You can configure the utility to dump logs to the local server or to an external server.
Procedure

Step 1 Step 2
Step 3

Log in to the Command Line Interface. Complete either of the following:
· To dump kernel crashes on the local server, run the utils os kernelcrash enable CLI command. · To dump kernel crashes to an external server, run the utils os kerneldump ssh enable
CLI command with the IP address of the external server.
Reboot the server.

Example

Note If you need to disable the kerneldump utility, you can run the utils os kernelcrash disable CLI command to disable the local server for core dumps and the utils os kerneldump ssh disable CLI command to disable the utility on the external server.

What to do next Configure an email alert in the Real-Time Monitoring Tool to be advised of core dumps. For details, see Enable Email Alert for Core Dump, on page 8 Refer to the Troubleshooting Guide for Cisco Unified Communications Manager for more information on the kerneldump utility and troubleshooting.
Enable Email Alert for Core Dump
Use this procedure to configure the Real-Time Monitoring Tool to email the administrator whenever a core dump occurs.
Procedure

Step 1 Step 2 Step 3

Select System > Tools > Alert > Alert Central. Right-click CoreDumpFileFound alert and select Set Alert Properties. Follow the wizard prompts to set your preferred criteria: a) In the Alert Properties: Email Notification popup, make sure that Enable Email is checked and click
Configure to set the default alert action, which will be to email an administrator.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 8

Troubleshooting Tools

Network Management

Step 4

b) Follow the prompts and Add a Recipient email address. When this alert is triggered, the default action is to email this address.
c) Click Save.
Set the default Email server: a) Select System > Tools > Alert > Config Email Server. b) Enter the e-mail server and port information to send email alerts. c) Enter the Send User Id. d) Click OK.

Network Management
Use the network management tools for Unified Communications Manager remote serviceability. · System Log Management
· Cisco Discovery Protocol Support
· Simple Network Management Protocol support
Refer to the documentation at the URLs provided in the sections for these network management tools for more information.
System Log Management
Although it can be adapted to other network management systems, Cisco Syslog Analysis, which is packaged with Resource Manager Essentials (RME), provides the best method to manage Syslog messages from Cisco devices. Cisco Syslog Analyzer serves as the component of Cisco Syslog Analysis that provides common storage and analysis of the system log for multiple applications. The other major component, Syslog Analyzer Collector, gathers log messages from Unified Communications Manager servers. These two Cisco applications work together to provide a centralized system logging service for Cisco Unified Communications Solutions. Refer to the following URL for RME documentation: http://www.cisco.com/en/US/products/sw/cscowork/ps2073/products_tech_note09186a00800a7275.shtml
Cisco Discovery Protocol Support
The Cisco Discovery Protocol Support enables discovery of Unified Communications Manager servers and management of those servers. Refer to the following URL for RME documentation: http://www.cisco.com/en/US/products/sw/cscowork/ps2073/products_tech_note09186a00800a7275.shtml

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 9

Simple Network Management Protocol Support

Troubleshooting Tools

Simple Network Management Protocol Support
Network management systems (NMS) use SNMP, an industry-standard interface, to exchange management information between network devices. A part of the TCP/IP protocol suite, SNMP enables administrators to remotely manage network performance, find and solve network problems, and plan for network growth.
An SNMP-managed network comprises three key components: managed devices, agents, and network management systems.
· A managed device designates a network node that contains an SNMP agent and resides on a managed network. Managed devices collect and store management information and make it available by using SNMP.
· An agent, as network management software, resides on a managed device. An agent contains local knowledge of management information and translates it into a form that is compatible with SNMP.
· A network management system comprises an SNMP management application together with the computer on which it runs. An NMS executes applications that monitor and control managed devices. An NMS provides the bulk of the processing and memory resources that are required for network management. The following NMSs share compatibility with Unified Communications Manager:
· CiscoWorks Common Services Software
· HP OpenView
· Third-party applications that support SNMP and Unified Communications Manager SNMP interfaces

Sniffer Traces
Typically, you collect sniffer traces by connecting a laptop or other sniffer- equipped device on a Catalyst port that is configured to span the VLAN or port(s) (CatOS, Cat6K-IOS, XL-IOS) that contains the trouble information. If no free port is available, connect the sniffer-equipped device on a hub that is inserted between the switch and the device.

Tip To help facilitate reading and interpreting of the traces by the TAC engineer, Cisco recommends using Sniffer Pro software because it is widely used within the TAC.
Have available the IP/MAC addresses of all equipment that is involved, such as IP phones, gateways, Unified Communications Managers, and so on.

Debugs

The output from debug privileged EXEC commands provides diagnostic information about a variety of internetworking event that relate to protocol status and network activity in general.
Set up your terminal emulator software (such as HyperTerminal), so it can capture the debug output to a file. In HyperTerminal, click Transfer; then, click Capture Text and choose the appropriate options.
Before running any IOS voice gateway debugs, make sure that servicetimestampsdebugdatetimemsec is globally configured on the gateway.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 10

Troubleshooting Tools

Cisco Secure Telnet

Note Avoid collecting debugs in a live environment during operation hours.
Preferably, collect debugs during non-working hours. If you must collect debugs in a live environment, configure no logging console and loggingbuffered. To collect the debugs, use show log. Because some debugs can be lengthy, collect them directly on the console port (default logging console) or on the buffer (logging buffer). Collecting debugs over a Telnet session may impact the device performance, and the result could be incomplete debugs, which requires that you re-collect them. To stop a debug, use the no debug all or undebug all commands. Verify that the debugs have been turned off by using the command show debug.
Cisco Secure Telnet
Cisco Secure Telnet allows Cisco Service Engineers (CSE) transparent firewall access to the Unified Communications Manager node on your site. Using strong encryption, Cisco Secure Telnet enables a special Telnet client from Cisco Systems to connect to a Telnet daemon behind your firewall. This secure connection allows remote monitoring and troubleshooting of your Unified Communications Manager nodes, without requiring firewall modifications.
Note Cisco provides this service only with your permission. You must ensure that a network administrator is available at your site to help initiate the process.
Packet Capture
This section contains information about packet capture. Related Topics
Packet Capturing Overview, on page 11 Configuration Checklist for Packet Capturing, on page 12 Adding an End User to the Standard Packet Sniffer Access Control Group , on page 13 Configuring Packet-Capturing Service Parameters, on page 13 Configuring Packet Capturing in the Phone Configuration Window, on page 14 Configuring Packet Capturing in Gateway and Trunk Configuration Windows, on page 14 Packet-Capturing Configuration Settings, on page 16 Analyzing Captured Packets, on page 17
Packet Capturing Overview
Because third-party troubleshooting tools that sniff media and TCP packets do not work after you enable encryption, you must use Unified Communications Manager to perform the following tasks if a problem occurs:
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 11

Configuration Checklist for Packet Capturing

Troubleshooting Tools

· Analyze packets for messages that are exchanged between Unified Communications Manager and the device [Cisco Unified IP Phone (SIP and SCCP), Cisco IOS MGCP gateway, H.323 gateway, H.323/H.245/H.225 trunk, or SIP trunk].
· Capture the Secure Real Time Protocol (SRTP) packets between the devices. · Extract the media encryption key material from messages and decrypt the media between the devices.
Tip Performing this task for several devices at the same time may cause high CPU usage and call-processing interruptions. Cisco strongly recommends that you perform this task when you can minimize call-processing interruptions.
For more information, see the Security Guide for Cisco Unified Communications Manager.
Configuration Checklist for Packet Capturing
Extracting and analyzing pertinent data includes performing the following tasks.
Procedure 1. Add end users to the Standard Packet Sniffer Users group. 2. Configure packet capturing service parameters in the Service Parameter Configuration window in Cisco
Unified Communications Manager Administration; for example, configure the Packet Capture Enable service parameter. 3. Configure packet capturing settings on a per-device basis in the Phone or Gateway or Trunk Configuration window.
Note Cisco strongly recommends that you do not enable packet capturing for many devices at the same time because this task may cause high CPU usage in your network.
4. Capture SRTP packets by using a sniffer trace between the affected devices. Refer to the documentation that supports your sniffer trace tool.
5. After you capture the packets, set the Packet Capture Enable service parameter to False. 6. Gather the files that you need to analyze the packets. 7. Cisco Technical Assistance Center (TAC) analyzes the packets. Contact TAC directly to perform this
task.
Related Topics Adding an End User to the Standard Packet Sniffer Access Control Group , on page 13 Analyzing Captured Packets, on page 17 Configuring Packet Capturing in Gateway and Trunk Configuration Windows, on page 14 Configuring Packet Capturing in the Phone Configuration Window, on page 14 Configuring Packet-Capturing Service Parameters, on page 13 Packet-Capturing Configuration Settings, on page 16

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 12

Troubleshooting Tools

Adding an End User to the Standard Packet Sniffer Access Control Group

Adding an End User to the Standard Packet Sniffer Access Control Group
End users that belong to the Standard Packet Sniffer Users group can configure the Packet Capture Mode and Packet Capture Duration settings for devices that support packet capturing. If the user does not exist in the Standard Packet Sniffer Access Control Group, the user cannot initiate packet capturing. The following procedure, which describes how to add an end user to the Standard Packet Sniffer Access Control Group, assumes that you configured the end user in Cisco Unified Communications Manager Administration, as described in the Administration Guide for Cisco Unified Communications Manager.
Procedure 1. Find the access control group, as described in the Administration Guide for Cisco Unified Communications
Manager. 2. After the Find/List window displays, click the Standard Packet Sniffer Users link. 3. Click the Add Users to Group button. 4. Add the end user, as described in the Administration Guide for Cisco Unified Communications Manager. 5. After you add the user, click Save.
Configuring Packet-Capturing Service Parameters
To configure parameters for packet capturing, perform the following procedure:
Procedure 1. In Unified Communications Manager, choose System > Service Parameters. 2. From the Server drop-down list box, choose an Active server where you activated the Cisco CallManager
service. 3. From the Service drop-down list box, choose the Cisco CallManager (Active) service. 4. Scroll to the TLS Packet Capturing Configuration pane and configure the packet capturing settings.
Tip For information on the service parameters, click the name of the parameter or the question mark that displays in the window.
Note For packet capturing to occur, you must set the Packet Capture Enable service parameter to True.
5. For the changes to take effect, click Save. 6. You can continue to configure packet-capturing.
Related Topics Configuring Packet Capturing in Gateway and Trunk Configuration Windows, on page 14 Configuring Packet Capturing in the Phone Configuration Window, on page 14

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 13

Configuring Packet Capturing in the Phone Configuration Window

Troubleshooting Tools

Configuring Packet Capturing in the Phone Configuration Window
After you enable packet capturing in the Service Parameter window, you can configure packet capturing on a per-device basis in the Phone Configuration window of Cisco Unified Communications Manager Administration.
You enable or disable packet capturing on a per-phone basis. The default setting for packet capturing equals None.

Caution

Cisco strongly recommends that you do not enable packet capturing for many phones at the same time because this task may cause high CPU usage in your network.
If you do not want to capture packets or if you completed the task, set the Packet Capture Enable service parameter to False.

To configure packet capturing for phones, perform the following procedure:

Procedure 1. Before you configure the packet-capturing settings, see the topics related to packet capturing configuration.
2. Find the SIP or SCCP phone, as described in the System Configuration Guide for Cisco Unified Communications Manager.
3. After the Phone Configuration window displays, configure the troubleshooting settings, as described in Packet-Capturing Configuration Settings.
4. After you complete the configuration, click Save.
5. In the Reset dialog box, click OK.

Tip Although Cisco Unified Communications Manager Administration prompts you to reset the device, you do not need to reset the device to capture packets.
Additional Steps Capture SRTP packets by using a sniffer trace between the affected devices. After you capture the packets, set the Packet Capture Enable service parameter to False. Related Topics
Analyzing Captured Packets, on page 17 Configuration Checklist for Packet Capturing, on page 12
Configuring Packet Capturing in Gateway and Trunk Configuration Windows
The following gateways and trunks support packet capturing in Unified Communications Manager. · Cisco IOS MGCP gateways · H.323 gateways

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 14

Troubleshooting Tools

Configuring Packet Capturing in Gateway and Trunk Configuration Windows

· H.323/H.245/H.225 trunks · SIP trunks
Tip Cisco strongly recommends that you do not enable packet capturing for many devices at the same time because this task may cause high CPU usage in your network. If you do not want to capture packets or if you completed the task, set the Packet Capture Enable service parameter to False.
To configure packet-capturing settings in the Gateway or Trunk Configuration window, perform the following procedure:
Procedure 1. Before you configure the packet-capturing settings, see the topics related to packet capturing configuration. 2. Perform one of the following tasks:
· Find the Cisco IOS MGCP gateway, as described in the System Configuration Guide for Cisco Unified Communications Manager.
· Find the H.323 gateway, as described in the System Configuration Guide for Cisco Unified Communications Manager.
· Find the H.323/H.245/H.225 trunk, as described in the System Configuration Guide for Cisco Unified Communications Manager.
· Find the SIP trunk, as described in the System Configuration Guide for Cisco Unified Communications Manager.
3. After the configuration window displays, locate the Packet Capture Mode and Packet Capture Duration settings.
Tip If you located a Cisco IOS MGCP gateway, ensure that you configured the ports for the Cisco IOS MGCP gateway, as described in the Administration Guide for Cisco Unified Communications Manager. The packet-capturing settings for the Cisco IOS MGCP gateway display in the Gateway Configuration window for endpoint identifiers. To access this window, click the endpoint identifier for the voice interface card.
4. Configure the troubleshooting settings, as described in Packet-Capturing Configuration Settings. 5. After you configure the packet-capturing settings, click Save. 6. In the Reset dialog box, click OK.
Tip Although Cisco Unified Communications Manager Administration prompts you to reset the device, you do not need to reset the device to capture packets.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 15

Packet-Capturing Configuration Settings

Troubleshooting Tools

Additional Steps
Capture SRTP packets by using a sniffer trace between the affected devices. After you capture the packets, set the Packet Capture Enable service parameter to False. Related Topics
Analyzing Captured Packets, on page 17 Configuration Checklist for Packet Capturing, on page 12

Packet-Capturing Configuration Settings

The following table describes the Packet Capture Mode and Packet Capture Duration settings when configuring packet capturing for gateways, trunks, and phones.

Setting

Description

Packet Capture Mode

This setting exists for troubleshooting encryption only; packet capturing may cause high CPU usage or call-processing interruptions. Choose one of the following options from the drop-down list box:

· None–This option, which serves as the default setting, indicates that no packet capturing is occurring. After you complete packet capturing, Unified Communications Manager sets the Packet Capture Mode to None.

· Batch Processing Mode– Unified Communications Manager writes the decrypted or nonencrypted messages to a file, and the system encrypts each file. On a daily basis, the system creates a new file with a new encryption key. Unified Communications Manager, which stores the file for seven days, also stores the keys that encrypt the file in a secure location. Unified Communications Manager stores the file in the PktCap virtual directory. A single file contains the time stamp, source IP address, source IP port, destination IP address, packet protocol, message length, and the message. The TAC debugging tool uses HTTPS, administrator username and password, and the specified day to request a single encrypted file that contains the captured packets. Likewise, the tool requests the key information to decrypt the encrypted file.

Tip

Before you contact TAC, you must capture the SRTP

packets by using a sniffer trace between the affected

devices.

Packet Capture Duration

This setting exists for troubleshooting encryption only; packet capturing may cause high CPU usage or call-processing interruptions.
This field specifies the maximum number of minutes that is allotted for one session of packet capturing. The default setting equals 0, although the range exists from 0 to 300 minutes.
To initiate packet capturing, enter a value other than 0 in the field. After packet capturing completes, the value, 0, displays.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 16

Troubleshooting Tools

Analyzing Captured Packets

Related Topics Configuring Packet Capturing in Gateway and Trunk Configuration Windows, on page 14 Configuring Packet Capturing in the Phone Configuration Window, on page 14
Analyzing Captured Packets
Cisco Technical Assistance Center (TAC) analyzes the packets by using a debugging tool. Before you contact TAC, capture SRTP packets by using a sniffer trace between the affected devices. Contact TAC directly after you gather the following information:
· Packet Capture File–https://<IP address or server name>/pktCap/pktCap.jsp?file=mm-dd-yyyy.pkt, where you browse into the server and locate the packet-capture file by month, date, and year (mm-dd-yyyy)
· Key for the file–https:///pktCap/pktCap.jsp?key =mm-dd-yyyy.pkt, where you browse into the server and locate the key by month, date, and year (mm-dd-yyyy)
· User name and password of end user that belongs to the Standard Packet Sniffer Users group
For more information, see Security Guide for Cisco Unified Communications Manager.

Common Troubleshooting Tasks, Tools, and Commands

This section provides a quick reference for commands and utilities to help you troubleshoot a Unified Communications Manager server with root access disabled. The following table provides a summary of the CLI commands and GUI selections that you can use to gather information troubleshoot various system problems.
Table 3: Summary of CLI Commands and GUI Selections

Information CPU usage
Process state Disk usage

Linux Command top
ps df/du

Serviceability GUI Tool

CLI commands

RTMT

Processor CPU usage:

Go to View tab and select Server > show perf query class Processor

CPU and Memory

Process CPU Usage for all processes:

show perf query counter Process “% CPU Time”

Individual process counter details (including CPU usage)

show perf query instance

RTMT

show perf query counter Process “Process Status”

Go to View tab and select Server > Process

RTMT

show perf query counter Partition”% Used”

Go to View tab and select Server > or show perf query class Partition Disk Usage

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 17

Common Troubleshooting Tasks, Tools, and Commands

Troubleshooting Tools

Information Memory

Linux Command
free

Network status Reboot server

netstats reboot

Collect Traces/logs Sftp, ftp

Serviceability GUI Tool

CLI commands

RTMT

show perf query class Memory

Go to View tab and select Server > CPU and Memory

show network status

Log in to Platform Web page on the utils system restart server
Go to Server > Current Version

RTMT

List file: file list

Go to Tools tab and select Trace > Download files: file get

Trace & Log Central

View a file: file view

The following table provides a list of common problems and tools to use to troubleshoot them.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 18

Troubleshooting Tools

Common Troubleshooting Tasks, Tools, and Commands

Table 4: Troubleshooting Common Problems with CLI Commands and GUI Selections

Task Accessing the database

GUI Tool none

CLI commands
Log in as admin and use any of the following show commands:
· show tech database · show tech dbinuse · show tech dbschema · show tech devdefaults · show tech gateway · show tech locales · show tech notify · show tech procedures · show tech routepatterns · show tech routeplan · show tech systables · show tech table · show tech triggers · show tech version · show tech params*
To run a SQL command, use the run command: · run sql

Freeing up disk space

Note

You can only delete

files from the Log

partition.

Using the RTMT client application, go file delete to the Tools tab and select Trace & Log Central > Collect Files.
Choose the criteria to select the files you want to collect, then check the option Delete Files. This will delete the files on the Unified Communications Manager server after downloading the files to your PC.

Viewing core files

You cannot view the core files;

utils core [options.]

however, you can download the Core

files by using the RTMT application and

selecting Trace & Log Central >

Collect Crash Dump.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 19

Troubleshooting Tips

Troubleshooting Tools

Task

GUI Tool

CLI commands

Rebooting the Unified

Log in to Platform on the server and go utils system restart

Communications Manager server to Restart > Current Version.

Changing debug levels for traces Log in to Cisco Unity Connection

set trace enable [Detailed, Significant, Error, Arbitrary,

Serviceability Administration at

Entry_exit, State_Transition, Special] [syslogmib,

https://:8443/ cdpmib, dbl, dbnotify]

ccmservice/ and choose Trace >

Configuration.

Looking at netstats

none

show network status

Troubleshooting Tips
The following tips may help you when you are troubleshooting the Unified Communications Manager.
Tip Check the release notes for Unified Communications Manager for known problems. The release notes provide descriptions and workaround solutions for known problems.

Tip Know where your devices are registered.
Each Unified Communications Manager log traces files locally. If a phone or gateway is registered to a particular Unified Communications Manager, the call processing gets done on that Unified Communications Manager if the call is initiated there. You will need to capture traces on that Unified Communications Manager to debug a problem. A common mistake involves having devices that are registered on a subscriber server but are capturing traces on the publisher server. These trace files will be nearly empty (and definitely will not have the call in them). Another common problem involves having Device 1 registered to CM1 and Device 2 registered to CM2. If Device 1 calls Device 2, the call trace occurs in CM1, and, if Device 2 calls Device 1, the trace occurs in CM2. If you are troubleshooting a two-way calling issue, you need both traces from both Unified Communications Managers to obtain all the information that is needed to troubleshoot.
Tip Know the approximate time of the problem.
Multiple calls may have occurred, so knowing the approximate time of the call helps TAC quickly locate the trouble. You can obtain phone statistics on a Cisco Unified IP Phone 79xx by pressing the i or? button twice during an active call. When you are running a test to reproduce the issue and produce information, know the following data that is crucial to understanding the issue:

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 20

Troubleshooting Tools

System History Log

· Calling number/called number · Any other number that is involved in the specific scenario · Time of the call
Note Remember that time synchronization of all equipment is important for troubleshooting.
If you are reproducing a problem, make sure to choose the file for the timeframe by looking at the modification date and the time stamps in the file. The best way to collect the right trace means that you reproduce a problem and then quickly locate the most recent file and copy it from the Unified Communications Manager server.
Tip Save the log files to prevent them from being overwritten.
Files will get overwritten after some time. The only way to know which file is being logged to is to choose View > Refresh on the menu bar and look at the dates and times on the files.
System History Log
This system history log provides a central location for getting a quick overview of the initial system install, system upgrades, Cisco option installations, and DRS backups and DRS restores, as well as switch version and reboot history. Related Topics
System History Log Overview, on page 21 System History Log Fields, on page 22 Accessing the System History Log, on page 23
System History Log Overview
The system history log exists as a simple ASCII file, system-history.log, and the data does not get maintained in the database. Because it does not get excessively large, the system history file does not get rotated. The system history log provides the following functions:
· Logs the initial software installation on a server. · Logs the success, failure, or cancellation of every software upgrade (Cisco option files and patches). · Logs every DRS backup and restore that is performed. · Logs every invocation of Switch Version that is issued through either the CLI or the GUI. · Logs every invocation of Restart and Shutdown that is issued through either the CLI or the GUI. · Logs every boot of the system. If not correlated with a restart or shutdown entry, the boot is the result
of a manual reboot, power cycle, or kernel panic.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 21

System History Log Fields

Troubleshooting Tools

· Maintains a single file that contains the system history, since initial installation or since feature availability. · Exists in the install folder. You can access the log from the CLI by using the file commands or from the
Real Time Monitoring Tool (RTMT).
System History Log Fields
The log displays a common header that contains information about the product name, product version, and kernel image; for example: ===================================== Product Name – Unified Communications Manager Product Version – 7.1.0.39000-9023 Kernel Image – 2.6.9-67.EL ===================================== Each system history log entry contains the following fields:
timestamp userid action description start/result The system history log fields can contain the following values:
· timestamp–Displays the local time and date on the server with the format mm/dd/yyyy hh:mm:ss. · userid–Displays the user name of the user who invokes the action. · action–Displays one of the following actions:
· Install · Windows Upgrade · Upgrade During Install · Upgrade · Cisco Option Install · Switch Version · System Restart · Shutdown · Boot · DRS Backup · DRS Restore
· description–Displays one of the following messages: · Version: Displays for the Basic Install, Windows Upgrade, Upgrade During Install, and Upgrade actions. · Cisco Option file name: Displays for the Cisco Option Install action.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 22

Troubleshooting Tools

Accessing the System History Log

· Timestamp: Displays for the DRS Backup and DRS Restore actions. · Active version to inactive version: Displays for the Switch Version action. · Active version: Displays for the System Restart, Shutdown, and Boot actions.
· result–Displays the following results: · Start · Success or Failure · Cancel
The following shows a sample of the system history log.
admin:file dump install system- history.log======================================= Product Name – Cisco Unified Communications Manager Product Version – 6.1.2.9901-117 Kernel Image – 2.4.21-47.EL.cs.3BOOT ======================================= 07/25/2008 14:20:06 | root: Install 6.1.2.9901-117 Start 07/25/2008 15:05:37 | root: Install 6.1.2.9901-117 Success 07/25/2008 15:05:38 | root: Boot 6.1.2.9901-117 Start 07/30/2008 10:08:56 | root: Upgrade 6.1.2.9901-126 Start 07/30/2008 10:46:31 | root: Upgrade 6.1.2.9901-126 Success 07/30/2008 10:46:43 | root: Switch Version 6.1.2.9901-117 to 6.1.2.9901-126 Start
07/30/2008 10:48:39 | root: Switch Version 6.1.2.9901-117 to 6.1.2.9901-126 Success
07/30/2008 10:48:39 | root: Restart 6.1.2.9901-126 Start 07/30/2008 10:51:27 | root: Boot 6.1.2.9901-126 Start 08/01/2008 16:29:31 | root: Restart 6.1.2.9901-126 Start 08/01/2008 16:32:31 | root: Boot 6.1.2.9901-126 Start
Accessing the System History Log
You can use either the CLI or RTMT to access the system history log.
Using the CLI You can access the system history log by using the CLI file command; for example:
· file view install system-history.log · file get install system-history.log
For more information on the CLI file commands, see the Command Line Interface Reference Guide for Cisco Unified Solutions.
Using RTMT You can also access the system history log by using RTMT. From the Trace and Log Central tab, choose Collect Install Logs.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 23

Audit Logging

Troubleshooting Tools

For more information about using RTMT, refer to the Cisco Unified Real-Time Monitoring Tool Administration Guide.
Audit Logging
Centralized audit logging ensures that configuration changes to the Unified Communications Manager system gets logged in separate log files for auditing. An audit event represents any event that is required to be logged. The following Unified Communications Manager components generate audit events:
· Cisco Unified Communications Manager Administration
· Cisco Unified Serviceability
· Unified Communications Manager CDR Analysis and Reporting
· Cisco Unified Real-Time Monitoring Tool
· Cisco Unified Communications Operating System
· Disaster Recovery System
· Database
· Command Line Interface
· Remote Support Account Enabled (CLI commands issued by technical supports teams)
In Cisco Business Edition 5000, the following Cisco Unity Connection components also generate audit events: · Cisco Unity Connection Administration
· Cisco Personal Communications Assistant (Cisco PCA)
· Cisco Unity Connection Serviceability
· Cisco Unity Connection clients that use the Representational State Transfer (REST) APIs
The following example displays a sample audit event:
CCM_TOMCAT-GENERIC-3-AuditEventGenerated: Audit Event Generated UserID:CCMAdministrator Client IP Address:172.19.240.207 Severity:3 EventType:ServiceStatusUpdated ResourceAccessed: CCMService EventStatus:Successful
Description: Call Manager Service status is stopped App ID:Cisco Tomcat Cluster ID:StandAloneCluster Node ID:sa-cm1-3
Audit logs, which contain information about audit events, get written in the common partition. The Log Partition Monitor (LPM) manages the purging of these audit logs as needed, similar to trace files. By default, the LPM purges the audit logs, but the audit user can change this setting from the Audit User Configuration window in Cisco Unified Serviceability. The LPM sends an alert whenever the common partition disk usage exceeds the threshold; however, the alert does not have the information about whether the disk is full because of audit logs or trace files.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 24

Troubleshooting Tools

Audit Logging

Tip The Cisco Audit Event Service, which is a network service that supports audit logging, displays in Control Center–Network Services in Cisco Unified Serviceability. If audit logs do not get written, then stop and start this service by choosing Tools > Control Center–Network Services in Cisco Unified Serviceability.
All audit logs get collected, viewed and deleted from Trace and Log Central in the Cisco Unified Real-Time Monitoring Tool. Access the audit logs in RTMT in Trace and Log Central. Go to System > Real-Time Trace > Audit Logs > Nodes. After you select the node, another window displays System > Cisco Audit Logs. The following types of audit logs display in RTMT:
· Application log
· Database log
· Operating system log
· Remote SupportAccEnabled log
Application Log
The application audit log, which displays in the AuditApp folder in RTMT, provides configuration changes for Cisco Unified Communications Manager Administration, Cisco Unified Serviceability, the CLI, Cisco Unified Real-Time Monitoring Tool (RTMT), Disaster Recovery System, and Cisco Unified CDR Analysis and Reporting (CAR). For Cisco Business Edition 5000, the application audit log also logs changes for Cisco Unity Connection Administration, Cisco Personal Communications Assistant (Cisco PCA), Cisco Unity Connection Serviceability, and clients that use the Representational State Transfer (REST) APIs. Although the Application Log stays enabled by default, you can configure it in Cisco Unified Serviceability by choosing Tools > Audit Log Configuration. For a description of the settings that you can configure for audit log configuration, see Cisco Unified Serviceability Administration Guide. If the audit logs get disabled in Cisco Unified Serviceability, no new audit log files get created.
Tip Only a user with an audit role has permission to change the Audit Log settings. By default, the CCMAdministrator has the audit role after fresh installs and upgrades. The CCMAdministrator can assign the “standard audit users” group to a new user that the CCMAdministrator specifically creates for audit purposes. The CCMAdministrator can then be removed from the audit user group. The “standard audit log configuration” role provides the ability to delete audit logs, read/update access to Cisco Unified Real-Time Monitoring Tool, Trace Collection Tool, RTMT Alert Configuration, the Control Center – Network Services window, RTMT Profile Saving, the Audit Configuration window, and a new resource called Audit Traces. For Cisco Unity Connection in Cisco Business Edition 5000, the application administration account that was created during installation has the Audit Administrator role and can assign other administrative users to the role.
Unified Communications Manager creates one application audit log file until the configured maximum file size is reached; then, it closes and creates a new application audit log file. If the system specifies rotating the log files, Unified Communications Manager saves the configured number of files. Some of the logging events can be viewed by using RTMT SyslogViewer. The following events get logged for Cisco Unified Communications Manager Administration:
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 25

Audit Logging

Troubleshooting Tools

· User logging (user logins and user logouts). · User role membership updates (user added, user deleted, user role updated). · Role updates (new roles added, deleted, or updated). · Device updates (phones and gateways). · Server configuration updates (changes to alarm or trace configurations, service parameters, enterprise
parameters, IP addresses, host names, Ethernet settings, and Unified Communications Manager server additions or deletions).
The following events get logged for Cisco Unified Serviceability: · Activation, deactivation, start, or stop of a service from any Serviceability window. · Changes in trace configurations and alarm configurations. · Changes in SNMP configurations. · Changes in CDR Management. · Review of any report in the Serviceability Reports Archive. View this log on the reporter node.
RTMT logs the following events with an audit event alarm: · Alert configuration. · Alert suspension. · E-mail configuration. · Set node alert status. · Alert addition. · Add alert action. · Clear alert. · Enable alert. · Remove alert action. · Remove alert.
The following events get logged for Unified Communications Manager CDR Analysis and Reporting: · Scheduling the CDR Loader. · Scheduling the daily, weekly, and monthly user reports, system reports, and device reports. · Mail parameters configurations. · Dial plan configurations. · Gateway configurations. · System preferences configurations. · Autopurge configurations.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 26

Troubleshooting Tools

Audit Logging

· Rating engine configurations for duration, time of day, and voice quality. · QoS configurations. · Automatic generation/alert of pregenerated reports configurations. · Notification limits configurations.
The following events gets logged for Disaster Recovery System: · Backup initiated successfully/failed · Restore initiated successfully/failed · Backup cancelled successfully · Backup completed successfully/failed · Restore completed successfully/failed · Save/update/delete/enable/disable of backup schedule · Save/update/delete of destination device for backup
For Cisco Business Edition 5000, Cisco Unity Connection Administration logs the following events: · User logging (user logins and user logouts). · All configuration changes, including but not limited to users, contacts, call management objects, networking, system settings, and telephony. · Task management (enabling or disabling a task). · Bulk Administration Tool (bulk creates, bulk deletes). · Custom Keypad Map (map updates)
For Cisco Business Edition 5000, Cisco PCA logs the following events: · User logging (user logins and user logouts). · All configuration changes made via the Messaging Assistant.
For Cisco Business Edition 5000, Cisco Unity Connection Serviceability logs the following events: · User logging (user logins and user logouts). · All configuration changes. · Activating, deactivating, starting or stopping services.
For Cisco Business Edition 5000, clients that use the REST APIs log the following events: · User logging (user API authentication). · API calls that utilize Cisco Unity Connection Provisioning Interface (CUPI).

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 27

Verify Cisco Unified Communications Manager Services Are Running

Troubleshooting Tools

Database Log The database audit log, which displays in the informix folder in RTMT, reports database changes. This log, which is not enabled by default, gets configured in Cisco Unified Serviceability by choosing Tools > Audit Log Configuration. For a description of the settings that you can configure for audit log configuration, see Cisco Unified Serviceability. This audit differs from the Application audit because it logs database changes, and the Application audit logs application configuration changes. The informix folder does not display in RTMT unless database auditing is enabled in Cisco Unified Serviceability.
Operating System Log The operating system audit log, which displays in the vos folder in RTMT, reports events that are triggered by the operating system. It does not get enabled by default. The utils auditd CLI command enables, disables, or gives status about the events. The vos folder does not display in RTMT unless the audit is enabled in the CLI. For information on the CLI, see Command Line Interface Reference Guide for Cisco Unified Solutions.
Remote Support Acct Enabled Log The Remote Support Acct Enabled audit log, which displays in the vos folder in RTMT, reports CLI commands that get issued by technical support teams. You cannot configure it, and the log gets created only if the Remote Support Acct gets enabled by the technical support team.
Verify Cisco Unified Communications Manager Services Are Running
Use the following procedure to verify which Cisco CallManager services are active on a server.
Procedure 1. From Cisco Unified Communications Manager Administration, choose Navigation > Cisco Unified
Serviceability.
2. Choose Tools > Service Activation.
3. From the Servers column, choose the desired server. The server that you choose displays next to the Current Server title, and a series of boxes with configured services displays. Activation Status column displays either Activated or Deactivated in the Cisco CallManager line. If the Activated status displays, the specified Cisco CallManager service remains active on the chosen server. If the Deactivated status displays, continue with the following steps.
4. Check the check box for the desired Cisco CallManager service.
5. Click the Update button.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 28

Troubleshooting Tools

Verify Cisco Unified Communications Manager Services Are Running

The Activation Status column displays Activated in the specified Cisco CallManager service line. The specified service now shows active for the chosen server.
Perform the following procedure if the Cisco CallManager service has been in activated and you want to verify if the service is currently running.
Procedure 1. From Cisco Unified Communications Manager Administration, choose Navigation > Cisco Unified
Serviceability. The Cisco Unified Serviceability window displays. 2. Choose Tools > Control Center ­ Feature Services. 3. From the Servers column, choose the server. The server that you chose displays next to the Current Server title, and a box with configured services displays. The Status column displays which services are running for the chosen server.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 29

Verify Cisco Unified Communications Manager Services Are Running

Troubleshooting Tools

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 30

3 C H A P T E R
Cisco Unified Communications Manager System Issues
This section covers solutions for the most common issues that relate to a Unified Communications Manager system.
· Cisco Unified Communications Manager System Not Responding, on page 31 · Database Replication, on page 37 · LDAP Authentication Fails, on page 43 · Issues with LDAP Over SSL, on page 44 · Open LDAP Cannot Verify the Certificate to Connect to the LDAP Server, on page 45 · JTAPI Subsystem Startup Problems, on page 46 · Security Issues, on page 50
Cisco Unified Communications Manager System Not Responding
This section covers issues related to a Unified Communications Manager system that is not responding. Related Topics
Cisco Unified Communications Manager System Stops Responding, on page 32 Cisco Unified Communications Manager Administration Does Not Display, on page 33 Error When Attempting to Access Cisco Unified Communications Manager Administration, on page 33 Error When Attempting to Access Cisco Unified Communications Manager Administration on a Subsequent Node, on page 33 You Are Not Authorized to View, on page 34 Problems Displaying or Adding Users with Cisco Unified Communications Manager, on page 34 Name to Address Resolution Failing, on page 35 Port 80 Blocked Between Your Browser and the Cisco Unified Communications Manager Server, on page 36 Improper Network Setting Exists in the Remote Machine, on page 36 Slow Server Response
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 31

Cisco Unified Communications Manager System Stops Responding

Cisco Unified Communications Manager System Issues

Cisco Unified Communications Manager System Stops Responding
Symptom The Unified Communications Managersystem does not respond. When the Cisco CallManager service stops responding, the following message displays in the System Event log:
The Cisco CallManager service terminated unexpectedly. It has done this 1 time. The following corrective action will be taken in 60000 ms. Restart the service.
Other messages you may see in this situation:
Timeout 3000 milliseconds waiting for Cisco CallManager service to connect.
The Cisco Communications Manager failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.
At this time, when devices such as the Cisco Unified IP Phones and gateways unregister from the Unified Communications Manager, users receive delayed dial tone, and/or the Unified Communications Managerserver freezes due to high CPU usage. For event log messages that are not included here, view the Unified Communications Manager Event Logs.
Possible Cause The Cisco CallManager service can stop responding because the service does not have enough resources such as CPU or memory to function. Generally, the CPU utilization in the server is 100 percent at that time.
Recommended Action Depending on what type of interruption you experience, you will need to gather different data that will help determine the root cause of the interruption. Use the following procedure if a lack of resources interruption occurs.
Procedure 1. Collect Cisco CallManager traces 15 minutes before and after the interruption. 2. Collect SDL traces 15 minutes before and after the interruption. 3. Collect perfmon traces if available. 4. If the traces are not available, start collecting the perfmon traces and track memory and CPU usage for
each process that is running on the server. These will help in the event of another lack of resources interruption.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 32

Cisco Unified Communications Manager System Issues

Cisco Unified Communications Manager Administration Does Not Display

Cisco Unified Communications Manager Administration Does Not Display
Symptom Cisco Unified Communications Manager Administration does not display.
Possible Cause The Cisco CallManager service stopped.
Recommended Action Verify that the Cisco CallManager service is active and running on the server. See related topics or the Cisco Unified Serviceability Administration Guide. Related Topics
Verify Cisco Unified Communications Manager Services Are Running, on page 28
Error When Attempting to Access Cisco Unified Communications Manager Administration
Symptom An error message displays when you are trying to access Unified Communications Manager.
Possible Cause The services did not start automatically as expected. One of the services stopping represents the most frequent reason for Cisco Unified Communications Manager Administration not displaying.
Recommended Action Try starting the other services.
Error When Attempting to Access Cisco Unified Communications Manager Administration on a Subsequent Node
Symptom An error message displays when you are trying to access the Unified Communications Manager Administration.
Possible Cause If the IP address of the first Unified Communications Manager node gets changed while a subsequent node is offline, you may not be able to log in to Cisco Unified Communications Manager Administration on the subsequent node.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 33

You Are Not Authorized to View

Cisco Unified Communications Manager System Issues

Recommended Action If this occurs, follow the procedure for changing the IP address on a subsequent Unified Communications Manager node in the document, Changing the IP Address and Host Name for Unified Communications Manager.
You Are Not Authorized to View
Symptom When you access Unified Communications Manager Administration, one of the following messages displays.
· You Are Not Authorized to View This Page · You do not have permission to view this directory or page using the credentials you supplied. · Server Application Error. The server has encountered an error while loading an application during the
processing of your request. Please refer to the event log for more detailed information. Please contact the server administrator for assistance. · Error: Access is Denied.
Possible Cause Unknown
Recommended Action Contact TAC for further assistance.
Problems Displaying or Adding Users with Cisco Unified Communications Manager
Symptom You cannot add a user or conduct a search in Unified Communications Manager.
Possible Cause You may encounter the following problems if you are working with Unified Communications Manager that is installed on a server that has a special character (such as an underscore) in its hostname or Microsoft Internet Explorer 5.5 with SP2 and a Q313675 patch or above.
· When you conduct a basic search and click submit, the same page redisplays. · When you try to insert a new user, the following message displays.
The following error occurred while trying to execute the command.Sorry, your session object has timed out.
Click here to Begin a New Search

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 34

Cisco Unified Communications Manager System Issues

Name to Address Resolution Failing

Recommended Action You may not be able to add a user or do a search on Unified Communications Manager Administration, if your Unified Communications Manager hostname contains any special characters such as underscore or period (for example, Call_Manager). Domain Name System (DNS)-supported characters include all letters (A-Z, a-z), numbers (0-9), and hyphen (-); any special characters are not allowed. If the Q313675 patch is installed on your browser, make sure that the URL does not contain any non-DNS supported characters. For more information about the Q313675 patch, refer to MS01-058: File Vulnerability Patch for Internet Explorer 5.5 and Internet Explorer 6. To resolve this problem, you have the following options:
· Access Cisco Unified Communications Manager Administration by using the IP address of the server. · Do not use non-DNS characters in the Server Name. · Use the localhost or IP address in the URL.
Name to Address Resolution Failing
Symptom One of the following messages displays when you try to access the following URL: http://your-cm-server-name/ccmadmin
· Internet Explorer–This page cannot be displayed · Netscape–Not Found. The requested URL /ccmadmin was not found on this server.
If you try to access the same URL by using the Cisco Communications Manager IP address (http://10.48.23.2/ccmadmin) instead of the name, the window displays.
Possible Cause The name that you entered as “your-cm-server-name” maps to the wrong IP address in DNS or hosts file.
Recommended Action If you have configured the use of DNS, check in the DNS to see whether the entry for the your-cm-server-name has the correct IP address of the Unified Communications Manager server. If it is not correct, change it. If you are not using DNS, your local machine will check in the “hosts” file to see whether an entry exists for the your-cm-server-name and an IP address that is associated to it. Open the file and add the Unified Communications Manager server name and the IP address. You can find the “hosts” file at C:WINNTsystem32driversetchosts.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 35

Cisco Unified Communications Manager System Issues Port 80 Blocked Between Your Browser and the Cisco Unified Communications Manager Server
Port 80 Blocked Between Your Browser and the Cisco Unified Communications Manager Server
Symptom One of the following messages displays when a firewall blocks the port that is used by the web server or the http traffic:
· Internet Explorer–This page cannot be displayed · Netscape–There was no response. The server could be down or is not responding
Possible Cause For security reasons, the system blocked the http access from your local network to the server network.
Recommended Action 1. Verify whether other types of traffic to the Unified Communications Manager server, such as ping or
Telnet, are allowed. If any are successful, it will show that http access to the Unified Communications Manager web server has been blocked from your remote network. 2. Check the security policies with your network administrator. 3. Try again from the same network where the server is located.
Improper Network Setting Exists in the Remote Machine
Symptom No connectivity exists, or no connectivity exists to other devices in the same network as the Unified Communications Manager. When you attempt the same action from other remote machines, Unified Communications Manager Administration displays.
Possible Cause Improper network configuration settings on a station or on the default gateway can cause a web page not to display because partial or no connectivity to that network exists.
Recommended Action 1. Try pinging the IP address of the Unified Communications Manager server and other devices to confirm
that you cannot connect. 2. If the connectivity to any other device out of your local network is failing, check the network setting on
your station, as well as the cable and connector integrity. Refer to the appropriate hardware documentation for detailed information. If you are using TCP-IP over a LAN to connect, continue with the following steps to verify the network settings on the remote station.
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 36

Cisco Unified Communications Manager System Issues

Manage Impact of Cisco RAID Operations

3. Choose Start > Setting > Network and Dial-up connections. 4. Choose Local Area Connection, then Properties.
The list of communication protocols displays as checked. 5. Choose Internet Protocol (TCP-IP) and click Properties again. 6. Depending on your network, choose either Obtain an ip address automatically or set manually your
address, mask and default Gateway. The possibility exists that a browser- specific setting could be improperly configured. 7. Choose the Internet Explorer browser Tools > Internet Options. 8. Choose the Connections tab and then verify the LAN settings or the dial-up settings. By default, the LAN settings and the dial-up settings do not get configured. The generic network setting from Windows gets used. 9. If the connectivity is failing only to the Unified Communications Manager network, a routing issue probably exists in the network. Contact the network administrator to verify the routing that is configured in your default gateway.
Note If you cannot browse from the remote server after following this procedure, contact TAC to have the issue investigated in more detail.
Manage Impact of Cisco RAID Operations
Cisco Redundant Array of Independent Disks (RAID) Controller conducts background operations such as Consistency Check (CC), Background Initialization (BGI), Rebuild (RBLD), Volume Expansion & Reconstruction (RLM) and Patrol Real (PR). These background operations are expected to limit their impact to I/O operations. However, there have been cases of higher impact during some of the operations like Format or similar input output operations. In these cases, both the I/O operation and the background operations may consume large amount of CPU resources. It is recommended that CC and Patrol Read jobs are scheduled when the load is relatively less. If there are CallManager servers where huge load is running at the same time, it is recommend that you limit possible concurrent background operations and other intensive I/O operations of CallManager.
Database Replication
This section covers database replication issues for aUnified Communications Manager system. Related Topics
Replication Fails Between the Publisher and the Subscriber Server, on page 38 Database Replication Does Not Occur When Connectivity Is Restored on Lost Node, on page 41 Database Tables Out of Sync Do Not Trigger Alert, on page 41 Resetting Database Replication When You Are Reverting to an Older Product Release, on page 42

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 37

Replication Fails Between the Publisher and the Subscriber Server

Cisco Unified Communications Manager System Issues

Replication Fails Between the Publisher and the Subscriber Server
Replicating the database represents a core function of Unified Communications Manager clusters. The server with the master copy of the database acts as the publisher (first node), while the servers that replicate the database comprise subscribers (subsequent nodes).
Tip Before you install Unified Communications Manager on the subscriber server, you must add the subscriber to the Server Configuration window in Cisco Unified Communications Manager Administration to ensure that the subscriber replicates the database that exists on the publisher database server. After you add the subscriber server to the Server Configuration window and then install Unified Communications Manager on the subscriber, the subscriber receives a copy of the database that exists on the publisher server.
Symptom Changes that are made on the publisher server do not get reflected on phones that are registered with the subscriber server.
Possible Cause Replication fails between the publisher and subscriber servers.
Recommended Action Verify and, if necessary, repair database replication, as described in the following procedure:
Procedure 1. Verify database replication. You can use the CLI, Cisco Unified Reporting , or RTMT to verify database
replication. · To verify by using the CLI, see 2, on page 38 . · To verify by using Cisco Unified Reporting, see 3, on page 39 . · To verify by using RTMT, see 4, on page 39 .
2. To verify database replication by using the CLI, access the CLI and issue the following command to check replication on each node. You will need to run this CLI command on each node to check its replication status. Also, after a subscriber is installed, depending on the number of subscribers, it may take a considerable amount of time to archive a status of 2.
admin:
show perf query class “Number of Replicates Created and State of Replication”
==>query class: – Perf class (Number of Replicates Created and State of Replication) has instances and values: ReplicateCount -> Number of Replicates Created = 344 ReplicateCount -> Replicate_State = 2
Be aware that the Replicate_State object shows a value of 2 in this case. The following list shows the possible values for Replicate_State:

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 38

Cisco Unified Communications Manager System Issues

Replication Fails Between the Publisher and the Subscriber Server

· 0–This value indicates that replication did not start. Either no subsequent nodes (subscribers) exist, or the Cisco Database Layer Monitor service is not running and has not been running since the subscriber was installed.
· 1–This value indicates that replicates have been created, but their count is incorrect.
· 2–This value indicates that replication is good.
· 3–This value indicates that replication is bad in the cluster.
· 4–This value indicates that replication setup did not succeed.
3. To verify database replication by using Cisco Unified Reporting, perform the following tasks. a. From the Navigation drop-down list box in the upper, right corner in Cisco Unified Communications Manager Administration, choose Cisco Unified Reporting .
b. After Cisco Unified Reporting displays, click System Reports .
c. Generate and view the Unified CM Database Status report, which provides debugging information for database replication. Once you have generated the report, open it and look at the Unified CM Database Status . It gives the RTMT replication counters for all servers in the cluster. All servers should have a replicate state of 2, and all servers should have the same number of replicates created. If you see any servers whose replicate states are not equal to 2 in the above status check, inspect the “Replication Server List” on this report. It shows which servers are connected and communicating with each node. Each server should show itself as local (in its list) and the other servers as active connected. If you see any servers as dropped, it usually means there is a communication problem between the nodes.
d. If you want to do so, generate and view the Unified CM Database Status report, which provides a snapshot of the health of the Unified Communications Manager database.
4. To verify database replication by using RTMT, perform the following tasks: a. Open the Cisco Unified Real-Time Monitoring Tool (RTMT).
b. Click the CallManager tab.
c. Click Database Summary . The Replication Status pane displays.
The following list shows the possible values for the Replication Status pane: · 0–This value indicates that replication has not started. Either no subsequent nodes (subscribers) exist, or the Cisco Database Layer Monitor service is not running and has not been running since the subscriber was installed.
· 1–This value indicates that replicates have been created, but their count is incorrect.
· 2–This value indicates that replication is good.
· 3–This value indicates that replication is bad in the cluster.
· 4–This value indicates that replication setup did not succeed.
· To view the Replicate_State performance monitoring counter, choose System > Performance > Open Performance Monitoring . Double-click the publisher database server (first node) to expand

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 39

Replication Fails Between the Publisher and the Subscriber Server

Cisco Unified Communications Manager System Issues

the performance monitors. Click Number of Replicates Created and State of Replication . Double-click Replicate_State . Click ReplicateCount from the Object Instances window and click Add .
Tip To view the definition of the counter, right click the counter name and choose Counter Description .
5. If all the servers have a good RTMT status, but you suspect the databases are not in sync, you can run the CLI command utils dbreplication status (If any of the servers showed an RTMT status of 4, proceed to Step 6 ) This status command can be run on all servers by using utils dbreplication status all or on one subscriber by using utils dbreplication status The status report will tell you if any tables are suspect. If there are suspect tables, you will want to do a replication repair CLI command to sync the data from the publisher server to the subscriber servers. The replication repair can be done on all subscriber servers (using the all parameter) or on just one subscriber server by using the following: utils dbreplication repair usage:utils dbreplication repair [nodename]|all After running the replication repair, which can take several minutes, you can run another status command to verify that all tables are now in sync. If tables are in sync after running the repair, you are successful in fixing replication.
Note Only do Step 6 if one of the servers showed an RTMT status of 4, or had a status of 0 for more than four hours.
6. Generate and view the Unified CM Database Status report, which provides debugging information for database replication. For each subscriber server that has a bad RTMT status, check that the hosts, rhosts, sqlhosts, and services files have the appropriate information. Generate and view the Unified CM Cluster Overview report. Verify that the subscriber servers have the same version, verify that connectivity is good, and verify that time delay is within tolerances. If the preceding conditions are acceptable, do the following to reset replication on that subscriber server: a. At the subscriber server, perform the CLI command utils dbreplication stop Do this for all subscriber servers that have an RTMT value of 4 b. At the publisher server, perform the CLI command utils dbreplication stop c. At the publisher server, perform the CLI command utils dbreplication reset where is the hostname of the subscriber server that needs to be reset. If all subscriber servers need to be reset, use command utils dbreplication reset all

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 40

Cisco Unified Communications Manager System Issues

Database Replication Does Not Occur When Connectivity Is Restored on Lost Node

For More Information Cisco Unified Real-Time Monitoring Tool Administration Guide Cisco Unified Reporting Administration Guide Command Line Interface Reference Guide for Cisco Unified Solutions
Database Replication Does Not Occur When Connectivity Is Restored on Lost Node
Symptom Database replication does not occur when connectivity is restored on lost node recovery. See the related topics for methods to verify the state of replication if replication fails. Only use the following procedure if you have already tried to reset replication on the node, and have been unsuccessful.
Possible Cause The CDR check remains stuck in a loop, due to a delete on device table.
Recommended Action 1. Run utils dbreplication stop on the affected subscribers. You can run them all at once. 2. Wait until step 1 completes, then run utils dbreplication stop on the affected publisher server. 3. Run utils dbreplication clusterreset from the affected publisher server. When you run the command,
the log name gets listed in the log file. Watch this file to monitor the process status. The path to the follows: /var/log/active/cm/trace/dbl/sdi 4. From the affected publisher, run utils dbreplication reset all. 5. Stop and restart all the services on all the subscriber servers [or restart/reboot all the systems (subscriber servers)] in the cluster to get the service changes. Do this only after utils dbreplication status shows Status 2.
Related Topics Replication Fails Between the Publisher and the Subscriber Server, on page 38
Database Tables Out of Sync Do Not Trigger Alert
Note “Out of sync” means that two servers in the cluster do not contain the same information in a specific database table.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 41

Resetting Database Replication When You Are Reverting to an Older Product Release

Cisco Unified Communications Manager System Issues

Symptom On Unified Communications Manager Version 6.x or later, the symptoms include unexpected call processing behaviors. Calls do not get routed or handled as expected. The symptoms may occur on either the publisher or on the subscriber servers. On Unified Communications Manager Version 5.x, the symptoms include unexpected call processing behaviors. Calls do not get routed or handled as expected but only when the publisher server is offline. If you see this symptom and you run utils dbrepication status at the CLI, it reports Out of sync. If Out of sync does not display, be aware that this is not the problem.
Possible Cause Database tables remain out of sync between nodes. Replication alerts only indicate failure in the replication process and do not indicate when database tables are out of sync. Normally, if replication is working, tables should remain in sync. Instances can occur in which replication appears to be working, but database tables are “Out of sync”.
Recommended Action 1. Reset cluster replication by using CLI commands. Ensure servers in the cluster are online with full IP
connectivity for this to work. Confirm that all servers in the cluster are online by using platform CLIs and Cisco Unified Reporting.
2. If the servers are in Replication State 2, run the following command on the publisher server:
3. utils dbreplication repair server name
4. If the servers are not in Replication State 2,
5. run the following command on all subscriber servers:
6. utils dbreplication stop
7. Then, run the following commands on the publisher server:
8. utils dbreplication stop
9. then
10. utils dbreplication reset all
Resetting Database Replication When You Are Reverting to an Older Product Release
If you revert the servers in a cluster to run an older product release, you must manually reset database replication within the cluster. To reset database replication after you revert all the cluster servers to the older product release, enter the CLI command utils dbreplication reset all on the publisher server. When you switch versions by using Cisco Unified Communications Operating System Administration or the CLI, you get a message reminding you about the requirement to reset database replication if you are reverting to an older product release.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 42

Cisco Unified Communications Manager System Issues

utils dbreplication clusterreset

utils dbreplication clusterreset
This command resets database replication on an entire cluster.

Usage Guidelines

Command Syntax utils dbreplication clusterreset
Before you run this command, run the command utils dbreplication stop first on all subscribers servers, and then on the publisher server.

Requirements Command privilege level: 0 Allowed during upgrade: Yes
utils dbreplication dropadmindb
This command drops the Informix syscdr database on any server in the cluster.

Usage Guidelines

Command Syntax utils dbreplication dropadmindb
You should run this command only if database replication reset or cluster reset fails and replication cannot be restarted.

Requirements Command privilege level: 0 Allowed during upgrade: Yes

LDAP Authentication Fails
This section describes a common issue when LDAP authentication failure occurs.
Symptom Login fails for end users. Authentication times out before the user can log in.
Possible Cause You misconfigured the LDAP Port in the LDAP Authentication window in Cisco Unified Communications Manager Administration.
Recommended Action How your corporate directory is configured determines which port number to enter in the LDAP Port field. For example, before you configure the LDAP Port field, determine whether your LDAP server acts as a Global Catalog server and whether your configuration requires LDAP over SSL. Consider entering one of the following port numbers:

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 43

Issues with LDAP Over SSL

Cisco Unified Communications Manager System Issues

Example: LDAP Port For When the LDAP Server Is Not a Global Catalog Server · 389–When SSL is not required. (This port number specifies the default that displays in the LDAP Port field.) · 636–When SSL is required. (If you enter this port number, make sure that you check the Use SSL check box.)
Example: LDAP Port For When the LDAP Server Is a Global Catalog Server · 3268–When SSL is not required. · 3269–When SSL is required. (If you enter this port number, make sure that you check the Use SSL check box.)
Tip Your configuration may require that you enter a different port number than the options that are listed in the preceding bullets. Before you configure the LDAP Port field, contact the administrator of your directory server to determine the correct port number to enter.
Issues with LDAP Over SSL
This section describes a common issue when you use LDAP over SSL.
Symptom LDAP over SSL does not work.
Possible Cause In most cases, problems with LDAP over SSL involve invalid, wrong, or incomplete certificates (chains) on the Unified Communications Manager server.
Explanation In some cases, you may use multiple certificates for SSL. In most cases, uploading the AD root certificate as a directory trust is the only certificate that you need to make LDAP over SSL work. However, if a different directory trust certificate is uploaded, that is, one other than a root certificate, that other certificate must be verified to a higher level certificate, such as a root certificate. In this case, a certificate chain is created because more than one extra certificate is involved. For example, you may have the following certificates in your certificate chain:
· Root Certificate–The top-level CA certificate in the trust chain which will have similar issuer and the subject name.
· Intermediate Certificate–The CA certificate that is part of the trust chain (other than the top level). This follows the hierarchy starting from root till the last intermediate.
· Leaf Certificate–The certificate issued to the service/server which is signed by the immediate intermediate.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 44

Cisco Unified Communications Manager System Issues

Open LDAP Cannot Verify the Certificate to Connect to the LDAP Server

For example, your company has two certificates and a root certificate in your certificate chain. The following example shows the contents of a certificate: Data:
Version: 3 (0x2) Serial Number: · 77:a2:0f:36:7c:07:12:9c:41:a0:84:5f:c3:0c:64:64
Signature Algorithm: sha1WithRSAEncryption Issuer: DC=com, DC=DOMAIN3, CN=jim Validity · Not Before: Apr 13 14:17:51 2009 GMT · Not After: Apr 13 14:26:17 2014 GMT
Subject: DC=com, DC=DOMAIN3, CN=jim
Recommended Action If you have a two node chain, the chain contains the root and leaf certificate. In this case, uploading the root certificate to the directory trust is all you need to do. If you have more than a two node chain, the chain contains the root, leaf, and intermediate certificates. In this case, the root certificate and all the intermediate certificates, excluding the leaf certificate, needs to be uploaded to the directory trust. At the highest level in the certificate chain, that is, for the root certificate, check to make sure that the Issuer field matches the Subject field. If the Issuer field and Subject field do not match, the certificate is not a root certificate; it is an intermediate certificate. In this case, identify the complete chain from root to the last intermediate certificate, and upload the complete chain to the directory trust store. In addition, check the Validity field to ensure the certificate has not expired. If the intermediate is expired, get the new chain from the certificate authority, along with the new leaf that is signed by using the new chain. If only the leaf certificate is expired, get a new signed certificate.
Open LDAP Cannot Verify the Certificate to Connect to the LDAP Server
Symptom End user authentication via CTI/JTAPI clients fails, but user authentication to Unified CM works.
Possible Cause Open LDAP cannot verify the certificate to connect to the LDAP server.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 45

JTAPI Subsystem Startup Problems

Cisco Unified Communications Manager System Issues

Explanation Certificates are issued with a Fully Qualified Domain Name (FQDN). The Open LDAP verification process matches the FQDN with the server that is being accessed. Because the uploaded certificate uses FQDN and the web form is using IP Address, Open LDAP cannot connect to the server.
Recommended Action · If possible, use DNS. During the Certificate Signing Request (CSR) process, ensure that you provide the FQDN as part of subject CN. Using this CSR when a self signed certificate or CA certificate is obtained, the Common Name will contain the same FQDN. Hence, no issues should occur when LDAP authentication is enabled for applications, such as CTI, CTL, and so on, with the trust certificate imported to the directory-trust.
· If you are not using DNS, enter an IP Address in the LDAP Authentication Configuration window in Unified Communications Manager Administration. Then, add the following line of text in /etc/openldap/ldap.conf: TLS_REQCERT never
You must have a remote account to update the file, which prevents the Open LDAP library from verifying that certificate from the server. However, subsequent communication still occurs over SSL.
JTAPI Subsystem Startup Problems
The JTAPI (Java Telephony API) subsystem represents a very important component of the Cisco Customer Response Solutions (CRS) platform. JTAPI communicates with the Unified Communications Manager and has responsibility for telephony call control. The CRS platform hosts telephony applications, such as Cisco Unified Auto-Attendant, Cisco IP ICD, and Cisco Unified IP-IVR. Although this section is not specific to any of these applications, keep in mind that the JTAPI subsystem is an underlying component that all of them use. Before starting the troubleshooting process, ensure that the software versions that you are using are compatible. To verify compatibility, read the Cisco Unified Communications Manager Release Notes for the version of Unified Communications Manager that you are using. To check the version of CRS, log in to AppAdmin by entering http://servername/appadmin, where servername specifies the name of the server on which CRS is installed. Find the current version in the lower- right corner of the main menu.
JTAPI Subsystem is OUT_OF_SERVICE
Symptom The JTAPI subsystem does not start.
Possible Cause One of the following exceptions displays in the trace file:
· MIVR-SS_TEL-4-ModuleRunTimeFailure

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 46

Cisco Unified Communications Manager System Issues

MIVR-SS_TEL-4-ModuleRunTimeFailure

· MIVR-SS_TEL-1-ModuleRunTimeFailure
Related Topics MIVR-SS_TEL-4-ModuleRunTimeFailure, on page 47 MIVR- SS_TEL-1-ModuleRunTimeFailure, on page 49
MIVR-SS_TEL-4-ModuleRunTimeFailure
Search for the MIVR-SS_TEL-1-ModuleRunTimeFailure string in the trace file. At the end of the line, an exception reason displays. The following list gives the most common errors: Related Topics
Unable to Create Provider­Bad Login or Password, on page 47 Unable to Create Provider-Connection Refused, on page 47 Unable to Create Provider­Login= , on page 48 Unable to Create Provider­Hostname, on page 48 Unable to Create Provider­Operation Timed Out , on page 49 Unable to Create Provider­Null , on page 49
Unable to Create Provider­Bad Login or Password
Possible Cause Administrator entered an incorrect user name or password in the JTAPI configuration.
Full Text of Error Message
%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-timefailure in JTAPI subsystem: Module=JTAPI Subsystem,Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — bad login or password. %MIVR-SS_TEL- 7EXCEPTION:com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — bad login or password.
Recommended Action Verify that the user name and password are correct. Try logging into the Unified CM User window (http://servername/ccmuser) on the Unified CM to ensure that the Unified CM cannot authenticate correctly.
Unable to Create Provider-Connection Refused
Possible Cause The Unified Communications Manager refused the JTAPI connection to the Unified Communications Manager.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 47

Unable to Create Provider­Login=

Cisco Unified Communications Manager System Issues

Full Text of Error Message
%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-timefailure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — Connection refused %MIVR- SS_TEL-7-EXCEPTION:com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — Connection refused
Recommended Action Verify that the CTI Manager service is running in the Cisco Unified Serviceability Control Center.
Unable to Create Provider­Login=
Possible Cause Nothing has been configured in the JTAPI configuration window.
Full Text of Error Message
%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-timefailure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — login= %MIVR-SS_TEL-7-EXCEPTION:com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — login=
Recommended Action Configure a JTAPI provider in the JTAPI configuration window on the CRS server.
Unable to Create Provider­Hostname
Possible Cause The CRS engine cannot resolve the host name of the Unified Communications Manager.
Full Text of Error Message
%M%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-timefailure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — dgrant-mcs7835.cisco.com %MIVR- SS_TEL-7-EXCEPTION:com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — dgrant-mcs7835.cisco.com
Recommended Action Verify that DNS resolution is working correctly from the CRS engine. Try using an IP address instead of the DNS name.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 48

Cisco Unified Communications Manager System Issues

Unable to Create Provider­Operation Timed Out

Unable to Create Provider­Operation Timed Out
Possible Cause The CRS engine does not have IP connectivity with the Unified Communications Manager.
Full Text of Error Message
101: Mar 24 11:37:42.153 PST%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — Operation timed out 102: Mar 24 11:37:42.168 PST %MIVR-SS_TEL-7-EXCEPTION: com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — Operation timed out
Recommended Action Check the IP address that is configured for the JTAPI provider on the CRS server. Check the default gateway configuration on the CRS server and the Unified Communications Manager. Make sure no IP routing problems exist. Test connectivity by pinging the Unified Communications Manager from the CRS server.
Unable to Create Provider­Null
Possible Cause No JTAPI provider IP address or host name get configured, or the JTAPI client is not using the correct version.
Full Text of Error Message
%MIVR-SS_TEL-4-ModuleRunTimeFailure:Real-timefailure in JTAPI subsystem: Module=JTAPI Subsystem, Failure Cause=7,Failure Module=JTAPI_PROVIDER_INIT, Exception=com.cisco.jtapi.PlatformExceptionImpl: Unable to create provider — null
Recommended Action Verify that a host name or IP address is configured in the JTAPI configuration. If the JTAPI version is incorrect, download the JTAPI client from the Unified Communications Manager Plugins window and install it on the CRS server.
MIVR-SS_TEL-1-ModuleRunTimeFailure
Symptom This exception usually occurs when the JTAPI subsystem cannot initialize any ports.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 49

JTAPI Subsystem is in PARTIAL_SERVICE

Cisco Unified Communications Manager System Issues

Possible Cause The CRS server can communicate with the Unified Communications Manager, but cannot initialize any CTI ports or CTI route points through JTAPI. This error occurs if the CTI ports and CTI route points are not associated with the JTAPI user.
Full Text of Error Message
255: Mar 23 10:05:35.271 PST%MIVR-SS_TEL-1-ModuleRunTimeFailure:Real-time failure in JTAPI subsystem: Module=JTAPI Subsystem,
Failure Cause=7,Failure Module=JTAPI_SS,Exception=null
Recommended Action Check the JTAPI user on the Unified Communications Manager and verify that CTI ports and CTI route points that are configured on the CRS server associate with the user.
JTAPI Subsystem is in PARTIAL_SERVICE
Symptom The following exception displays in the trace file: MIVR- SS_TEL-3-UNABLE_REGISTER_CTIPORT
Possible Cause The JTAPI subsystem cannot initialize one or more CTI ports or route points.
Full Text of Error Message
1683: Mar 24 11:27:51.716 PST%MIVR-SS_TEL-3-UNABLE_REGISTER_CTIPORT: Unable to register CTI Port: CTI Port=4503, Exception=com.cisco.jtapi.InvalidArgumentExceptionImpl: Address 4503 is not in provider’s domain. 1684: Mar 24 11:27:51.716 PST%MIVR-SS_TEL-7-EXCEPTION: com.cisco.jtapi.InvalidArgumentExceptionImpl: Address 4503 is not in provider’s domain.
Recommended Action The message in the trace tells you which CTI port or route point cannot be initialized. Verify that this device exists in the Unified Communications Manager configuration and also associates with the JTAPI user on the Unified Communications Manager.
Security Issues
This section provides information about security-related measurements and general guidelines for troubleshooting security-related problems.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 50

Cisco Unified Communications Manager System Issues

Security Alarms

Note This section does not describe how to reset the Cisco Unified IP Phone if it has been corrupted by bad loads, security bugs, and so on. For information on resetting the phone, refer to the Cisco Unified IP Phone Administration Guide for Cisco Unified Communications Manager that matches the model of the phone.
For information about how to delete the CTL file from Cisco Unified IP Phone models 7960, and 7940 only, see the System Configuration Guide for Cisco Unified Communications Manager or the Cisco Unified IP Phone Administration Guide for Cisco Unified Communications Manager that matches the model of the phone.
Related Topics Security Alarms, on page 51 Security Performance Monitor Counters, on page 51 Reviewing Security Log and Trace Files, on page 52 Troubleshooting Certificates, on page 53 Troubleshooting CTL Security Tokens, on page 54 Troubleshooting CAPF, on page 56 Troubleshooting Encryption for Phones and Cisco IOS MGCP Gateways, on page 57
Security Alarms
Cisco Unified Serviceability generates security-related alarms for X.509 name mismatches, authentication errors, and encryption errors. Cisco Unified Serviceability provides the alarm definitions.
Alarms may get generated on the phone for TFTP server and CTL file errors. For alarms that get generated on the phone, refer to the Cisco Unified IP Phone Administration Guide for Cisco Unified Communications Manager for your phone model and type (SCCP or SIP).
Security Performance Monitor Counters
Performance monitor counters monitor the number of authenticated phones that register with Unified Communications Manager, the number of authenticated calls that are completed, and the number of authenticated calls that are active at any time. The following table lists the performance counters that apply to security features.

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 51

Reviewing Security Log and Trace Files

Cisco Unified Communications Manager System Issues

Table 5: Security Performance Counters

Object Unified Communications Manager
SIP Stack TFTP Server

Counters
AuthenticatedCallsActive AuthenticatedCallsCompleted AuthenticatedPartiallyRegisteredPhone AuthenticatedRegisteredPhones EncryptedCallsActive EncryptedCallsCompleted EncryptedPartiallyRegisteredPhones EncryptedRegisteredPhones SIPLineServerAuthorizationChallenges SIPLineServerAuthorizationFailures SIPTrunkServerAuthenticationChallenges SIPTrunkServerAuthenticationFailures SIPTrunkApplicationAuthorization SIPTrunkApplicationAuthorizationFailures TLSConnectedSIPTrunk
StatusCodes4xxIns StatusCodes4xxOuts For example: 401 Unauthorized (HTTP authentication required) 403 Forbidden 405 Method Not Allowed 407 Proxy Authentication Required
BuildSignCount EncryptCount

Refer to the Cisco Unified Real-Time Monitoring Tool Administration Guide for accessing performance monitors in RTMT, configuring perfmon logs, and for more details about counters.
The CLI command show perf displays performance monitoring information. For information about using the CLI interface, refer to the Command Line Interface Reference Guide for Cisco Unified Solutions.

Reviewing Security Log and Trace Files
Unified Communications Manager stores log and trace files in multiple directories (cm/log, cm/trace, tomcat/logs, tomcat/logs/security, and so on).

Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 52

Cisco Unified Communications Manager System Issues

Troubleshooting Certificates

Note For devices that support encryption, the SRTP keying material does not display in the trace file.
You can use the trace collection feature of Cisco Unified Real-Time Monitoring Tool or CLI commands to find, view, and manipulate log and trace files.
Troubleshooting Certificates
The certificate management tool in Cisco Unified Communications Platform Administration allows you to display certificates, delete and regenerate certificates, monitor certificate expirations, and download and upload certificates and CTL files (for example, to upload updated CTL files to Unity). The CLI allows you to list and view self-signed and trusted certificates and to regenerate self-signed certificates. The CLI commands show cert, show web-security, set cert regen, and set web-security allow you to manage certificates at the CLI interface; for example, set cert regen tomcat. For information about how to use the GUI or CLI to manage certificates, refer to Administration Guide for Cisco Unified Communications Manager and the Command Line Interface Reference Guide for Cisco Unified Solutions.
Troubleshooting Ciphers
The Cipher Management page has no default values. Instead, the Cipher Management feature takes effect only when you configure Ciphers. For information about Ciphers, see Security Guide for Cisco Unified Communications Manager This section provides information to help you troubleshoot problems with Unified Communications Manager Ciphers:
Troubleshooting DRS and CDR Functionality
Symptom Breakage to DRS and CDR functionality.
Possible Cause Configuring hmac-sha2-512 in SSH MAC interface affects the DRS and CDR functionality. Configuring Ciphers
· aes128-gcm@openssh.com · aes256-gcm@openssh.com
in SSH Cipher’s field or configuring only ecdh-sha2-nistp256 algorithm in “SSH KEX” breaks the DRS and CDR functionalities.
Recommended Action 1. From Cisco Unified OS Administration, choose Security > Cipher Management 2. Remove or Delete the above mentioned ciphers if they are already configured and Save the settings.
Troubleshooting Guide for Cisco Unified Communications Manager, Release 12.5(1) 53

Troubleshooting CTL Security Tokens

Cisco Unified Communications Manager System Issues

3. Reboot the server for the changes to take effect.
Troubleshooting CTL Security Tokens
The section contains information about troubleshooting CTL security tokens. If you lose all security tokens (etokens), contact Cisco TAC for further assistance.
Troubleshooting a Locked Security Token After You Consecutively Enter an Incorrect Security Token Password
Note These troubleshooting steps are not required if you manage cluster s

Read User Manual Online (PDF format)

Read User Manual Online (PDF format)  >>

Download This Manual (PDF format)

Download this manual  >>