Troubleshoot C3 AI Applications with Server Logs using OpenSearch
You can monitor, troubleshoot, and analyze logs and metrics using the C3 Agentic AI Platform integration with OpenSearch.
Analyze environment and application logs with OpenSearch, an integrated monitoring tool in the C3 Agentic AI Platform. OpenSearch records all user and system actions for standard deployments.
See the OpenSearch documentation at Analyzing data to learn how to analyze and filter data in OpenSearch.
Access OpenSearch
To access OpenSearch, contact your C3 AI representative to obtain credentials. Navigate to <server_domain>/opensearch/. For example, https://yourserverdomain/opensearch.
From the Index patterns dropdown menu, select c3server to view your environment and application logs.
Interpret common log fields
Each c3server log entry has multiple fields ingested from the message body and other cloud information sources. Commonly used fields include the following:
| Field name | Explanation | Common values |
|---|---|---|
t_type | C3 AI Type of the action | String |
t_action | Method of the action | String |
a_id | Unique identifier for the current action | Decimal number |
a_pid | The parent action identifier of your current action | Decimal number |
a_rid | The root action identifier, which refers to the top-level action that invoked all child actions | Decimal number |
a_async_pid | The async parent action. Indicates the parent action in async distribution (<xx>Queue.dispatchCompute) | Decimal number |
a_async_rid | The async root action. Points to the shared root action that submits async actions (job.start, AsyncAction.submit) | Decimal number |
a_st | Status of the action, if it is successful or not | Completed, Failed |
a_t | Timing of the action in seconds | Double value (seconds) |
timestamp | Unix timestamp value | Unix timestamp |
thrd | Thread number of the action | Integer |
env | Environment name. Alternatively environment field contains full environment ID | String |
app | Application name. Alternatively application field contains full application ID | String |
user | User that initiated the action | String |
You can query and filter these fields in OpenSearch to investigate logs.
Get all log entries of a large action
To get all log entries of a large action, determine if the action is synchonous or asynchronous. If you ran a method that does one of the following, the action is likely asynchronous:
- Invokes action using InvalidationQueue, such as
AsyncAction,<xxxx>BatchJob, or<xxxx>MapReduceJob - Uses a flag in a specification, such as
sync: False.
If the action is asynchronous, it goes through the invalidation queue. Note the following fields for asynchronous actions:
a_rid: Points to the action that is triggered in queuea_async_rid: Points to the action that triggers the queue
If the action does not have the previous properties, it is likely synchronous. For synchronous actions, use the a_rid field.
Look for the a_rid and a_async_rid fields and replace the query with their values to get all actions related to those fields.
Filter data to troubleshoot
Apply the environment or application fields and enter the environment or application name to filter the data by particular environments or applications. For example:
application: applicationNameenvironment: environmentName
Apply the c3hostname field and enter node names to filter the data by particular nodes. For example:
c3hostname: domain-nodename-service-c3-k8sdeploy-appsinglenode-001-12345cc64lnnc
Apply the message field and enter the following keywords as values to filter the data by message. For example:
message: "ERROR"message: "connection" AND message: "closed"message: "exception" OR message: "fatal" OR message: "error" OR message: "deadlock" OR message: "shutdown"
Here are some example keywords to anchor your search around events of interest such as exceptions, restarts, or errors:
fatalerrorexceptionGAMEONshutdowndeadlock
Apply the kubernetes.container.name field and the c3-server-log value to filter by log groups. For example:
- Formatted logs:
kubernetes.container.name: "c3-server-log" - Uncaught or standout exception logs:
NOT kubernetes.container.name: "c3-server-log"
Debug an application
Set the log level for an application to DEBUG to record all debug-level logs and higher. Run the following code in the C3 AI Console for the application you want to debug:
App.forName("<appname>").setLogLevel("DEBUG")Then use the message filter to search the logs. For example, search for message: "DEBUG" and message: "Executing" with an environment or application filter.
Use OpenSearch to debug a frequently restarting server
Apply the filter message: "c.server - Stopped" and view the surrounding documents to view logs that occurred before the server restart. Look for errors that might have caused the server to shutdown. Refer to the OpenSearch documentation to learn more about the surrounding documents feature.
Investigate preStop hooks
PreStop hooks are called before a container is terminated due to a failure or error. To learn more about preStop hooks in Kubernetes, see the Kubernetes documentation at Container Lifecycle Hooks.
To investigate Kubernetes preStop information, apply the environment and kubernetes.container.name filters. For example, environment: environmentName and kubernetes.container.name: "c3-server-log".
Then apply a message filter to investigate preStop information:
- Process status information:
message: "Process Status info" - JVM thread dump:
message: "Thread dump:" - JVM memory usage:
message: "JVM Memory usage"