Default Cron Jobs
The C3 Agentic AI Platform runs cron jobs to schedule tasks that run automatically at specific times or intervals, or upon user-initiated requests. These cron jobs automate essential maintenance, data management, and operational tasks across the cluster, environment, and application levels.
Review the following default cron jobs and their settings to understand their purposes in the C3 Agentic AI Platform.
Default cron jobs in the C3 Agentic AI Platform
The following cron jobs are seeded in every C3 AI application. Each cron job calls a specific Type and method to perform its operation and has a default active or inactive setting:
| Cron Job | Description | Associated Type | Method | Active or Inactive | Schedule |
|---|---|---|---|---|---|
clean-failed-resources | Cleans up resources that have failed or are in an error state. | K8sCleanupUtil | removeFailedPods | Inactive | Every 10 minutes |
clean-user-event-login | Removes or archives old user login event records. | User | cleanupLoginEvents | Active | Weekly (Sunday at midnight) |
close-merged-jira-tickets-c3ci | Closes JIRA tickets merged via the C3CI process. | Tas | closeMergedJiraTickets | Inactive | Every 2 hours |
close-merged-jira-tickets | Closes JIRA tickets that have been merged, regardless of process. | Tas | closeMergedJiraTickets | Inactive | Hourly |
console-notebook-cleanup | Deletes or archives old or unused console notebook files. | ImplLanguage.Session | cleanupExpiredSessions | Active | Daily at midnight |
ex-machina-checkShutdown | Checks for and initiates shutdown of idle or unnecessary Ex Machina instances. | ExMachinaSession | checkShutdown | Active | Every 5 minutes |
ex-machina-instance-update-statuses | Updates the status of Ex Machina instances based on current state. | ExMachinaInstance.Status.Controller | updateStatuses | Active | Every 15 seconds |
ex-machina-manage-pending-invitations | Manages and cleans up pending invitations in Ex Machina. | ExMachinaUserManagement | managePendingInvitations | Inactive | Daily at midnight |
ex-machina-manage-pending-requests | Handles and clears pending requests in Ex Machina. | ExMachinaApprovalRequest | managePendingRequests | Inactive | Daily at midnight |
ex-machina-report-monthly-usage | Generates monthly usage reports for Ex Machina. | ExMachinaUsageReport | generateForPreviousMonth | Active | Monthly (1st day at 00:10) |
ex-machina-shutdown-sessions-for-completed-schedules | Shuts down sessions in Ex Machina that are no longer needed after schedules complete. | ExMachinaSession | shutdownSessionsForCompletedRuns | Active | Every 5 minutes |
ex-machina-start-unassigned-instances | Starts Ex Machina instances that are unassigned but required. | ExMachinaInstance | startUnassignedInstances | Inactive | Hourly |
expire-disclosure-review | Expires disclosure review records that have passed their review period. | DisclosureReviewRequest | updateExpired | Active | Daily |
export-app-telemetry-mapping | Exports telemetry mapping data for application monitoring. | TelemetryMappingExporter | exportAppTelemetryMapping | Active | Every 15 minutes |
gc-feature-store-blocks | Performs garbage collection on feature store blocks to free up space. | Feature.Store.Block | gc | Inactive | Daily at 2am |
norm-hot-cold-migrator | Migrates data between hot and cold storage tiers for optimal performance and cost. | NormalizedTimeseriesHotColdMigrator | trigger | Inactive | Monthly (4th day at midnight) |
periodic-heaphistogram-export | Periodically exports heap histogram data for memory analysis. | App | dumpJvmHeapHistogram | Inactive | Daily at midnight |
periodic-tmpfilesystem-cleanup | Cleans up temporary filesystems to reclaim disk space. | TmpFileSystem | reapTmpFiles | Inactive | Every 8 hours |
prune-tas-known-issues | Removes or archives known issues in the TAS (Test Automation System). | Tas | pruneKnownIssues | Inactive | Daily at 2am |
seed-nodepool-autoscaler-cronjobs | Seeds cron jobs for node pool autoscaling. | App.NodePool.AutoScaleStrategy.CronJob | seedScalingCronJobs | Active | Every minute |
self_healing_request_recovery | Attempts to recover failed or stuck requests automatically. | C3RequestReconciler | processAll | Active | Every minute |
spark-clean-idle-cluster | Cleans up idle Spark clusters to save resources. | SparkCluster | stopIdleCluster | Inactive | Every 10 minutes |
validate-db-schema | Validates the database schema for consistency and correctness. | Db.TypeSchema | validateSchemas | Active | Daily at 5am |
workflow-run-cleanup | Cleans up completed or failed workflow runs. | Workflow | removeStaleInstances | Active | Daily at midnight |
workflow-run-recover | Attempts to recover or restart failed workflow runs. | Workflow.Run | recoverStuck | Active | Every 30 minutes |
Default cron jobs in C3 AI Studio
The following cron jobs start, stop, resume, upgrade, and terminate environment-level and application-level C3 AI Studio instances based on user-initiated requests:
| Cron Job | Description | Associated Type | Method | Active or Inactive | Schedule |
|---|---|---|---|---|---|
startStudioEnvsApps | Starts an environment-level or application-level C3 AI Studio instance. | Cluster for environment or Env for application | startEnv or startApp | Active | User initiated in C3 AI Studio |
stopStudioEnvsApps | Stops an environment-level or application level C3 AI Studio instance. | Cluster for environment or Env for application | stopEnv or stopApp | Active | User initiated in C3 AI Studio |
resumeStudioEnvsApps | Resumes an environment or application. | Cluster for environment or Env for application | resumeEnv or resumeApp | User initiated | User initiated in C3 AI Studio |
upgradeStudioEnvsApps | Upgrades an environment or application. | Env | UpgradeAppById | Active | User initiated in C3 AI Studio |
terminateStudioEnvsApps | Terminates an environment or application. | Cluster for environment or Env for application | terminateEnv or terminateApp | Active | User initiated in C3 AI Studio |
refreshStudioEnvsApps | Fetches all environments and applications in the cluster, creates C3 AI Studio instances for environments and applications created outside of the C3 AI Studio instance, marks unresponsive C3 AI Studio instances as unreachable, and removes C3 AI Studio instances for environments or applications terminated outside of the C3 AI Studio instance. | StudioCluster | refreshStudioEnvsApps | Active | Every 5 minutes |
Hibernation cron jobs in C3 AI Studio
C3 AI Studio runs the following cron jobs to regularly to hibernate inactive environments and applications to conserve resources:
| Cron Job | Description | Associated Type | Method | Active or Inactive | Schedule |
|---|---|---|---|---|---|
scheduleBasedHibernation | Hibernates all environments and applications at a set cycle, except for ones excluded by a C3.StudioAdmin user. | StudioCluster.Hibernation | processScheduled | Active | Daily at 4am |
activityBasedHibernation | Hibernates single node environments (SNEs) and their applications. Only applies to environments on C3 AI server versions 8.4.0 and lower. | StudioCluster.Hibernation | processInactive | Active | Every 15 minutes |
Activity cron jobs in C3 AI Studio
C3 AI Studio runs the following cron jobs to clean up error-level and stale activities:
| Cron Job | Description | Associated Type | Method | Active or Inactive | Schedule |
|---|---|---|---|---|---|
cleanUpErrorLevelStudioAppActivities | Cleans up error-level application activities according to limit configuration in StudioClusterConfig to avoid overwhelming the database. | StudioCluster | cleanUpErrorLevelActivities | Active | Daily at midnight |
cleanUpErrorLevelStudioEnvActivities | Cleans up error-level environment activities according to limit configuration in StudioCluster to avoid overwhelming the database. | StudioCluster | cleanUpErrorLevelActivities | Active | Every hour at the start of the hour |
cleanUpStaleStudioEnvAppActivities | Cleans up stale environment-level and application-level C3 AI Studio activities based on activityRetentionPeriod configuration in StudioActivityLogConfig. | StudioCluster | cleanUpStaleActivities | Active | Daily at midnight |
View cron jobs
To view all cron jobs, run the following code in the cluster (c3-c3), environment, or application C3 AI Console:
CronJob.fetch()To view a single cron job and its details, run the following code in the cluster (c3-c3), environment, or application C3 AI Console:
CronJob.forId("<cron id>")Replace <cron id> with the name of the cron job you want to view.
Set a cron job to active or inactive
To set a cron job to active or inactive, run the following code in the cluster (c3-c3), environment, or application C3 AI Console:
CronJob.forId("<cron id>").withField("inactive", <true or false>).upsert()Replace <cron id> with the name of the cron job you want to set. Set the "inactive" field to true or false to disable or enable the cron job.
Troubleshoot cron jobs
To check the status of a previously run cron job, run the following code in the cluster (c3-c3), environment, or application C3 AI Console:
CronJob.forId("<cron id>").lastRun()Replace <cron id> with the name of the cron job you want to view.
This code returns cron job information such as execution attempts, the job ID, status, and more.
Or, run the following code to check the status of multiple runs for a cron job:
CronJob.forId("<cron id>").lastRuns(<#>)Replace <cron id> with the name of the cron job and <#> with the number of runs you want to view.
Example
The following example demonstrates how to perform simple troubleshooting on a default cron job.
The cron job clean-failed-resources defaults to an inactive state. Run the following code in an application C3 AI Console to enable it for the application:
CronJob.forId("clean-failed-resources").withField("inactive", false).upsert()If set to active, this cron job runs every 10 minutes. After 10 minutes, run the following code to check the status of the cron job:
CronJob.forId('clean-failed-resources').lastRun()Review the output and notice the following error and status fields:
error : {message: 'API can be executed only on c3-c3', httpStatusCode: 0, action: 'K8sCleanupUtil#removeFailedPods', engine: 'java-server-java', timestamp: {…}, …}status: "FAILED"The response provides a status and error field that you can use to troubleshoot. In this example, the cron job clean-failed-resources failed because you can only enable it on the c3-c3 cluster level.
Consider the resources required to run a cron job. For example, the clean-failed-resources cron job clears failed resources in the node that the job is active on. If you were to enable this cron job on all applications in a cluster, then it would overwhelm the server with unnecessary calls.
However, it's useful to enable this cron job at the cluster (c3-c3) level, because c3-c3 is the control plane responsible for maintaining cluster health. Calls to clean up resources at the cluster level are more optimal than calls to clean up resources on many applications.
Run the following code to enable clean-failed-resources again, but this time in the cluster (c3-c3) C3 AI console:
CronJob.forId("clean-failed-resources").withField("inactive", false).upsert()After 10 minutes, run the following code to check the status of the cron job:
CronJob.forId('clean-failed-resources').lastRun()The output indicates the the cron job ran successfully:
status: "SUCCESS"