Troubleshoot Jupyter Startup and Spawning

If you are encountering startup or spawning issues with the Jupyter service, follow the guidance in this topic to troubleshoot.

Copy link to this section

URL access management

You must access the Jupyter service from the same URL it was started from, regardless of whether you or someone else started the service. For example, consider the following:

Your application has the canonical URL mycluster.com/myenv/myapp, and you or someone else has set up an application URL myapp.mycluster.com.

If Jupyter was first started from the canonical URL mycluster.com/myenv/myapp/jupyter, you must continue to access Jupyter through the canonical URL. You will not be able to directly access Jupyter through the application URL myapp.mycluster.com/jupyter. If you attempt to access Jupyter through the application URL, you will be redirected to the canonical URL.

Or, if Jupyter was first started from the application URL myapp.mycluster.com/jupyter, you must continue to access Jupyter through the application URL. You will not be able to directly access Jupyter through the canonical URL mycluster.com/myenv/myapp/jupyter. If you attempt to access Jupyter through the canonical URL, you will be redirected to the application URL.

Copy link to this section

Startup issues

If the Jupyter service fails to start up after you navigate to the service URL, try the following troubleshooting steps.

Copy link to this section

JupyterHub troubleshooting commands

You can run the following commands in the C3 AI Console to troubleshoot the Jupyter Service:

• JupyterHub.ensureService(): If you have not already, run this command to initialize the service and list errors. This command requires C3.Developer permissions to run.
• JupyterHub.inst().status(): If this command returns an available status but the service is not available in your browser, an administrator with Kubernetes infrastructure access should check if the ingress rule for the associated Jupyter service has the correct cluster URL for routing.
• JupyterHub.config(): Check the Jupyter service configuration to see if the AppUrl configuration is missing. If it is, create one at the cluster level.
• JupyterHub.config().namePrefix: Run this command to obtain the cloud service hash ID. Run this command in the application C3 AI Console where you started the Jupyter service. Use the cloud service hash ID to identify resources, search logs, and submit support tickets.

Copy link to this section

Check network connection

See Troubleshoot Jupyter using OpenSearch to learn how to access and read Jupyter logs in OpenSearch.

JupyterHub might not be able to connect to proxy. Under the kube-system namespace, check logs for kube-proxy and errors with respect to the iptable refresh. Jupyter service relies on iptable refresh for inter-pod communications.

If your platform deployment has a custom setup, your network policy might interfere with proxy communication between JupyterHub and pods. Delete the networkPolicy Kubernetes resource and try to start the Jupyter service again.

If Jupyter returns a 503 error, check the JupyterHub and proxy status. Check if pod images are present in the registry, and make sure the Docker credentials for pulling the images are correct.

Copy link to this section

Spawning issues

View the event log in JupyterLab.

If you see logs that regard node availability or resource constraint, check resource availability on the Kubernetes node in your cloud provider interface. For example, the following error indicates nodes in the GKE cluster don't have enough resources to schedule the pod with the given resource requirements:

Node scale up failed due to GCE out of resources

If you see logs regarding Volume.NotFound, a Kubernetes administrator should check if the associated volume for a PVC is present. If the volume is not present, run the following command in C3 AI Console to delete the PVC claim:

JupyterLab.terminate(true)

If you see a connection error regarding Kubernetes resources, check overall cluster health. For example, the following log indicates an error with the storage driver pod:

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins/pd.csi.storage.gke.io/csi.sock: connect: connection refused

If the logs show no errors, and overall cluster health is fine, see the Kubernetes documentation Network Plugins and check for Kubernetes Container Network Interface (CNI) issues.

Copy link to this section

JupyterLab issues

View the event log in JupyterLab.

If the JupyterLab connection header icon is red, it indicates a disconnected state and the JupyterLab pod cannot reach the corresponding application. Check application health and for issues with inter-pod communication in the cluster.

To learn how to use C3 AI Studio monitoring tools to monitor application health, see Monitor Environments and Applications. To look into inter-pod communication, see the Kubernetes documentation Network Plugins and check for Kubernetes Container Network Interface (CNI) issues.

If you encounter issues with file access, find out if the file is also problematic in your cloud provider. Check the corresponding path shown in the error message on the blob storage of the corresponding cloud provider. It might not exist in your cloud storage, or the file might be corrupted.

If you experience constant restarts on the JupyterLab pod, check pod behavior:

• Check the jupyterSingleUserCullTimeout value on JupyterHub.config(). This value controls the culling for inactive JupyterLab pods.
• Kubernetes might shuffle pods due to autoscaling settings. Check your cloud provider's autoscaling settings and review the timing of node scaling and pods disappearing.

Copy link to this section

Custom networking setup error

Investigate Jupyter networking errors in OpenSearch logs or c3server container logs on the affected pod.

If the Jupyter service does not spawn and your platform has a custom networking setup, check logs for the following error:

<time_stamp> [vert.x-worker-thread-75] WARN  c.p.m.Retry - Retrying: attempt 103 with wait time: 14050 millis: POST https://kubernetes.default.svc/apis/networking.k8s.io/v1/namespaces/<cluster_id>/ingresses failed with error: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"admission webhook \"validate.nginx.ingress.kubernetes.io\" denied the request: ingress contains invalid paths: path /npsne/sodev/jupyter/? cannot be used with pathType Prefix","reason":"BadRequest","code":400}

If you see this error, custom networking setup likely caused the spawning issue. Run the following code in C3 AI Console to address the error and spawning issue:

JupyterHub.config().setConfigValue('jupyterHubPathSuffix', "")
JupyterHub.stop({waitForReady: true})

Navigate to <yourURL>/jupyter to try spawning Jupyter again.

You can also view environment and application logs to further troubleshoot. See Troubleshoot Environments and Applications in an Error State.

Copy link to this section

Critical warning

Do not run JupyterLab.terminate(true) unless you want to delete unsynced data for all users. This command terminates and deletes unsynced data for every user on the application. A user with C3.AppAdmin permissions can run this command. If you run this command, confirm with all active Jupyter users that their JupyterLab instances are fully synced.

Copy link to this section

See also

See Common Errors in Jupyter for more information on how to troubleshoot common Jupyter errors.