Jupyter Setup

Installing Jupyter Lab/Notebook

Local Installation

Install Jupyter using UV (recommended for modern Python projects):

# Add Jupyter to your project
uv add jupyter

# Or install globally
uv tool install jupyter

# Alternative: Install [JupyterLab](https://jupyterlab.readthedocs.io/) (more modern interface)
uv add jupyterlab

# Or install both
uv add jupyter jupyterlab

Verify Installation

# Check Jupyter installation
jupyter --version

# Check JupyterLab installation
jupyter lab --version

# List available kernels
jupyter kernelspec list

Running Jupyter on PACE

Basic Setup on PACE

# SSH into PACE
ssh pace

# Load Python module (if using module system)
module load python/3.11

# Create or activate your virtual environment
source ~/.venvs/research-env/bin/activate

# Install Jupyter in your environment
uv add jupyterlab

# Alternative: using pip if UV not available
pip install jupyterlab

Running Jupyter on Login Node (Limited Use)

Only use login nodes for light testing. For actual work, use interactive or batch jobs.

# Quick test on login node (use sparingly)
jupyter lab --no-browser --port=8888

# Better: specify IP to avoid conflicts
jupyter lab --no-browser --ip=0.0.0.0 --port=8888

Method 1: Interactive Session + Port Forwarding

# 1. Allocate interactive session
salloc --account=paceship-dsgt_clef2025 --nodes=1 --ntasks=1 --cpus-per-task=8 --mem-per-cpu=4G --time=4:00:00 --qos=inferno

# 2. Note the allocated node hostname
hostname
# Example output: atl1-1-02-007-30-1.pace.gatech.edu

# 3. Update your SSH config (from local machine)
./update-pace-interactive.sh atl1-1-02-007-30-1.pace.gatech.edu

# 4. Start Jupyter on the interactive node
jupyter lab --no-browser --ip=0.0.0.0 --port=8888

# 5. From another terminal on your local machine, forward the port
ssh -L 8888:localhost:8888 pace-interactive

Method 2: SLURM Batch Job for Long-Running Notebooks

Create a SLURM script jupyter_job.slurm:

#!/bin/bash
#SBATCH --job-name=jupyter-server
#SBATCH --account=paceship-dsgt_clef2025
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=4G
#SBATCH --time=8:00:00
#SBATCH --qos=inferno
#SBATCH --output=jupyter-%j.out
#SBATCH --error=jupyter-%j.err

# Load modules
module load python/3.11

# Activate environment
source ~/.venvs/research-env/bin/activate

# Get the node hostname
NODE=$(hostname)
echo "Jupyter server running on node: $NODE"
echo "Use this command to connect:"
echo "ssh -L 8888:$NODE:8888 pace"

# Start Jupyter
jupyter lab --no-browser --ip=0.0.0.0 --port=8888

Submit and monitor the job:

# Submit the job
sbatch jupyter_job.slurm

# Check job status
squeue -u $USER

# View output (contains connection instructions)
cat jupyter-JOBID.out

Port Forwarding for Remote Access

Simple Port Forwarding

# Forward port 8888 from PACE to your local machine
ssh -L 8888:localhost:8888 pace

# If using interactive node
ssh -L 8888:localhost:8888 pace-interactive

# Multiple ports (Jupyter + MLflow + TensorBoard)
ssh -L 8888:localhost:8888 -L 5000:localhost:5000 -L 6006:localhost:6006 pace-interactive

VS Code Integration

If using VS Code with Remote SSH:

  1. Connect to PACE via Remote SSH
  2. Open terminal in VS Code
  3. Start Jupyter: jupyter lab --no-browser --port=8888
  4. VS Code will automatically offer to forward the port
  5. Click the notification or go to Ports tab

Best Practices for Notebook Organization

Project Structure

research-project/
├── notebooks/
│   ├── 01-data-exploration.ipynb
│   ├── 02-preprocessing.ipynb
│   ├── 03-model-training.ipynb
│   ├── 04-evaluation.ipynb
│   └── 99-final-results.ipynb
├── src/
│   ├── __init__.py
│   ├── data/
│   ├── models/
│   └── utils/
├── data/
│   ├── raw/
│   ├── processed/
│   └── external/
├── pyproject.toml
└── README.md

Notebook Naming Conventions

# Use numbered prefixes for workflow order
01-data-exploration.ipynb
02-feature-engineering.ipynb
03-model-training.ipynb
04-evaluation.ipynb

# Use descriptive names with dates for experiments
2025-01-15-bert-fine-tuning.ipynb
2025-01-16-ensemble-methods.ipynb

# Separate exploration from production
exploratory/
├── data-analysis-jan-15.ipynb
└── model-experiments.ipynb
production/
├── final-model-training.ipynb
└── evaluation-metrics.ipynb