Nagelfluh User Guide

This guide explains how to use Nagelfluh for geophysics data processing.

Getting Started

After installation (see Deployment Guide), open your browser to http://localhost:3000.

First Time Setup

  1. Select Environment: Choose "Bootstrap" from the environment dropdown (top of screen)
  2. Explore the Interface: The default layout shows:
  3. FlowView (left): Visual graph of processes
  4. ProcessEditor (top right): Create/edit processes
  5. ProcessLog (bottom right): Real-time logs

You can rearrange these widgets by dragging panes, creating splits, or opening tabs.

Understanding the Interface

Main Widgets

FlowView - Process Graph

Shows a visual graph of all processes and their dependencies:

ProcessEditor - Create and Edit Processes

Dual-mode editor that changes based on whether a process is selected:

Create Mode (no process selected): 1. Select Process Type: Choose from dropdown (e.g., "fft", "inversion") 2. Enter Process Name: Give your process a meaningful name 3. Configure Resources: - CPU: 0.1 - 8 cores (default: 1) - Memory: 0.5 - 32 GB (default: 2 GB) - Deadline: 1 - 1440 minutes (default: 60 minutes) 4. View Cost Estimate: See maximum possible cost 5. Fill Parameters: Form fields based on process type 6. Submit: Click to create and run process

Edit Mode (process selected): - View current parameters - See output datasets - Create new version with modified parameters - Cancel a version that is still queued or running - View process status and history

ProcessLog - Real-time Logs

Displays logs from running and completed processes:

PlotView - Data Visualization

Interactive scientific plotting:

MapView - Geographic Visualization

Display survey data on interactive maps:

Layout Customization

Creating Splits

Right-click pane header → "Split Horizontal" or "Split Vertical"

Or drag a pane to edge of another pane to create split.

Creating Tabs

Drag a pane to the center of another pane to create tabs.

Popout Windows

Click ⧉ button in pane header to open in separate window (great for multi-monitor setups).

Changing Widget Type

Use dropdown in pane header to switch widget (e.g., PlotView → MapView).

Closing Panes

Click × button in pane header.

Creating and Running Processes

Process Lifecycle

  1. Create Process: Define parameters and resources
  2. Estimation: System calculates maximum cost
  3. Validation: Checks balance and parameter schema
  4. Hold Funds: Reserves maximum possible cost
  5. Queuing: Kueue queues job until resources available
  6. Execution: Kubernetes pod runs process, streams logs
  7. Completion: Actual cost charged, held funds released
  8. Outputs: Datasets registered and available for visualization

To stop a process before it finishes, click the Cancel button in ProcessEditor while the version is shown as pending or running. The Kubernetes job is deleted immediately and the version is marked as failed.

Step-by-Step: Creating a Process

  1. Deselect any process: Click empty area in FlowView (ProcessEditor shows "Create" mode)

  2. Select process type: Choose from dropdown

  3. fft: Fast Fourier Transform analysis
  4. inversion: Geophysical inversion
  5. processing: AEM data processing

  6. import_data: Import external data

  7. Name your process: Enter descriptive name (e.g., "FFT Analysis - Line 1")

  8. Configure resources:

CPU Cores: - 0.1 cores: Light processing - 1 core: Standard processing (default) - 2-4 cores: Heavy computation - 8 cores: Maximum (very intensive)

Memory: - 0.5 GB: Minimal data - 2 GB: Standard (default) - 4-8 GB: Large datasets - 16-32 GB: Very large datasets

Deadline: - How long process is allowed to run before timeout - Be generous - unused time doesn't cost extra - Default: 60 minutes

  1. Review cost estimate: Shows maximum possible cost based on deadline
  2. Actual cost will be less (based on actual runtime)

  3. Example: 1 core, 2 GB, 60 min → ~$0.0024 max

  4. Fill in parameters:

  5. Parameters depend on process type
  6. Dataset fields: Use searchable dropdown to select from previous process outputs
    • Type to search by process name or dataset name
    • Format: "Process Name / v123 / dataset-name"
    • Grouped when >4 datasets from same process

  7. Other fields: Numbers, text, dropdowns as needed

  8. Submit: Click "Create Process" button

  9. Monitor progress:

  10. Process appears in FlowView
  11. Logs stream to ProcessLog
  12. Status updates in real-time

Example: Running FFT on Imported Data

Assume you've already run an "Import Data" process:

  1. Click "Create Process" mode in ProcessEditor
  2. Select process type: fft
  3. Name: "FFT - Survey Line 1"
  4. Resources: 1 core, 2 GB, 60 minutes (defaults are fine)
  5. Parameters:
  6. Input Data: Search "Import", select the import process output
  7. Click "Submit"
  8. Watch FlowView - new "FFT - Survey Line 1" node appears
  9. Watch ProcessLog - see "Starting FFT...", progress messages
  10. When complete, status shows 🟢 Completed
  11. Click the process to view outputs in ProcessEditor

Working with Datasets

What are Datasets?

Datasets are output files from processes. Each process can produce multiple datasets (e.g., "result", "diagnostics", "metadata").

Dataset Types

Accessing Datasets

In ProcessEditor (when process selected): - "Outputs" section lists all datasets - Click dataset name to download - Copy URL to share or use in API calls

In PlotView: - Add plot element - Select dataset from searchable dropdown - Visualize immediately

In MapView: - Select geographic datasets - Overlay on map

When selecting datasets in forms:

Monitoring Processes

In the UI

FlowView Status

ProcessLog

Real-time log streaming: - All processes: Shows logs from all processes by default - Filter by process: Click process in FlowView to filter - Auto-scroll: Keeps latest logs visible - Status badges: Quick state overview

ProcessEditor Status

When a process is selected: - State: Current process state - Parameters: What settings were used - Outputs: Links to result datasets - History: Version history if process was modified

Via Command Line (Advanced)

For administrators or developers:

# Check jobs
kubectl get jobs -n nagelfluh-jobs

# Check pods
kubectl get pods -n nagelfluh-jobs

# Stream logs
kubectl logs -f <pod-name> -n nagelfluh-jobs

# Check queue status
kubectl get workloads -n nagelfluh-jobs

Billing and Costs

How Billing Works

Nagelfluh uses a hold/release billing model to ensure fair resource pricing:

  1. Process Creation:
  2. System calculates maximum possible cost (deadline × resources)
  3. Checks your account balance
  4. If sufficient: Creates HOLD transaction (reserves funds)

  5. If insufficient: Rejects process creation

  6. Process Execution:

  7. Pod runs in Kubernetes cluster
  8. Uses resources (CPU, memory)

  9. Streams logs to UI

  10. Process Completion:

  11. System calculates actual cost (actual runtime × resources used)
  12. Creates DEBIT transaction (charges actual cost)
  13. Creates RELEASE transaction (frees remaining held funds)
  14. Updates account balance

Cost Formula

Max Cost = (CPU cores × $0.0001/minute) + (Memory GB × $0.00002/minute) × Deadline
Actual Cost = (CPU cores × $0.0001/minute) + (Memory GB × $0.00002/minute) × Actual Runtime

Example Costs

Configuration Deadline Max Cost 5-second Runtime 60-minute Runtime
1 core, 2 GB 60 min $0.0024 ~$0.0006 $0.0024
4 cores, 8 GB 120 min $0.0384 ~$0.0024 $0.0192
8 cores, 16 GB 240 min $0.1536 ~$0.0048 $0.0768

Tips for Managing Costs

  1. Set realistic deadlines: Don't overestimate - you're not charged for unused time
  2. Right-size resources: Start with defaults (1 core, 2 GB), increase if needed
  3. Monitor usage: Check ProcessLog to see how long processes actually run
  4. Reuse results: Datasets persist - don't re-run unnecessarily
  5. Test with small data: Validate workflow before processing full datasets

Viewing Balance and Transactions

(UI features coming soon)

Managing Projects and Environments

Projects

Each project has: - Isolated storage bucket: Per-project S3/GCS bucket - Separate processes: Processes don't cross projects - Dedicated credentials: Scoped IAM permissions - Independent billing: Track costs per project

Environments

Environments define the available process types and Docker images:

Creating Custom Environments

(Coming soon: Process-based environment builder)

  1. Run "create_environment" process
  2. Specify base image and dependencies
  3. System builds Docker image
  4. New environment appears in dropdown

Troubleshooting

Process Stuck in "Pending"

Cause: Insufficient cluster resources

Solutions: 1. Wait - Kueue will schedule when resources free up 2. Check cluster capacity: kubectl get nodes 3. Reduce resource requirements (fewer cores/memory) 4. Contact administrator to scale cluster

Process Failed Immediately

Cause: Parameter validation error or missing dependencies

Solutions: 1. Check ProcessLog for error messages 2. Verify all required parameters filled 3. Check dataset URLs are valid 4. Ensure input datasets exist

Can't Find Dataset in Selector

Cause: Dataset not created yet or search too broad

Solutions: 1. Verify source process completed successfully 2. Refine search - type more specific process name 3. Click grouped results to narrow search 4. Check ProcessEditor outputs of source process

Logs Not Updating

Cause: WebSocket connection lost

Solutions: 1. Refresh browser page 2. Check browser console for errors 3. Verify backend is running: curl http://localhost:8000 4. Check network connectivity

Process Exceeded Deadline

Cause: Process took longer than deadline setting

Solutions: 1. Increase deadline in next run 2. Optimize process parameters (smaller dataset, fewer iterations) 3. Increase CPU cores to speed up processing 4. Check if process hung (logs stopped updating)

Storage Permission Denied

Cause: IAM policy misconfiguration

Solutions: 1. Verify project storage was created automatically 2. Check Kubernetes secret exists: kubectl get secret project-{id}-storage 3. Contact administrator to verify IAM policies 4. Check ProcessLog for specific error message

Out of Balance

Cause: Insufficient funds for process creation

Solutions: 1. Check current balance (UI coming soon) 2. Contact administrator to add funds 3. Reduce resource requirements or deadline 4. Delete unnecessary processes to free held funds (if cancelled)

Best Practices

Process Naming

Resource Allocation

Dataset Management

Workflow Organization

Performance Tips

Keyboard Shortcuts

(Coming soon)

Getting Help

Documentation

Support

Glossary