Signals Data Factory - Use of spm utility in windows

In order to use spm utility from windows, once the utility is download to the windows machine, for each user that will use the utility a configuration file called signals.cfg must exist with the details to connect to the application

For example, file must content the default setting, the URL of the environment and user/password that will connect

default http://inventa.infdev.org/ super password

Once the file is created from POWER SHELL (important cannot be use from command line/command prompt/cmd) spm commands can be executed to perform multiple actions like

Data Loading & Import

bash

# Load data from files
spm -e "load --file data.csv --format csv"

# Import from database
spm -e "import --db-connection 'server:port/db' --table experiments"

# Batch load multiple files
spm -e "load --directory /data/ --pattern '*.csv' --merge"

Data Export & Output

bash

# Export dataset to file
spm -e "export --dataset experiment_001 --format csv --output results.csv"

# Export to database
spm -e "export --dataset batch_results --db-connection 'server:port/db' --table output_data"

# Export with filtering
spm -e "export --dataset large_data --filter 'concentration > 0.1' --format json"

Data Querying & Analysis

bash

# Query datasets
spm -e "query --dataset experiment_001 --select 'sample_id,concentration' --where 'ph > 7.0'"

# List available datasets
spm -e "list --datasets --project drug_discovery"

# Get dataset statistics
spm -e "stats --dataset batch_results --columns 'concentration,absorbance'"

Data Validation & Quality Control

bash

# Validate dataset
spm -e "validate --dataset experiment_001 --schema validation_rules.json"

# Check data quality
spm -e "qc --dataset batch_data --outlier-detection --threshold 3.0"

# Validate all datasets in project
spm -e "validate --project-name 'Q1_2024' --all-datasets"

Data Processing & Transformation

bash

# Apply preprocessing pipeline
spm -e "process --dataset raw_data --pipeline preprocess_config.yaml"

# Normalize data
spm -e "normalize --dataset signal_data --method 'min-max' --columns 'intensity'"

# Apply baseline correction
spm -e "baseline --dataset spectra_data --method 'polynomial' --order 2

Dataset Management

bash

# Create new dataset
spm -e "create --dataset-name new_experiment --schema experiment_schema.json"

# Delete dataset
spm -e "delete --dataset old_experiment --confirm"

# Copy dataset
spm -e "copy --source experiment_001 --target experiment_001_backup"

Scheduling & Automation

bash

# Schedule regular data loading
spm -e "schedule --name 'daily_load' --directory /incoming/ --frequency daily --time '02:00'"

# Watch directory for new files
spm -e "watch --directory /dropzone/ --pattern '*.csv' --auto-load"

# List scheduled jobs
spm -e "schedule --list --status active"

Metadata & Schema Management

bash

# View dataset metadata
spm -e "metadata --dataset experiment_001 --show-all"

# Update dataset schema
spm -e "schema --dataset batch_data --update schema_v2.json"

# Compare schemas
spm -e "schema --compare dataset1 dataset2"

Version Control & History

bash

# View dataset history
spm -e "history --dataset experiment_001 --limit 10"

# Create dataset snapshot
spm -e "snapshot --dataset critical_data --name 'pre_analysis_backup'"

# Restore from snapshot
spm -e "restore --dataset experiment_001 --snapshot 'pre_analysis_backup'"

Data Synchronization

bash

# Sync datasets between environments
spm -e "sync --source-env prod --target-env dev --dataset experiment_001"

# Replicate dataset
spm -e "replicate --dataset master_data --target-server backup-server"

# Check sync status
spm -e "sync --status --job-id sync_12345"

Performance & Monitoring

bash

# Monitor system performance
spm -e "monitor --metrics 'cpu,memory,disk' --duration 300"

# View active jobs
spm -e "jobs --list --status running"

# Cancel running job
spm -e "jobs --cancel --job-id load_67890"

Configuration & Settings

bash

# View current configuration
spm -e "config --show --section database"

# Update configuration
spm -e "config --set 'max_workers=8' --section processing"

# Reset to defaults
spm -e "config --reset --section logging"

Utility Commands

bash

# Check system status
spm -e "status --health-check --verbose"

# View logs
spm -e "logs --level error --since '2024-01-01' --tail 100"

# Clear cache
spm -e "cache --clear --type 'processed_data'"

Help & Documentation

bash

# Get help for specific command
spm load --help

# List all available commands
spm --help

# Show version information
spm --version