Summary

This document outlines a proposal for technical design decisions, modular architecture, and implementation strategy for the next generation of OpenSTEF. The new design emphasizes modularity, type safety, modern Python tooling, and a clear separation of concerns while maintaining compatibility pathways for existing users.

The present document should be interpreted as proposal that is to be refined together with the community. We welcome all of your thoughts, considerations and suggestions.

Background and Motivation

Opportunities for Improvement in OpenSTEF 4.0

As OpenSTEF continues to evolve, the upcoming 4.0 release presents a valuable opportunity to enhance it’s robustness, flexibility, and usability. Building on the lessons from version 3.0, the following areas offer key directions for improvement:

1. Code Quality Enhancements

Increase test coverage and streamline test execution to ensure reliability and maintainability.
Standardize coding practices and documentation styles across the codebase for better readability and collaboration.
Centralize data preprocessing logic to improve clarity and reduce duplication across validation and model components.

2. Architectural Advancements

Decouple external dependencies (e.g., MLFlow, openstef-dbc, xgboost/gblinear) to enhance modularity and portability.
Adopt a modular design that simplifies the integration of new models and features.
Introduce flexible configuration mechanisms to replace hard-coded assumptions and improve adaptability.

3. Broader Domain Applicability

Generalize domain-specific logic to support use cases beyond Alliander and the Netherlands (e.g., customizable holiday calendars, dynamic energy pricing).
Relax rigid input data constraints to allow more flexible data formats and structures.
Improve support for diverse data availability scenarios, enabling more resilient forecasting pipelines.

4. Improved Documentation and User Experience

Revamp the getting started guide to provide a smoother onboarding experience for new users.
Clarify assumptions and requirements throughout the documentation to reduce ambiguity.
Clearly distinguish between the standalone library and reference implementation to help users navigate the ecosystem more effectively.

Design Goals and Principles

Core Principles

Modularity First: Components should work in isolation and be easily composable into larger systems.
Type Safety: Full type safety throughout the codebase to catch bugs early and improve maintainability.
Extensibility: Clear interfaces for adding custom models, transforms, and metrics without modifying core code.
Performance: Efficient implementations optimized for production use cases.
Documentation Excellence: Comprehensive, well-structured documentation following the Diátaxis framework

Target Deployments

Research and Experimentation:

Low-code notebooks with pre-built components
Flexible APIs for custom and quick implementations/overrides
Educational tutorials and examples

Small-Scale Deployments:

Docker-compose based deployment examples
Minimal infrastructure requirements
Clear migration paths from examples to production

Enterprise Integration:

Pipeline APIs for existing systems
Flexible callback mechanisms
Custom component development support

Example Forecasting Use Cases

OpenSTEF 4.0 is designed to support diverse forecasting applications, each with specific accuracy requirements, optimization targets, and aggregation characteristics. Examples include:

Congestion Management Forecasts:

Primary Focus: Accuracy near peak load periods
Aggregation Levels: Highly variable - from very aggregated points to low aggregation use cases and even individual customers. Individual customer forecasts can be particularly unpredictable due to behavioral variability.
Typical Applications: Substation forecasting, individual customer predictions, MSRs (middenspannings ruimtes - medium voltage substations)
Business Context: Grid operators need precise predictions at congestion points to implement effective mitigation strategies
Key Metrics: Effective precision and recall, rMAE@50th quantile at peaks, rCRPS
Model Optimization: Emphasis on peak detection and high-quantile accuracy, with robust handling of high variability in low-aggregation scenarios

Transport Forecasts:

Primary Focus: Overall forecast accuracy across all time periods
Aggregation Levels: Medium aggregated points, providing balance between predictability and granularity
Business Context: Grid operators require reliable forecasts to communicate planned energy usage and behavior to upstream network operators and receive similar forecasts from downstream customers. For example, Alliander provides transport forecasts to Tennet (transmission system operator) while receiving forecasts from its customers, enabling coordinated grid management and capacity planning. Additionally some operators require the transport forecasts split in components (solar, wind, other) which necessitates split components models.
Key Metrics: rMAE
Model Optimization: Balanced performance across the entire forecast horizon with emphasis on reliability

Grid Losses Forecasting:

Primary Focus: Overall accuracy with cost-weighted error minimization
Aggregation Levels: Highly aggregated points where system-level (temporal, cyclic) patterns dominate
Predictive Characteristics: Weather predictors have diminished impact at this aggregation level, with stronger temporal patterns and system-wide behavioral trends becoming dominant factors
Business Context: Financial optimization of grid operations considering market price fluctuations
Key Metrics: Similar to transport forecasts plus total error cost minimization based on market prices
Model Optimization: Error weighting based on real-time market prices and operational costs

TODO: add District Heating and non-DSO/TSO related examples

Technical Architecture

Monorepo Structure

OpenSTEF 4.0 proposes a monorepo architecture with specialized packages, each serving distinct purposes while maintaining cohesive development workflows:

openstef/
├── openstef-models/               # Core ML components and feature engineering
├── openstef-beam/                 # Backtesting, evaluation, analysis and metrics
├── openstef-compatibility/        # OpenSTEF 3.0 compatibility layer (optional)
├── openstef-foundational-models/  # Foundational models (renamed from deep-learning)
├── openstef-benchmarks/           # Performance benchmarks
└── openstef-examples/             # Deployment and tutorial examples

Feature Hierarchy Convention

The architecture follows a four-level hierarchy which provides the perfect balance between flexible and ease of use. This is done by defining a level that caters to each user’s need by also provides a reusable component for higher levels:

Level 1 - Standalone Functions:

Pure functions operating on highly typed arguments (numpy arrays, structs)
Metrics, utilities, basic transformations
No dependencies on framework components

Level 2 - Standardized Components:

Classes with defined APIs (data transformers, models, metrics)
Extensible and configurable
Can be stateless or trainable

Level 3 - Multi-Component Pipelines:

Orchestrate Level 2 components
Extensible but without standardized APIs
Training, forecasting, evaluation pipelines

Level 4 - Pre-configured Solutions:

Ready-to-use configurations of Level 3 components
Domain-specific optimizations
Out-of-the-box functionality for common use cases

Package-Specific Design Decisions

openstef-models

Core Data Handling:

Custom Time Series Dataset: First-class support for both a timestamp and an available_at timestamp. This distinction allows the system to account for data that is revised or becomes available at different times (e.g., weather forecasts). By default, available_at is set to the same value as timestamp unless otherwise specified. This flexibility enables more accurate model training, especially for features like lag transforms.

Example table:

| timestamp           | available_at        | feature_1 | feature_2 |
|---------------------|---------------------|-----------|-----------|
| 2025-01-01 00:00:00 | 2025-01-01 00:00:00 | ...       | ...       |
| 2025-01-01 01:00:00 | 2025-01-01 02:00:00 | ...       | ...       |

Internally, the data structure maintains an index to track which points in time have values for any availability. Datasets with different availabilities can be concatenated as this transformation is not trivial.

Flexible Schema: Input data is accepted as a pandas DataFrame, supporting any data source (including CSV for small files and Parquet for larger datasets). Input validation is performed by a wrapper around the DataFrame called TimeseriesDataset, which enforces expected shape, columns, and validation rules (e.g., presence/absence of weather data, timezone handling). Users can pass individual data sources (weather, price, etc.) as series or DataFrames and specify the desired resolution; a helper function will resample to the correct format.

Parquet Format: Parquet is used internally for medium to large datasets due to its efficiency and schema preservation, eliminating the need for type juggling or timestamp re-parsing. While Parquet is the default for internal artifacts (such as STEF BEAM outputs), input data can be provided in any format supported by pandas.

Polarity convention: Input should adhere to the convention consumption as positive and generation as negative

Module Architecture:

openstef-models/
├── feature_engineering/
│   ├── validation_transforms/     # Data validation checks (completeness, flatliners, etc.)
│   ├── temporal_transforms/       # Lags, holidays, cyclic features
│   ├── forecasting_transforms/    # Normalization, trend, seasonality
│   ├── weather_transforms/        # Radiation, humidity calculations
│   └── energy_domain_transforms/  # Wind/solar energy specific features
├── models/
│   ├── forecasting/              # Core forecasting models
│   └── component_splitting/      # Component decomposition models
├── anomaly_detection/            # Utilities for anomaly detection (can be embedded in transforms)
├── pipelines/                    # Model training and inference pipelines
├── postprocessing/               # Quantile processing and calibration
└── explainability/               # Model interpretation tools

Key Design Decisions:

Separation of Concerns: Feature engineering operates independently from model pipelines
Configurable Transforms: Transforms are generic and support a fit() method, enabling arbitrary data transformations. Dimensionality reduction (e.g., PCA) is supported in principle; an out-of-the-box implementation may not be available in the first release, but the design makes it easy to implement.
Probabilistic by Default: All models support quantile predictions as a first-class feature
Anomaly Detection Module: Dedicated module for detecting data quality issues including flatliners, polarity switches (out of scope for initial release), and other anomalies.
Custom Anomaly Detection: Both univariate and multivariate anomaly detection can be implemented as transforms, allowing users to define their own methods as needed.
Data Leakage Prevention: Strict enforcement of feature availability constraints to prevent training on future information. The feature engineering pipeline validates that all features used for training are available at prediction time with appropriate horizons, ensuring no look-ahead bias in production deployments
Callback System: Decoupled helper logic through callbacks for training monitoring, model persistence, and custom integrations
Explainability: Importance plots will be supported in OpenSTEF 4.0. SHAP values are planned as a new feature and will be introduced in a later release.
Configuration: The configuration system is now fully modular. Rather than a monolithic configuration, individual components are configurable. Pre-made, opinionated combinations of components will have larger, more centralized configs, but the underlying modularity remains. For detailed configuration workflows, refer to the OpenSTEF 4.0 Architecture Document .

openstef-beam

High-Level Pipeline Architecture:

openstef-beam/
├── core/
│   ├── dataset/                  # Advanced dataset operations
│   └── utils/                    # Shared utilities
├── backtesting/                  # Backtesting framework with retrain strategies        
├── evaluation/                   # Model evaluation and comparison pipelines     
├── analysis/                     # Analysis plotting and reporting
└── metrics/                      # Comprehensive forecasting metrics

Key Features:

Backtesting Pipeline: Advanced backtesting with configurable retrain strategies, and data availability to simulate real world deployments.
Evaluation Pipeline: Standardized model evaluation with support for multiple metrics, evaluation windows, and ad-hoc execution. It should be fit for backtest analysis, but also production forecast performance evaluation.
Analysis Pipeline: Report creation based on plot selection and configuration. Allowing in depth analysis of a single model or comparison of multiple models.
Metrics Library: Comprehensive metrics including domain-specific measures like effective precision/recall at peaks and rCRPS. These should be extensible to allow definition of custom metrics and data driven metrics like cost based metrics.

Pipeline Naming: The term "pipeline" is used for components that combine or orchestrate multiple sub-components. This naming is consistent across benchmarking, evaluation, and analysis, providing clarity and consistency. The analysis_pipeline specifically orchestrates multiple visualization providers. Alternative names like report_template were considered, but "pipeline" was chosen for consistency.

Plotting and Visualization Strategy

Technology Stack:

Primary Library: Plotly for interactive visualizations
Notebook Compatibility: PNG backend configuration for non-interactive notebooks

Performance Considerations: Plotly’s API is versatile and lightweight when used for plotting. The backend determines the output: for interactive (but potentially slower) plots with large datasets, use the interactive Plotly backend; for fast, compact, non-interactive visualizations, use the Plotly PNG backend. This should be configurable by the user.

Architecture:

# Notebook configuration for non-interactive rendering
import plotly.io as pio
pio.renderers.default = "plotly_mimetype+notebook"  # For interactive environments
# Alternative: pio.renderers.default = "png"  # For static notebook exports

Three-Tier Visualization Design:

Level 1: Core plotting utilities for formatted data
Level 2: Visualization providers converting evaluation outputs to plots
Level 3: Report pipelines chaining multiple visualization components

Notebook Management:
As part of the linting process, all notebooks must have their outputs cleared to avoid introducing unnecessary environment-specific information into the repository. This is enforced through pre-commit hooks using tools like nbstripout. During the documentation pipeline, notebooks are executed fresh to create reproducible outputs and incorporated into the docs, ensuring that all examples work correctly and generate consistent visualizations across different environments.

Benefits of Plotly Choice:

Interactive plots for exploratory analysis
Consistent API across plot implementations
Professional publication-quality outputs
Seamless integration with Dash as Streamlit for web applications
PNG backend support ensures compatibility with non-interactive notebook environments and static documentation

openstef-examples

Deployment Examples:

Orchestration Options: Dagster, Celery, Airflow integrations
Container Strategy: Everything works with docker-compose for easy local development
Production Patterns: Scalable deployment architectures

Educational Content:

Getting Started Notebooks: Progressive learning path for new users
Advanced Tutorials: Domain-specific examples and best practices
Dataset Integration: Optional open-source datasets for learning

CI/CD And Development Tooling

Modern Development Stack

Python Environment:

Minimum Version: Python 3.12+ to leverage the latest language features, significant performance improvements, and enhanced standard library capabilities that streamline development workflows
Type Safety: Full type safety throughout the codebase to catch bugs early, improve maintainability, and enhance developer experience
Dependency Strategy: Core functionality with minimal required dependencies, optional extras for specialized features, ensuring clean separation between required and optional components

Package Management: uv for fast, reliable dependency management

Code Quality Automation:

Linting & Formatting: Ruff (with custom rule selection) for quick formatting and linting
Type Checking: Pylance for full type safety validation
Testing: pytest with coverage
Configuration: Centralized in pyproject.toml following modern Python standards

Quality Standards:

Test Coverage: Comprehensive unit and integration testing focused on important code paths. Preferring, synthetic data over static testing datasets.
Performance: Optional datasets benchmarking to find performance regressions or improvements improvements quickly.
Documentation: Public API’s documented with examples and type hints

CI/CD Pipeline Architecture

GitHub Actions Implementation:

Check Pipeline (Every PR):

Code quality validation (ruff linting and formatting)
Type safety verification (pylance)
Comprehensive test suite execution
Example Notebook Validation: Automated execution of all example notebooks to ensure they remain functional
Run tests in a matrix on linux, mac and windows devices.

Release Pipeline (Push to Main):

Automated release draft creation
Manual publishing with optional version increment (to major in case of breaking changes or patch in case of fixes)

Documentation Pipeline (Post-Release):

Automatic documentation generation and deployment
Example gallery updates with validated notebooks

Example Validation Strategy

Automated Notebook Testing:
Sphinx-Gallery executes Python examples during documentation builds to generate galleries with outputs. Following this approach of validating examples during the build process, all example notebooks are automatically executed as part of the CI pipeline:

# Example GitHub Actions step
- name: 'Validate Examples'
  run: |
    # Install dependencies and execute all notebooks
    find examples/ -name "*.ipynb" | xargs jupyter nbconvert --execute --to notebook

Benefits:

Ensures examples remain up-to-date with API changes
Catches breaking changes before release
Maintains high quality of educational content
Provides confidence in documentation accuracy

Documentation Strategy

Framework and Technology

Documentation Stack:

Framework: Sphinx with PyData theme for modern data library appearance (similar to SKLearn, Matplotlib, Pandas)
Enhanced Features:
- sphinx-gallery: Automated example gallery generation with API integration
- nbsphinx: Embedded Jupyter notebooks with executable examples

Structure (Diátaxis Framework):

Tutorials: Step-by-step learning materials for beginners
How-to Guides: Task-oriented instructions for specific problems
Reference: Comprehensive API documentation with embedded examples
Explanation: Background concepts and design rationale

Quality Assurance:

Example Validation: All documentation examples are tested in CI
Version Management: Synchronized documentation versions with releases
Community Features: Searchable documentation with version switching

Compatibility and Migration

If there is a need from the community for a soft migration, we can consider adding an additional package openstef-compatibility. (out of scope for the initial release unless discussed)

Compatibility Package: openstef-compatibility provides API-compatible wrappers around OpenSTEF 4.0 components, enabling gradual migration from OpenSTEF 3.0. A compatibility layer will translate OpenSTEF 3.0 pipeline calls to OpenSTEF 4.0, allowing soft migration. Note: making model checkpoints compatible is not planned due to pipeline and data revisions in 4.0. Instead if necessary a conversion strategy should be used.

Migration Strategy:

Detailed Migration Guide: Step-by-step instructions with code examples also for version jumps between OpenSTEF 4.0.
Parallel Maintenance: Limited-time support for critical OpenSTEF 3.0 fixes

OpenSTEF 4.0 Proposal