Project

Optimizing Transformer-Based Time Series Forecasting

PyTorch optimization work incorporating FlashAttention into Temporal Fusion Transformer training

FeaturedAcademic projectDeveloper
GitHub

Incorporated FlashAttention into the Temporal Fusion Transformer to improve training efficiency for time series forecasting workloads.

Overview

The project focused on performance optimization for transformer-based forecasting. By combining FlashAttention, post-training quantization, and Optuna tuning, the training and inference path became more efficient while preserving the forecasting architecture.

Technical Focus

  • PyTorch implementation work for Temporal Fusion Transformer training
  • Attention-path optimization using FlashAttention
  • Post-training quantization for deployment-friendly model size reduction
  • Optuna tuning across batch size and attention-head configurations
  • Benchmarking training performance before and after optimization
  • Measuring practical runtime and memory improvements on forecasting workloads

Result

The optimized training path achieved a 10.9% reduction in training time, 42.4% faster inference on A100, and a 74.7% model size reduction through quantization.