AWS re:Invent 2025 - Architecting AI solutions for mission-critical systems w/ UK MetOffice (ARC406)

Architecting AI Solutions for Mission-Critical Systems: Lessons from the UK Met Office

Introduction

• The presentation showcases the Met Office's efforts to leverage AI and machine learning to enhance its weather forecasting capabilities, with a focus on the iconic shipping forecast.
• The Met Office is the UK's National Weather Service, responsible for delivering trusted weather and climate intelligence across various domains, from defense to aviation.

The Shipping Forecast: A Cornerstone of Maritime Safety

• The shipping forecast is the world's oldest and longest-running weather forecast, dating back to 1859 and issued four times a day.
• It provides detailed predictions for 31 sea areas, with specific information on wind speed, direction, weather, and visibility, delivered in a highly structured format.
• The shipping forecast is a critical component of regional maritime safety and a British cultural institution.

The Evolution of Weather Forecasting

• Over the years, weather forecasting has undergone a "quiet revolution" driven by improvements in physics-based numerical models, data assimilation, and model resolution.
• These physics-based models remain an essential component of the Met Office's forecasting approach.
• However, the rise of machine learning and deep learning technologies is rewriting the rules of weather prediction, with data-driven models starting to outperform traditional physics-based models for certain parameters.

The "Last Mile" in Weather Forecasting

• While the underlying weather prediction capabilities have advanced, the "last mile" in weather forecasting - the translation of complex data into actionable information for end-users - remains a significant challenge.
• The shipping forecast serves as an example of this challenge, as the Met Office must take large, multi-dimensional arrays of weather data and convert them into the highly structured, text-based format of the forecast.

Leveraging Large Language Models (LLMs) and Vision-Language Models (VLMs)

• The Met Office explored two approaches to automate the generation of the shipping forecast:
  1. LLM-based approach: Using the Amazon Nova Foundation model to generate the text-based forecast directly from the weather data.
  2. VLM-based approach: Encoding the weather data into video format and fine-tuning a vision-language model to interpret the video and generate the forecast.
• The LLM-based approach achieved 52-62% accuracy in less than 5 minutes, compared to the hours or days it typically takes a human meteorologist.
• The VLM-based approach showed promising results, with the potential to outperform the LLM approach in the future by reducing the information bottleneck of the text-based intermediary.

Experiments and Evaluations

• The Met Office conducted various experiments to optimize the performance of the LLM and VLM approaches, including:
  • Comparing combined versus individual models for different weather attributes
  • Evaluating continuous versus categorical data representations
  • Analyzing the trade-offs between overfitting and early stopping
  • Comparing full-rank fine-tuning versus low-rank adaptation (LoRA) techniques
• The experiments highlighted the importance of careful data formatting and model selection to achieve the best results.
• The team used a strict, word-based F1 score to evaluate the models, as opposed to more lenient metrics like BERT score, to ensure the generated forecasts matched the precise language and format required.

Architectural Considerations

• The team developed scalable, modular architectures to support the AI-powered weather forecasting solutions, leveraging technologies like AWS SageMaker, AWS Bedrock, and AWS Lambda.
• The architectures considered factors such as data processing, model training, and deployment, with options for batch processing, streaming, and serverless approaches.
• The use of AWS Bedrock for model hosting enabled the team to take advantage of the latest foundation models and easily swap them in as new versions became available.

Business Impact and Real-World Applications

• The Met Office's work in this area has the potential to deliver significant benefits to the UK economy and society, with a recent report estimating a return on investment of 19:1 from the taxpayer.
• The automated shipping forecast generation is just one example of how the Met Office is leveraging AI to transform its weather and climate services, enabling more personalized, multimodal, and efficient delivery of critical information to a wide range of stakeholders.
• The lessons learned and architectural patterns developed can be applied to a variety of other weather-related products and services, further enhancing the Met Office's ability to fulfill its mission of providing the most trusted weather and climate intelligence.