AWS re:Invent 2025 - Beyond web browsers: HITL and tool integration for Nova Agents (AIM3334)

Beyond Web Browsers: HITL and Tool Integration for Nova Agents

Overview

• The presentation focuses on Amazon's Nova Act, an agent-based system for automating computer tasks and workflows.
• Nova Act aims to enable models that can interact with web interfaces and applications like humans, overcoming the limitations of traditional code-based automation solutions.
• Key goals include achieving high reliability, enabling human oversight and intervention, and expanding beyond just web browser interactions.

Challenges with Traditional Automation Solutions

• Traditional browser automation solutions using code-based approaches have several limitations:
  • Long development timelines, often taking months to get up and running
  • Brittleness, where small changes to websites would break the automation
  • Limited generalizability, as each workflow had to be explicitly coded
• Humans can easily adapt to using different applications and interfaces, but replicating this flexibility in software has been a challenge.

Nova Act Approach

• Nova Act trains models to interact with web interfaces in a more human-like way:
  • The model looks at the screen, understands the task and context, and determines the next action to take.
  • This loop of observation, understanding, and action allows the model to be more robust and adaptable.
• Key focus areas for achieving reliability:
  1. Improving element understanding to handle complex UI components like date pickers and dropdowns
  2. Using reinforcement learning on web simulations to explore different patterns and workflows
  3. Emphasizing real-world evaluation and measuring customer success metrics

Human-in-the-Loop (HITL) Capabilities

• Nova Act now includes HITL capabilities, allowing developers to configure the agent to call on human oversight or intervention when needed.
• This enables a hybrid approach where the agent can handle the majority of the workflow, but can hand off to a human for tasks it is unable to complete reliably.
• HITL can be integrated through platforms like Slack or custom UIs.

Expanding Beyond the Browser

• Nova Act is now being extended beyond just web browser interactions.
• Customers are using Nova Act to automate tasks that span multiple systems, like reading test cases from Jira and executing them in the browser.
• This allows Nova Act to be integrated into broader enterprise workflows and automation efforts.

AWS Integration and Developer Experience

• Nova Act is now available as a fully integrated AWS service, providing:
  • A state-of-the-art Nova Act model
  • An AWS console and playground for prototyping
  • SDKs, IDE extensions, and a CLI for development and deployment
  • Observability and logging capabilities through the AWS console
• Key benefits highlighted include frontier-class accuracy, cost-effectiveness, and a streamlined developer experience.

Common Use Cases

• The presentation highlighted four key use cases where Nova Act is providing significant value:
  1. Web QA testing: Using natural language to create robust, adaptable regression tests
  2. Data entry automation: Automating repetitive, manual data entry tasks across web applications
  3. Data extraction: Scraping data from fragmented web sources without APIs
  4. Checkout flow automation: Streamlining e-commerce and travel booking workflows

Customer Examples

1. One Password: Using Nova Act to enable "universal sign-on" that can automatically log users into websites, handling various login methods.
2. Amazon Leo: Leveraging Nova Act for scalable, self-healing QA automation across web and mobile applications.
3. Sol: Integrating Nova Act as a core component of their agentic process automation platform, enabling reliable automation of complex enterprise workflows.

Key Takeaways

• Nova Act represents a significant advancement in agent-based automation, overcoming the limitations of traditional code-based solutions.
• The focus on reliability, human oversight, and expanding beyond just web browsers positions Nova Act as a powerful tool for enterprise-grade automation.
• The tight integration with AWS and the streamlined developer experience make Nova Act an attractive option for organizations looking to leverage agent-based automation at scale.
• The presented use cases demonstrate the broad applicability of Nova Act across various industries and business functions.