Tool use & agents at the frontier: Advanced techniques for LLM actions (AIM306)

Here is a detailed summary of the key takeaways from the video transcription, broken down into sections:

Function Calling and Tool Use

  • Early language models were limited in their ability to access real-time information, leading to the need for models to be able to "read from and write to the outside world".
  • Function calling or tool use allows models to call external functions or tools to acquire the necessary data or perform actions to fulfill user requests.
  • The function calling syntax involves:
    • Describing the available functions/tools the model can use
    • Combining the user request with the tool definitions in the prompt
    • Checking the model's response for function calls, executing them, and passing the results back to the model
  • Best practices for function calling include:
    • Using a standardized syntax for function calls
    • Providing a "stop sequence" to prevent the model from hallucinating or generating inconsistent responses
    • Returning function results in a natural language format

Agents

  • The term "agent" has become overloaded and there is no clear consensus on its definition.
  • Agents can be viewed along two axes:
    • Agency - The degree of autonomy and decision-making the model has
    • Effect - The impact or capabilities of the model, including the ability to read and write to the outside world
  • Different "agent" approaches exist, ranging from purely prescriptive models to more autonomous, agentic models that can develop and execute their own plans.
  • As models' capabilities expand, they are able to handle more complex, multi-step tasks involving the use of a large number of tools.

Combining Domain Knowledge and Function Calling

  • Domain knowledge is crucial for effectively solving problems, even more so than knowledge of a specific technology.
  • The example of calculating stock price percentage returns demonstrates how models can leverage their business domain knowledge, combined with function calling, to provide the correct calculation method.
  • Bedrock Agents from Amazon allow the integration of tools, knowledge bases, and code interpretation to solve complex, multi-step problems.
  • Agents can orchestrate the use of these tools and the model's own reasoning to arrive at a final solution, with the option to customize the orchestration strategy (e.g., react, plan-execute-solve, etc.) based on the specific requirements of the problem.

Conclusion

  • Function calling and agent-based approaches provide powerful mechanisms for leveraging language models to automate work and solve complex problems.
  • The choice of orchestration style and tool granularity involves trade-offs that should be considered based on the specific requirements of the problem domain.
  • Exploring the business domain knowledge and standard practices within a model like Claude can be a valuable way to enhance the effectiveness of agent-based solutions.

Your Digital Journey deserves a great story.

Build one with us.

Cookies Icon

These cookies are used to collect information about how you interact with this website and allow us to remember you. We use this information to improve and customize your browsing experience, as well as for analytics.

If you decline, your information won’t be tracked when you visit this website. A single cookie will be used in your browser to remember your preference.

Talk to us