Meet AutoPlan

A New Way to Build Agentic AI Applications with Dynamic Planning

If you’re into building AI-powered applications, you must be familiar with a series of prompt engineering techniques, external tools to augment LLMs' capabilities and architectural patterns to build effective solutions. While these techniques are great for building modular and flexible AI applications, we realized there’s room for something complementary: a way to break down applications into smaller parts and letting LLMs dynamically plan how to use available tools to solve tasks effectively. That’s why we’re so excited to share Autoplan with you.

What difference does Autoplan make concretely?

To illustrate this through a simple example, let's say we want to create a conversational agent that can answer questions related to stock performance, such as "Compare Nvidia's performance to Alphabet and Amazon". This kind of an agent would need to use external tools to download real time data from the stock market and compute certain metrics and summarize all this information to a benchmark report.

How would one create an orchestration layer using these tools? Imagine a straightforward workflow: an orchestration layer defined by predefined code paths, as specified by Anthropic. This layer would use a specific LLM prompt to extract entities from the user query, loop over each entity to download their data and compute key metrics, and finally summarize these findings. If you were to implement this workflow with a graph framework, it would look like the following.

With Autoplan, you can take an agentic approach and let the framework generate the plan for you, shortcutting certain steps like entity extraction and looping, efficiently executing the tasks, and summarizing the results for presentation to the user.

But wait, if you're not convinced yet, think about another common question a user might ask: "Compare Nvidia's performance against the market". The graph implementation which is programmed to extract entities from the user query to feed the loop may just fail at that moment because it may not know what "against the market" means. This is where the common-sense knowledge of foundational LLMs can make a difference. These models are likely aware of which stock markets are relevant to Nvidia and which indices, like Nasdaq, SP500, and Dow Jones, serve as meaningful benchmarks. With Autoplan, the planner can dynamically decide to compare Nvidia against these indices, generate a plan to retrieve the necessary data, and compute the metrics, without requiring updates to your application’s architecture.

What difference does this make fundamentally?

LLMs have made creating AI applications simpler by enabling instructions through textual prompts. However, they are not a one-size-fits-all solution for applications that can be deployed at scale. Developers often face challenges in breaking applications into smaller components and managing the control flow and data exchange between these parts to make these applications robust enough to be deployed.

Most existing LLM frameworks such as LangGraph, Amazon Bedrock and Autogen provide patterns like self-criticism, sequencing, parallelizing, or looping over tasks. However, these frameworks still require developers to manually define control flows at design time, relying on prior knowledge about the tasks the system is expected to handle. While this approach gives developers a certain degree of control, it limits flexibility. Applications built this way struggle to adapt dynamically to new tasks or contexts at runtime.

In contrast, an agentic system that automatically generates control flows provides a more adaptable and powerful solution. Instead of relying on predefined workflows, it leverages an LLM to generate a continuum of plans dynamically based on the specific context of a user query at run time. Advances in LLM reasoning and common-sense knowledge make this increasingly feasible. While autonomous agentic systems still face challenges in reliability and efficiency, we saw this as the perfect time to create Autoplan—a framework that empowers developers to explore the trade-offs between static workflows and more dynamic agentic flows.

How does Autoplan compare to function calling?

Autoplan’s core functionality of dynamic planning shares similarities with the function calling feature found in major foundational models like GPT, Claude and Gemini. However, these models only return the functions (i.e., tools) to be called as a JSON object, leaving developers with several open questions: How should the models be prompted to generate an accurate plan that effectively uses the tools? How can the functions be executed efficiently? How should data flow between tools? How should the output of tools be summarized to generate the final output?

Autoplan fully integrates dynamic planning and execution as a first-class capability, addressing these challenges:

• Structured Inputs and Outputs: Autoplan leverages Python type annotations and Pydantic classes to specify tools with structured and typed inputs and outputs. This allows the planner to precisely understand how to interact with external tools and manage data flow effectively.

• Planner Prompting: Autoplan provides a structured input and output format for plan generation, making it easy to apply best practices such as chain-of-thought reasoning and tool documentation injection. This ensures the Planner LLM receives high-quality inputs, which are essential for generating effective plans.

• Efficient Execution: Autoplan’s orchestration layer automatically captures task interdependencies, identifies opportunities for parallel execution, and optimally executes tasks to minimize overall runtime latency.

• Dataflow Management: Autoplan efficiently tracks and manages data dependencies, ensuring proper sequencing of tool executions and efficient transfer of outputs from upstream to downstream tasks.

• Output composition: Autoplan supports generating the final output to meet application requirements by providing a pattern that allows developers to define how intermediate outputs from various tools should be composed.

What other key features does Autoplan offer?

Beyond dynamic planning and execution, Autoplan provides developers with additional key benefits that improve usability, debugging, and flexibility in building and scaling their applications:

• Streaming Output: Autoplan streams incremental results, such as the generated plan and the status of each step’s execution, enabling real-time progress tracking and user interfaces that display results as they are produced.

• Observability: Autoplan enhances debugging, auditing, and monitoring through its robust logging capabilities, capturing inputs and outputs for all tools. It also supports multiple observability providers, including Weight & Biases, delivering out-of-the-box observability.

• Extensibility: Adding new tools is straightforward. Any typed Python function with a well written docstring can be a tool, making it easy to expand and customize your application.

• LLM Agnostic: Autoplan integrates with LiteLLM, supporting a diverse range of LLMs, including the latest DeepSeek R1 models. This flexibility allows developers to experiment with and combine different models, enabling them to identify the best ones for their specific needs.

Where to start?

Stay tuned for more blog posts where we'll dive deeper into the key features of Autoplan. In the meantime, the best way to get started is by installing it, exploring a few examples in the documentation, bootstrapping your own Autoplan application, checking out the code and most importantly, having fun!