Alibaba Unveils AI Framework That Reduces Agent Token Usage by 99% by Streamlining Tool Loading

As organizations increasingly rely on AI systems to manage intricate workflows, a key hurdle lies in effectively directing subtasks to the appropriate tools and capabilities. With agents often equipped with numerous tools, determining which to apply at various stages can lead to confusion.

To tackle this issue, researchers from Alibaba have introduced SkillWeaver, a framework that develops an execution graph tailored to specific tasks, selecting suitable skills for each element. They also present a novel method called Skill-Aware Decomposition (SAD), which employs a feedback mechanism to help the agent progressively identify and evaluate relevant tool options. This iterative strategy, combined with its distinctive compositional approach, sets SkillWeaver apart from conventional frameworks that typically opt for tools in a single step.

SkillWeaver has significant implications for real-world AI applications, enabling agents to autonomously manage multi-tool ecosystems in scenarios such as the Model Context Protocol (MCP). This is particularly relevant for executing multi-step business tasks, including data downloading, processing, and report generation.

Experimental results demonstrate that employing this iterative retrieve-and-route methodology considerably enhances accuracy while slashing token usage by over 99% compared to simply providing agents with unrestricted access to an entire tool library.

For developers working on AI agents, the primary insight is that how finely tasks are decomposed is the main challenge to achieving precise tool retrieval.

Challenges in Skill Routing

Skills play a vital role in modern large language model (LLM) agent architectures, serving as modular tools with structured documentation.

As enterprise agents integrate with extensive tool collections, accurately directing user inquiries to the appropriate skills becomes increasingly complex. Presenting an entire library to an LLM to identify the right tool is inefficient, often overwhelming context limits and resulting in exorbitant token usage.

Most existing frameworks address this through methods such as API retrieval and documentation matching, relying on hierarchical models that treat routing as either a single-skill selection issue or a per-step dilemma.

However, this approach is inadequate for enterprise contexts where real-world queries are typically compositional. For example, a standard business task like “Download the dataset, transform it, and create visual reports” requires multiple tools, necessitating a breakdown and sequencing of an API client, a data processor, and a visualization tool into a coherent multi-step plan.

How SkillWeaver and SAD Operate

To address this, the researchers define the challenge of managing complex, multi-skill tasks as “compositional skill routing.” When faced with a complicated user request and a large array of tools, an agent must simultaneously decompose the prompt into specific sub-tasks, map each one to the best available skill, and integrate those skills into a workable plan.

SkillWeaver facilitates this process in three defined stages: Decompose, Retrieve, and Compose. Initially, an LLM acts as the decomposer, breaking down the user’s complex request into clearly defined sub-tasks requiring individual skills. After identifying these subtasks, the framework utilizes an embedding model to benchmark each against the skill library, generating a shortlist of suitable tools for each step.

The Composition phase involves a planner that assesses the retrieved options based on their compatibility, ensuring that the output of one tool aligns with the input of the next. It ultimately constructs a final execution plan as a Directed Acyclic Graph (DAG), allowing independent tasks to execute simultaneously where feasible.

For instance, if a user instructs an AI agent to “Download the dataset, transform it, and create visual reports,” the decomposition stage will identify three distinct sub-tasks: downloading the dataset, data processing, and report generation.

During the retrieval phase, the system will locate candidates like “api-client” or “http-fetch” for the initial task and “csv-parser” or “etl-pipeline” for the data transformation step, among others. The final composition will determine the best combination of “api-client,” “csv-parser,” and “chart-gen” to create a seamlessly connected workflow.

A critical challenge within this sequential process is that LLMs may produce vague descriptions that don’t match the specific terminology used for skills in the library. To remedy this, SkillWeaver incorporates the Iterative Skill-Aware Decomposition (SAD) feedback loop. Initially, the LLM drafts a preliminary plan, conducts a search to identify loosely matching skills, and feeds that information back to refine its task breakdown, aligning the granularity and terminology with the existing tools.

SkillWeaver in Practice

To assess SkillWeaver’s performance in realistic enterprise contexts, the researchers designed a custom benchmarking suite called CompSkillBench. This includes 300 multi-step queries of varying complexities, using a library populated with 2,209 real-world skills spanning 24 domains, such as cloud infrastructure and finance.

The core engine primarily utilizes a lightweight 7-billion parameter model for task decomposition, paired with a standard semantic search retriever to identify tools. SkillWeaver was evaluated against three models: a brute-force “LLM-Direct” method, a basic LLM-based decomposition lacking SAD, and a ReAct-style agent configuration.

The findings reveal that task decomposition presents a significant bottleneck. Traditional LLMs struggle with vast tool libraries, whereas the SAD feedback loop greatly enhances performance. In one setup, the 7B model correctly decomposed tasks only 51% of the time, but with SAD, accuracy increased to 67.7% (reaching 92% with a larger model). For complex tasks needing four to five distinct skills, SAD boosted accuracy by 50%.

Interestingly, larger models can perform worse when not guided. In an unstructured setup, a 14-billion parameter model underperformed compared to the 7B version due to its tendency to fragment tasks into overly detailed steps. However, the introduction of SAD provided necessary focus, leading to a performance increase. This underscores the importance of aligning an agent’s approach with the specific vocabulary of tools, often proving more beneficial than utilizing larger models.

Another crucial finding pertains to token efficiency. The LLM-Direct baseline using the Qwen-Max model demonstrated that flooding a large LLM with tool names proved ineffective; despite its ability to break down tasks effectively, it only retrieved the proper tool category 21.1% of the time. SkillWeaver’s targeted method significantly outperformed this, achieving a dramatic reduction in context window consumption from about 884,000 tokens to approximately 1,160 tokens per query—a 99.9% decrease, which can lead to significantly lower API costs and quicker response times for users.

In contrast, the traditional ReAct baseline failed to achieve any decomposition accuracy, as it tends to reduce multi-step plans into isolated actions instead of creating a cohesive, multi-tool sequence.

Recommendations for Developers

While the researchers have not yet made the SkillWeaver code publicly available, their work utilizes commonly available tools that can be easily replicated.

Skill-Aware Decomposition (SAD) is the main innovation of this framework, blending clever prompt construction with a retrieval loop. The authors have made the prompt templates available in their paper, making it straightforward for developers to implement the technique using libraries such as LangChain, LlamaIndex, or even basic Python scripts.

For the retrieval component, SkillWeaver relies on the all-MiniLM-L6-v2, a significant open-source embedding model. Researchers observed improved accuracy when substituting a more robust encoder (BGE-base-en-v1.5), highlighting the potential for off-the-shelf solutions to enhance performance without fine-tuning. However, while a basic bi-encoder effectively places a relevant tool within the top candidates 70% of the time, it doesn’t consistently rank the optimal option first, achieving this only approximately 37% of the time. To address this, implementing an additional cross-encoder or LLM-based re-ranker may be necessary to re-order those top candidates accurately.

A requisite step is vectorizing the tool library and establishing a FAISS index beforehand, a minor obstacle as embedding and indexing all 2,209 skills in this benchmark took just 15 seconds. Once the index is created, retrieving tools from it incurs minimal latency, less than 15 milliseconds per query. For enterprise settings, synchronizing the tool index is a straightforward background process.

A current limitation of SkillWeaver is its lack of error recovery. While the framework efficiently plans a compatible DAG for execution, challenges remain in multi-step tool chains. For instance, if an API call fails in the second step, it affects the entire sequence. The research primarily focuses on routing and planning, leaving it to users to develop their own error recovery, fallback, and retry mechanisms for practical scenarios involving potential API timeouts or issues with output.