Local Ai Agents Microsoft 6

---
name: local-ai-agents
description: "'Build local-first AI agents that run entirely on a developer workstation with Microsoft Foundry Local and Qwen function-calling models. Covers Small Language Models (SLMs), the OpenAI-compatible local endpoint, sandboxed local tools, local RAG with Chroma, local MCP servers, hybrid cloud/local routing, and the privacy/cost/offline trade-offs. Based on Lesson 17 of AI Agents for Beginners. USE FOR: run an agent locally, offline agent, on-device agent, Foundry Local, Qwen function calling, local tool calling, local RAG, Chroma vector database, local MCP server, privacy-preserving agent, hybrid local and cloud agent, small language model agent, engineering assistant on my machine. DO NOT USE FOR: deploying agents to the cloud at scale (use deploying-scalable-agents / Lesson 16), building your first agent concept (Lesson 01), Foundry (cloud) hosted agents, GPU cluster / server-side..."
category: ai-agents-and-harness
source_repo: microsoft/ai-agents-for-beginners
source_path: "translations/en/.agents/skills/local-ai-agents/SKILL.md"
source_url: https://github.com/microsoft/ai-agents-for-beginners/blob/HEAD/translations/en/.agents/skills/local-ai-agents/SKILL.md
---
# Creating Local AI Agents with Foundry Local and Qwen

> Companion skill for [Lesson 17 – Creating Local AI Agents](../../../17-creating-local-ai-agents/README.md).
> Use it to help a learner build an agent that reasons, calls tools, and searches
> documentation entirely on their own machine — no cloud inference. Ground every
> recommendation in the lesson content and the runnable notebook.

## Triggers

Activate this skill when a learner wants to:
- Run an agent **fully on-device** for privacy, cost, or offline reasons.
- Serve a model locally with **Foundry Local** and connect via the OpenAI-compatible endpoint.
- Use a **Qwen function-calling** model to drive reliable local tool calls.
- Add **local RAG** (Chroma) or a **local MCP server**.
- Design a **hybrid** local/cloud routing strategy.

## Core mental model

An SLM trades breadth for privacy, cost, and offline operation. The winning
strategy: **let the SLM orchestrate and let tools do the heavy lifting.** The
model does not need to *know* the codebase — it needs to know when to call
`read_file` and `search_docs`. That plays to an SLM's strength (bounded decisions
like tool selection) and away from its weakness (broad knowledge, long multi-hop
reasoning).

## Why these specific pieces

- **Foundry Local** exposes an **OpenAI-compatible HTTP endpoint**, so cloud agent code transfers by changing only `base_url` (and using a local placeholder API key). It also auto-selects the best build (CPU/GPU/NPU) for the machine.
- **Qwen** models are trained for function calling and emit well-formed tool calls consistently — this is what turns a local *chat* model into a local *agent*.
- **Chroma** runs in-process and stores vectors on disk, so the whole RAG pipeline (embed → store → retrieve → reason) stays local.
- **MCP** is a transport, not a cloud service: an MCP server can run locally over `stdio`.

## Setup essentials

```bash
foundry model run qwen2.5-7b-instruct
foundry service status
```

```python
from foundry_local import FoundryLocalManager
from openai import OpenAI

manager = FoundryLocalManager("qwen2.5-7b-instruct")
client = OpenAI(base_url=manager.endpoint, api_key=manager.api_key)  # local placeholder
```

~8 GB RAM is a realistic minimum; a GPU/NPU helps but is not required.

## Key patterns to reproduce

Point the learner at the notebook
[`17-local-agent-foundry-local.ipynb`](../../../17-creating-local-ai-agents/code_samples/17-local-agent-foundry-local.ipynb):

- **Sandboxed tools**: every file tool resolves paths and rejects anything outside a single project root — even locally, a tool runs with the user's permissions.
- **Tool-calling loop**: register tools with the OpenAI tools schema, execute requested tools locally, feed results back, repeat until a final answer.
- **Local RAG**: upsert docs into a Chroma collection; `search_docs` returns top-k chunks.
- **Local MCP**: connect to a local server over `stdio`; scope it to a project directory and validate its outputs.

## Hybrid routing (local as one of the models)

| Situation | Where it runs |
|-----------|---------------|
| Sensitive data / offline | Local SLM |
| Simple, bounded task | Local SLM (cheap, fast) |
| Hard multi-hop reasoning on non-sensitive data | Cloud model |
| Cloud outage | Local SLM (graceful degradation) |

This mirrors the model-routing idea from Lesson 16, with the workstation as one
of the routes. Prefer designs that fall back to local so the agent degrades in
quality rather than failing outright.

## Guardrails for the assistant

- Keep every file/tool operation scoped to a sandboxed project directory.
- Do not send code or data to the cloud when the learner's stated goal is privacy/offline — keep the whole pipeline local.
- Set realistic expectations for SLM quality; lean on tools and RAG rather than the model's memorised knowledge.
- Note that Lesson 17 has **no** Foundry Responses endpoint, so the cloud smoke-test action does not apply — validate by running the notebook locally.

---

<!-- CO-OP TRANSLATOR DISCLAIMER START -->
**Disclaimer**:
This document has been translated using AI translation service [Co-op Translator](https://github.com/Azure/co-op-translator). While we strive for accuracy, please be aware that automated translations may contain errors or inaccuracies. The original document in its native language should be considered the authoritative source. For critical information, professional human translation is recommended. We are not liable for any misunderstandings or misinterpretations arising from the use of this translation.
<!-- CO-OP TRANSLATOR DISCLAIMER END -->

---

**Source:** [`microsoft/ai-agents-for-beginners`](https://github.com/microsoft/ai-agents-for-beginners) → `translations/en/.agents/skills/local-ai-agents/SKILL.md`