Varsni, Notes

Setting Up Agent Sandboxes in Kubernetes

Fri, 28 Aug 2026 00:00:00 GMT

Allowing an autonomous AI agent to run arbitrary shell commands is terrifying. Allowing it inside your primary production Kubernetes cluster is even worse.

The official documentation directs developers toward langchain-kubernetes and kubernetes-sigs/agent-sandbox. When deployed to a live cluster, that path quickly hits broken SDK wrappers and missing runtime components.

Here is how we achieved local, self-hosted code execution in Kubernetes without exposing cloud credentials or paying for a third-party sandbox SaaS.

The Architecture and Trust Boundary

The primary chat agent has read-only AWS access via MCP tools. The sandbox environment must be structurally incapable of accessing those cloud credentials. Rather than relying on soft prompt instructions, isolation is enforced directly through Kubernetes primitives.

Why We Chose Native Kubernetes Sandboxes

We evaluated several execution environments before committing to an in-cluster solution:

Solution	Verdict	Why It Was Ruled Out
AWS AgentCore	Rejected	Managed Bedrock dependency; not native to Kubernetes
E2B	Rejected	Requires bare metal or Firecracker; does not fit standard managed EKS or Minikube
Daytona	Rejected	BYOC model still relies on an external SaaS control plane
Modal / Runloop	Rejected	Adds external network hops and introduces vendor lock-in
k8s-agent-sandbox	Adopted	Native CRDs, fully self-hosted, integrates with standard Kubernetes NetworkPolicies

The Implementation: Adapter and Lifecycle

deepagents defaults to StateBackend, an in-memory virtual filesystem that cannot execute bash commands or persist files across turns.

Because the langchain-kubernetes package had method and parameter mismatches with the real k8s-agent-sandbox SDK, we built a 120-line custom adapter directly against BaseSandbox.

class _AgentSandbox(BaseSandbox):
    def __init__(self, sandbox) -> None:
        self._sandbox = sandbox

    @property
    def id(self) -> str:
        return self._sandbox.claim_name  # Stable across warm-pool rebinds

    def execute(self, command: str, *, timeout: int | None = None) -> ExecuteResponse:
        result = self._sandbox.commands.run(command, timeout=timeout or 60)
        output = result.stdout
        if result.stderr:
            output = f"{output}\n{result.stderr}" if output else result.stderr
        return ExecuteResponse(output=output, exit_code=result.exit_code)

    def upload_files(self, files: list[tuple[str, bytes]]) -> list[FileUploadResponse]:
        responses = []
        for path, content in files:
            try:
                self._sandbox.files.write(path, content)
                responses.append(FileUploadResponse(path=path, error=None))
            except Exception as e:
                responses.append(FileUploadResponse(path=path, error=str(e)))
        return responses

    def download_files(self, paths: list[str]) -> list[FileDownloadResponse]:
        responses = []
        for path in paths:
            try:
                content = self._sandbox.files.read(path)
                responses.append(FileDownloadResponse(path=path, content=content, error=None))
            except Exception as e:
                responses.append(FileDownloadResponse(path=path, content=None, error=str(e)))
        return responses

To maintain persistent workspaces across conversation turns without locking memory to a single API replica, claim IDs are stored in Redis:

async def get_sandbox_backend(conversation_id: str) -> _AgentSandbox | None:
    if not settings.sandbox_enabled or not settings.sandbox_warmpool_name:
        return None

    namespace = settings.sandbox_namespace or "default"
    redis_key = f"costops:sandbox:{conversation_id}"
    existing_claim_name = await redis.get(redis_key)

    client = SandboxClient()
    try:
        sandbox = None
        if existing_claim_name:
            try:
                sandbox = await asyncio.to_thread(client.get_sandbox, existing_claim_name, namespace)
            except SandboxNotFoundError:
                pass
        if sandbox is None:
            sandbox = await asyncio.to_thread(
                client.create_sandbox,
                warmpool=settings.sandbox_warmpool_name,
                namespace=namespace,
                shutdown_after_seconds=settings.sandbox_idle_timeout_seconds,
            )
    except Exception:
        return None

    if sandbox.claim_name != existing_claim_name:
        await redis.set(redis_key, sandbox.claim_name)
    return _AgentSandbox(sandbox)

The Gotcha: The Cross-Namespace RBAC Trap

The most confusing failure during implementation was an RBAC error masked as a broken controller.

SandboxClient connects using connection_mode="tunnel", which shells out to a local kubectl port-forward command. That command must resolve and forward to the sandbox-router running in the agent-sandbox-system namespace, not just resources in your application namespace.

Your API ServiceAccount requires two separate RBAC bindings:

# 1. In your application namespace (manage sandbox lifecycle)
- apiGroups: ["agents.x-k8s.io"]
  resources: ["sandboxes"]
  verbs: ["get", "list", "watch", "create", "delete", "patch"]
- apiGroups: ["extensions.agents.x-k8s.io"]
  resources: ["sandboxclaims"]
  verbs: ["get", "list", "watch", "create", "delete", "patch"]
- apiGroups: ["extensions.agents.x-k8s.io"]
  resources: ["sandboxwarmpools"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["pods", "pods/portforward"]
  verbs: ["get", "list", "watch"]
---
# 2. In agent-sandbox-system (allow port-forwarding to the router)
- apiGroups: [""]
  resources: ["services", "pods", "pods/portforward"]
  verbs: ["get", "list", "watch"]

Hardening and Safety Verification

Security relies on strict infrastructure policies:

No IAM Role Binding: The costops-sandbox ServiceAccount carries no AWS IAM annotations ([eks.amazonaws.com/role-arn](https://eks.amazonaws.com/role-arn)) and mounts no cloud tokens.
DNS-Only Egress: Network policies deny all outbound internet and VPC traffic except UDP/TCP port 53 for internal cluster DNS.
Pod Hardening: Containers run as UID 1000 with allowPrivilegeEscalation: false and all Linux capabilities dropped.

We verified the isolation directly inside a live sandbox pod:

kubectl -n costops exec -it <sandbox-pod> -- env | grep -i aws
kubectl -n costops exec -it <sandbox-pod> -- aws sts get-caller-identity

Both commands return empty results and connection timeouts.

Key Takeaways

Treat every agent sandbox as an untrusted workload by dropping Linux capabilities and enforcing non-root execution.
Do not rely on system prompts for security boundaries. Use Kubernetes NetworkPolicies and clean ServiceAccounts instead.
Store conversation-to-claim mappings in shared storage like Redis rather than local process memory.
Grant port-forward RBAC permissions in both the application namespace and the system routing namespace.
Keep the runtime execution container separate from your main backend image to avoid credential leakage.

Building a secure code execution sandbox for AI agents does not require expensive managed platforms or complex bare-metal virtualization. By combining kubernetes-sigs/agent-sandbox with strict NetworkPolicies, dedicated ServiceAccounts, and a lightweight adapter, you get full control over your infrastructure, complete data privacy, and robust security guarantees right inside your existing cluster.

Cost Governance for LLM Applications

Wed, 12 Aug 2026 00:00:00 GMT

Most teams confuse cost visibility with cost governance.

Seeing a sudden $15,000 spike on a line-item graph is visibility. Preventing a rogue agent loop from spending $15,000 in twenty minutes while you sleep is governance. Confusing the two is how a $2,000/month pilot turns into an eye-watering invoice that ruins your quarter.

When I finished building the tracing, evals, guardrails, and prompt versioning for an end-to-end LLMOps pipeline, cost was the final item marked "we'll figure it out later."

Here is what actually "figuring it out" looks like in production.

The Default Trapping: One Line Item, Zero Context

By default, managed platforms like AWS Bedrock bill LLM usage as a single consolidated line item. You watch the total cost climb, but you can’t answer critical post-mortem questions:

Which engineering team owns this burst?
Which specific agent script spiraled into an infinite retry loop?
Which bad prompt template inflated our token counts?

Solving this is the key task. Every optimization effort before attribution is just uneducated guessing.

The 4-Layer LLM Governance Framework

To move from passive monitoring to active governance, implement these four layers sequentially.

Layer 1: Granular Attribution

Application Inference Profiles (AIPs): Wrap Bedrock model ARNs inside tagged profiles. Route calls through these ARNs so metadata automatically flows to AWS Cost Explorer and Cost & Usage Reports (CUR). Tag by team or cost center—never per user (it won't scale).
IAM Principal Allocation: Enable caller identity data in CUR 2.0 to allocate costs directly to IAM roles and users without building custom middleware.
Per-Request Metadata: For sub-penny telemetry (like prompt/completion token ratios), pass a requestMetadata payload containing your tenantId or userId and query it via CloudWatch Logs Insights.

Layer 2: Proactive Alerting

Avoid single, org-wide budget caps—nobody takes ownership of them. Set project-level budgets using AWS Budgets paired with Cost Anomaly Detection.

Remember: An alert only tells you that damage has occurred. It does not stop the bleed.

Layer 3: Hard Enforcement (The Missing Layer)

This is the step most engineering teams skip until their first billing incident. An runaway loop burns thousands of dollars in minutes, long before your CloudWatch alarm evaluates or an email alert triggers.

Enforcement requires an in-path check. Run a lightweight proxy or API middleware backed by a fast state store (like Redis) that evaluates live token/dollar spend and active-gates incoming requests before they hit the upstream model provider.

Layer 4: Tactical Optimization

Once you know who is spending and where, apply the levers that actually impact the bottom line:

Context & Prompt Caching: Cache static system prompts and static RAG context blocks.
Dynamic Model Routing: Route simple sub-tasks (classification, extraction, rewrites) to lighter, cheaper models.
Batch Inference: Offload non-real-time jobs to async batch endpoints for ~50% cost reductions.
Provisioned Throughput: Commit to dedicated capacity only after baseline usage stabilizes.

Centralized Governance Gateways: Native vs. Proxy

AWS-native tools work well within a single ecosystem. But the moment your application makes calls across OpenAI, Anthropic, and Bedrock, native tagging falls apart.

You need a centralized Governance Gateway in front of your invocation pipeline.

Choosing Your Control Plane

AWS-Native Only (AIPs + CUR 2.0): Best if you are single-cloud, have modest scale, and don't want to maintain additional infrastructure proxy layers.
AWS-Native + LiteLLM / Portkey: Stay entirely on Bedrock infrastructure, but deploy an open-source gateway (LiteLLM) or managed control plane (Portkey) in front of it to gain fallbacks, prompt caching, and dynamic routing logic.
Multi-Provider Proxy (LiteLLM / Portkey / Helicone): Standardize all outgoing calls across multiple providers behind a single OpenAI-compatible interface.

Rule of thumb: Place a governance gateway in front of your LLM calls before integrating a second model provider. Refactoring call sites after going multi-provider is painful and prone to governance leaks.

Production Readiness Checklist

Before marking your LLMOps cost governance task as "Done," verify these items:

[ ] Every model invocation maps to a defined team, service, or project.
[ ] Project-level budgets are set with automated anomaly detection alerts.
[ ] In-path enforcement (circuit breakers/rate limits) exists for volatile workloads.
[ ] At least one optimization lever (caching, routing, or batching) is active in production.
[ ] Model and vector store pricing assumptions are updated against current rates.
[ ] A proxy/gateway layer is architecture-ready before multi-provider expansion.

The Takeaway

Cost dashboards are popular because they are easy to deploy. But visibility only tells you how much money you’ve already lost.

True governance should be structural: hard gates, real-time attribution, and dynamic routing that protect your infrastructure budgets automatically. Build visibility first, but just don't stop there.

Why I Chose AWS Batch for GPU Workloads (And What It Took to Make It Work)

Thu, 18 Jun 2026 00:00:00 GMT

Running GPU jobs in the cloud looks simple at first. It rarely is. Queues get slow. Costs creep up. Instances behave in ways you do not expect.

This post walks through how I set up GPU batch processing so it was fast, cheaper, and reliable. It also covers the three changes that actually moved the needle.

The problem: heavy GPU jobs, not constant traffic

My workload was not a steady stream. Jobs arrived in bursts. Sometimes a bunch at once. Sometimes nothing for a while. The compute needs were real and big. They just were not 24/7.

That left a few bad choices:

Keep a GPU instance running all the time and burn money when idle.
Start EC2 instances by hand for each job and take on all the ops work.
Write a custom queue with my own logic for retries and concurrency.

None of that felt right. I needed something that could sit quiet when no work was there. It should wake up on new jobs, run things in order, handle retries, and not need manual care.

Why AWS Batch made sense

AWS Batch is built for this kind of pattern. It manages the lifecycle of compute for you. Instances come up when jobs arrive. They go away when there is no work.

For GPU jobs this works well. Batch supports GPU instance families and runs everything as containers. Jobs run in the same environment every time. No custom setup per run.

A few parts were key.

Job queues with real controls. Batch gives you job queues with priorities, retry settings, and job dependencies. No polling scripts. No custom schedulers. No guessing what ran.

Compute environments with launch templates. You can attach a launch template to a compute environment. This gives control over userdata, EBS volumes, and ECS agent config. That control became important later.

GPU resource targeting. In the job definition you say how many GPUs a job needs. Batch handles the scheduling. For my case the g5 family worked well. Enough GPU memory and good throughput.

Step Functions for orchestration

Batch runs jobs. It does not decide what should run next.

For that I used AWS Step Functions. Real workloads are pipelines. They have steps and branches. They need retries and error handling.

Step Functions gave me a state machine to model that:

Run jobs in sequence and pass outputs forward.
Branch on success or failure without custom code.
Handle retries with backoff at the workflow level.
See where a pipeline is stuck or moving.

Batch handled how jobs run. Step Functions handled when they run and in what order.

Where things broke and how we fixed them

The setup on paper looks clean. In practice the first version was slower and more expensive than it should have been. Some instance behavior was also not obvious until we looked closer.

Here are the three main problems. And what fixed them.

Problem 1: Image pulls slowed every job

Each job pulled the Docker image from ECR before it could start. The image was large. It had CUDA, ML libraries, and other tools. The pull took time. For a single job it was already noticeable. For a busy queue it was painful.

The root cause was ECS image cleanup.

By default the ECS agent on the Batch instance removes Docker images from time to time. This frees disk space. That is fine for long running services. For batch GPU jobs it hurts. You actually want the image to stay on disk so the next job can reuse it.

The fix was to disable image cleanup.

Set ECS_IMAGE_CLEANUP_ENABLED=false in /etc/ecs/ecs.config.

The clean way to do this is in userdata inside the launch template. The instance boots, reads the config, and starts ECS with cleanup turned off. The image then stays cached.

After that change the first job on a new instance still pulls the image. Jobs that follow on the same instance skip the pull and start compute right away.

Problem 2: Scale-down removed the cache benefit

Caching helps only if the instance stays alive long enough. Our compute environment scaled down fast. When a job finished the instance often shut down. The next job then got a fresh instance and a fresh pull. We were back to cold starts.

The fix was to adjust scale-down behavior.

Instead of killing instances as soon as there were no jobs, we kept them alive for a short window. During this time the image stayed cached. If new jobs arrived, they landed on a warm instance and skipped the pull.

This is a tradeoff. You pay for some idle time. In return you get lower latency and better throughput during bursts. For workloads that come in clusters this works well.

I also wanted this to be flexible. So the scale-down window is configurable from the app.

If I expect a quiet phase I reduce the window. Idle instances shut down fast.
If I expect a spike I increase the window. Instances stay warm for longer.

The control lives next to the usage signal. So the cost of warm instances appears only when it brings value.

Problem 3: Multi-GPU instances looked wasteful

A g5.12xlarge has four GPUs. A single GPU job uses one. At first this looked bad. You pay for four GPUs. One is busy. Three are idle.

The simple view is that multi-GPU instances are not good for single GPU jobs. With caching and warm instances that view changes.

Once you:

Keep images cached, and
Keep instances warm for a short time

a multi-GPU node turns into a throughput boost.

The first job on that node pays the warmup cost. Instance launch plus image pull. After that three GPUs are still free. The next three jobs can start right away.

No new instances. No extra image pulls. Just scheduling and compute.

For bursty workloads this is a big shift. You do not pay the warmup cost four times on four machines. You pay once and reuse it across all GPUs on that node.

How it behaved after the fixes

With all three changes in place the system behaved very differently.

The first job on a new instance still has a warmup cost.
Jobs after that on the same instance start at compute.
The scale-down window keeps instances alive just long enough to catch bursts.
On multi-GPU nodes the warmup cost spreads across several jobs.

Queues moved faster. The cost per job dropped. Bursts felt smoother with less random latency.

The bigger takeaway

AWS Batch gives you managed compute and native job queues. Step Functions gives you a solid way to build pipelines. The jump from “it works” to “it works well” came from three small but important tweaks:

Turn off image cleanup so large GPU images stay cached.
Tune scale-down so instances stay warm when bursts are likely.
Use multi-GPU instances for throughput once caching is in place.

These details do not stand out in the basic docs. In practice they made a clear difference.

Designing Reliable Agent Workflows

Mon, 15 Dec 2025 06:21:01 GMT

Hi there!

In this post, I'll show you how to design reliable workflows that improve agent performance. We'll cover how to fix vague roles, enforce strict inputs/outputs, and manage state explicitly.

Why agents fail

Most failures come from design, not intelligence.

Vague roles confuse agents. If one agent has to plan, do the work, and judge the result, it often loses focus and goes off track.
Loose inputs and outputs hide problems. Free-form text makes it easy to sound right while being wrong.
Missing state causes loops. If the system doesn’t track what ran and what failed, it repeats the same mistakes.
No guardrails means risky actions slip through. With unchecked access, one bad step can cause real damage fast.
Weak error handling makes failures messy. Blind retries and late escalation lead to quiet breakage or endless retries.

When do you need this?

Not every task needs three agents and a state machine. If you just need to "summarize this text" or "write a haiku," a single prompt is fine.

Use this structured workflow when the task is:

Complex and multi-step: Requires planning, research, and synthesis (e.g., "Plan a travel itinerary").
High risk: Involves sending emails, making purchases, or modifying data.
Long-running: Needs to run for minutes or hours without losing context.
Data-heavy: Involves querying databases, analyzing large datasets, or cross-referencing multiple sources.

A bad workflow (what not to do)

Goal: Pull product metrics, analyze them, and email a summary to the team.

Design: A single “analyst agent” with a broad prompt:

“You are an expert analyst. Gather metrics, analyze them, write a summary, and email the team.”

What could go wrong

The workflow crumbles quickly. Wrong tables are queried because the prompt is vague, and metrics are invented when data access fails. The write-up might sound professional but contains fake numbers. Worse, emails often go to wrong recipients due to parsing errors. When failures occur, the agent simply retries with new wording, entering an endless loop.

Root causes

This failure stems from a lack of structure. There is no split between planning, execution, and validation. Without clear input/output formats or success rules, the agent guesses. Explicit state like “data fetched” or “email sent” is missing, and there is no write protection or approval for emails. Instead of structured failure, the only strategy is "try again."

A better workflow (step by step)

Use three small agents, each with one job:

Planner agent – decides steps and defines inputs and outputs
Executor agent – runs steps with limited access
Validator agent – checks outputs and blocks unsafe actions

1. Define clear inputs and outputs

Every step needs a small contract:

Input format: typed JSON fields
Output format: JSON with a fixed schema
Success condition: true/false with a reason

Avoid free-form text at the interface level. Natural language can exist inside fields, but the structure must be strict.

Example: Fetch weekly metrics

Input

{ "week_start": "ISO date", "metrics": ["active_users", "revenue"] }

Output

{ "success": boolean, "data": { "active_users": number, "revenue": number }, "errors": [string] }

Success rule: Success is true, all fields exist, and values fall within expected ranges.

2. Manage state explicitly

Store state in a place all agents can read. Keep track of done steps (outputs and timestamps), failed attempts (errors with codes), pending items, and notes (links/assumptions).

Never rely on only chat memory. Each agent reads state at the start and writes state at the end. Use IDs for steps and outputs so everything is traceable.

3. Guardrails by default

Security should be baked in. Start with read-only access scopes, granting write access only when needed. Enforce tool limits by allowing only whitelisted tools with strict schemas. Require human approval for high-risk actions like emails or external calls. Implement rate limits to prevent retry storms, and use content rules to block PII leaks or unsafe text.

4. Failure handling that ends cleanly

Failures should be expected and designed for. Limit retries by error type with backoff, and allow only one change per retry never rewrite the whole plan. Escalate after N failures, stopping to hand off context. Ensure a safe exit where, if assumptions break, the workflow terminates without loops or made-up data.

Putting it together: a concrete flow

A) Planning

The planner receives the high-level goal and converts it into a structured checklist.

Input: task = "Weekly metrics summary email"

Output:

{
  "plan_id": "plan_123",
  "steps": [
    { "step": 1, "action": "fetch_metrics", "params": { "range": "last_7_days" }, "status": "pending" },
    { "step": 2, "action": "analyze_trends", "params": { "focus": ["growth", "churn"] }, "status": "pending" },
    { "step": 3, "action": "draft_email", "params": { "template": "weekly_summary" }, "status": "pending" }
  ]
}

B) Execution (read-only first)

The executor picks up pending steps one by one. It reads the shared state, calls the necessary tools, and writes the output back to state.

It first fetches the metrics, then analyzes trends, and only then drafts the email using the fetched data.

Action: draft_email Result:

{
  "step": 3,
  "action": "draft_email",
  "success": true,
  "output": {
    "subject": "Weekly Metrics: +5% Growth",
    "body": "Hi Team, Active users are up to 10,500...",
    "recipients": ["team@company.com"]
  },
  "timestamp": "2025-12-15T10:05:00Z"
}

(Note: No emails are sent yet. The executor only produces data and drafts.)

C) Validation

The validator reviews the drafted email against safety rules and data mapping.

Check: draft_email output vs fetch_metrics data Rules:

Data Mapping: Do the numbers in the email body match the fetched JSON data?
Safety: No PII (Personally Identifiable Information) or leaked secrets.
Policy: Recipients match the internal allow-list etc.

Validation Result:

{
  "validation_id": "val_456",
  "plan_id": "plan_123",
  "validated_step": 3,
  "approved": true,
  "risk_level": "low",
  "reason": "Draft matches source data. No sensitive info found."
}

D) Approval and sending

The final gatekeeper decides whether to proceed based on the validation result.

Low risk: Auto-send via the API.
High risk (or validation failed): Trigger a human approval request with a diff of the proposed email.

Best Practices for Reliability

1. Implementation Tips

Give the agent a "Task Tracker" tool Instead of asking the agent to remember the plan in its context window, give it a tool to read and write to a persistent checklist.

Map the work: The agent calls create_tasks([...]) to lock in the plan.
Track progress: It calls complete_task(id) after every success.
Ensure completion: Before finishing, it calls get_pending_tasks() to verify nothing was missed.

This externalizes memory, making the agent robust against distractions and context limits.

Treat prompts like code Prompts are logic. Version control them, review changes in PRs, and test them against golden datasets. Don't let prompts live in database strings or loose variables.

Prefer deterministic tools Reliability drops when tools are fuzzy. Prefer structured APIs over web scraping. If you must parse text, wrap it in a dedicated "extraction" agent that outputs JSON.

2. Common pitfalls to avoid

Pitfall	Fix
“One agent is simpler”	It isn’t. Split roles.
“We’ll pass chat history”	Persist real state instead.
“We need creative freedom”	Keep creativity inside strict containers.
“Retries will fix it”	Retry less, with better signals.
“Humans slow things down”	Humans prevent costly mistakes.

3. Reliability Checklist

Clear roles: Planner, Executor, Validator
Contracts for every step
Explicit state, no chat memory
Read-only by default, limited writes
Structured failures and safe exits
Traceable IDs and logs
Data sanity checks
Per-step access control
Tests for bad inputs and edge cases
Regular audits of failures and near misses

Closing thought

Reliable agent systems are built like reliable software systems. Separation of concerns, Strict interfaces, Explicit state and Careful access.

Models help, but workflows carry the weight. Design the workflow assuming it will fail and it won't surprise you when it tries.

Thanks for reading! 🎉

If you're building agent workflows, I hope these patterns help you create more reliable systems. Remember: good design beats clever prompts every time. If you have questions or want to share your own reliability patterns, feel free to reach out!

MCP Explained: How AI Tools and Agents Work Together

Wed, 02 Jul 2025 11:06:17 GMT

What is MCP?

MCP stands for Model Context Protocol.
It's a new open standard that enables AI agents to talk to tools and APIs and share information securely, in real time.

Think of it like this:

LLM + Tools = Agent.
MCP is what makes that “+ Tools” part easier, flexible, and secure.

Think of MCP Like a USB-C Cable

Let’s break it down with a simple analogy:

You have a laptop (Agent) and a smartphone (Data Source).
You want to transfer photos from your phone to your laptop.
So, you use a USB-C cable (MCP) to connect them.
The cable doesn’t care what brand your devices are as long as they speak the same protocol, they can share data seamlessly.

Now imagine the same idea, but with AI tools:

The laptop is an AI agent like Amazon Q or Claude.
The phone is a data source like a database, API, or local file.
The USB-C cable is MCP the protocol that lets these tools talk to each other and get things done.

MCP acts like the connector between agents and the tools they need to be useful in real-world workflows.

A Real-World Use Case

Let’s say your company has important business data in a PostgreSQL database. You’d like a GenAI assistant to read from that data and generate reports or automate tasks.

With MCP, this becomes very doable:

Set up a PostgreSQL MCP server to expose that data.
Connect an agent (like Amazon Q) to it using MCP.
The agent can now securely query the database, generate insights, or automate reporting all while respecting any permissions you configure on the MCP server.

Basically, MCP works like an API layer between your tools and the agent, but with more flexibility and control.

Components of MCP

Let’s look at the core pieces that make MCP work:

MCP Server:

A service that exposes a data source or tool like a database, CLI, or document for an agent to use.

MCP Client:

Usually the AI agent or model (like Claude or Amazon Q) that initiates the interaction by sending requests.

MCP Protocol:

The communication standard that defines how the client and server talk. Think of it like the “language” they both understand.

MCP Host:

Where the MCP client or server runs. This might be:

Your local machine (e.g., Claude Desktop, Cursor IDE)
An app or agent hosted in the cloud using frameworks like LangChain or Strands

Wrapping It All Up

As you can see, MCP acts like the API/backend for agents, making them capable of interacting with the real world. Plugging in or removing tools could become as easy as installing a plugin or extension (like in Cursor IDE).

It’s early days, but the potential is huge and growing fast.

🚀 Want to Try It Yourself?

If you’re curious to explore MCP, here are some next steps you can try out:

Set up your own MCP server using the open-source fastmcp template: fastmcp on PyPI
Connect it to a database, folder, or API
Then test how an agent like Claude Desktop or Amazon Q can interact with it

Thanks for reading! 🎉

If this post helped clarify what MCP is all about, feel free to share it, or let me know what you’re building with it. Excited to see where this goes!

Build Lightweight MCP Servers Easily with FastMCP

Wed, 02 Jul 2025 11:03:12 GMT

Hi there!

In this post, I'll show you how to create MCP servers easily with FastMCP. To know the basics of MCP, you can read my previous post here.

What is FastMCP?

FastMCP is a new open-source library that makes it easy to create MCP servers. It's built on top of the FastAPI framework and uses the latest version of the MCP protocol.

How to install FastMCP?

pip install fastmcp

How to create an MCP server with FastMCP?

Creating an MCP server with FastMCP is straightforward. Let's build a simple example that exposes a file system tool and a resource.

Basic Setup

First, create a new Python file for your MCP server:

from fastmcp import FastMCP

# Create a new MCP server instance
mcp = FastMCP("My MCP Server")

That's it! You've created an MCP server. Now let's add some functionality.

Adding Tools

Tools are functions that the MCP client (like an AI agent) can call. Let's add a tool to read files:

from fastmcp import FastMCP
import os

mcp = FastMCP("File System Server")

@mcp.tool()
def read_file(file_path: str) -> str:
    """Read the contents of a file from the local filesystem.
    
    Args:
        file_path: The path to the file to read
        
    Returns:
        The contents of the file as a string
    """
    if not os.path.exists(file_path):
        return f"Error: File '{file_path}' not found"
    
    with open(file_path, 'r', encoding='utf-8') as f:
        return f.read()

Adding Resources

Resources are data sources that agents can access. Let's add a resource that lists files in a directory:

@mcp.resource("file://directory/{directory}")
def list_directory(directory: str) -> str:
    """List all files in the specified directory.
    
    Args:
        directory: The directory path to list
        
    Returns:
        A formatted string listing all files
    """
    if not os.path.isdir(directory):
        return f"Error: '{directory}' is not a valid directory"
    
    files = os.listdir(directory)
    return "\n".join(f"- {f}" for f in files)

Adding Prompts

Prompts are reusable templates that help guide the agent's behavior:

@mcp.prompt()
def analyze_codebase() -> str:
    """Prompt template for analyzing a codebase.
    
    Returns:
        A prompt string for code analysis
    """
    return """Analyze the codebase structure:
1. Identify the main entry points
2. List all dependencies
3. Find potential security issues
4. Suggest improvements"""

Complete Example

Here's a complete MCP server that combines all these features:

from fastmcp import FastMCP
import os
import json
from datetime import datetime

mcp = FastMCP("Development Tools Server")

@mcp.tool()
def read_file(file_path: str) -> str:
    """Read the contents of a file from the local filesystem."""
    if not os.path.exists(file_path):
        return f"Error: File '{file_path}' not found"
    
    try:
        with open(file_path, 'r', encoding='utf-8') as f:
            return f.read()
    except Exception as e:
        return f"Error reading file: {str(e)}"

@mcp.tool()
def write_file(file_path: str, content: str) -> str:
    """Write content to a file on the local filesystem.
    
    Args:
        file_path: The path where to write the file
        content: The content to write
    """
    try:
        os.makedirs(os.path.dirname(file_path), exist_ok=True)
        with open(file_path, 'w', encoding='utf-8') as f:
            f.write(content)
        return f"Successfully wrote to '{file_path}'"
    except Exception as e:
        return f"Error writing file: {str(e)}"

@mcp.tool()
def get_file_info(file_path: str) -> str:
    """Get metadata about a file.
    
    Args:
        file_path: The path to the file
        
    Returns:
        JSON string with file information
    """
    if not os.path.exists(file_path):
        return json.dumps({"error": "File not found"})
    
    stat = os.stat(file_path)
    info = {
        "path": file_path,
        "size": stat.st_size,
        "modified": datetime.fromtimestamp(stat.st_mtime).isoformat(),
        "is_file": os.path.isfile(file_path),
        "is_dir": os.path.isdir(file_path)
    }
    return json.dumps(info, indent=2)

@mcp.resource("file://directory/{directory}")
def list_directory(directory: str) -> str:
    """List all files and subdirectories in the specified directory."""
    if not os.path.isdir(directory):
        return f"Error: '{directory}' is not a valid directory"
    
    items = []
    for item in os.listdir(directory):
        item_path = os.path.join(directory, item)
        item_type = "directory" if os.path.isdir(item_path) else "file"
        items.append(f"{item_type}: {item}")
    
    return "\n".join(items) if items else "Directory is empty"

@mcp.prompt()
def code_review_prompt() -> str:
    """Generate a prompt for code review tasks."""
    return """Review the provided code and check for:
1. Code quality and best practices
2. Potential bugs or errors
3. Security vulnerabilities
4. Performance optimizations
5. Documentation completeness

Provide specific, actionable feedback."""

Running Your Server

To run your MCP server, add this at the end of your file:

if __name__ == "__main__":
    mcp.run()

Then run it with:

python your_server.py

Or use the FastMCP CLI:

fastmcp serve your_server.py

Connecting to an Agent

Once your server is running, you can connect it to MCP-compatible agents like:

Claude Desktop: Update claude settings and add your server to the claude_desktop_config.json file
Amazon Q: Configure it through Q CLI
Cursor IDE: Add it to your cursor IDE MCP settings

Best Practices

Error Handling: Always wrap file operations in try-except blocks and return clear error messages
Type Hints: Use Python type hints in your function signatures for better documentation
Docstrings: Write clear docstrings that explain what each tool/resource does
Security: Be careful about file paths - validate and sanitize inputs to prevent directory traversal attacks
Resource Limits: Consider adding limits on file sizes or operation timeouts for production use

Advanced Features

FastMCP also supports:

Async tools: Use async def for non-blocking operations
Streaming responses: For large data transfers
Custom schemas: Define complex input/output types
Middleware: Add authentication, logging, or rate limiting

Example: Database MCP Server

Here's a more advanced example that connects to a database:

from fastmcp import FastMCP
import sqlite3
from typing import List, Dict

mcp = FastMCP("Database Server")

@mcp.tool()
def query_database(db_path: str, query: str) -> str:
    """Execute a SQL query on a SQLite database.
    
    Args:
        db_path: Path to the SQLite database file
        query: SQL query to execute
        
    Returns:
        JSON string with query results
    """
    try:
        conn = sqlite3.connect(db_path)
        conn.row_factory = sqlite3.Row
        cursor = conn.cursor()
        cursor.execute(query)
        
        rows = cursor.fetchall()
        results = [dict(row) for row in rows]
        
        conn.close()
        return json.dumps(results, indent=2)
    except Exception as e:
        return json.dumps({"error": str(e)})

@mcp.resource("db://tables/{db_path}")
def list_tables(db_path: str) -> str:
    """List all tables in a SQLite database."""
    try:
        conn = sqlite3.connect(db_path)
        cursor = conn.cursor()
        cursor.execute("SELECT name FROM sqlite_master WHERE type='table'")
        tables = [row[0] for row in cursor.fetchall()]
        conn.close()
        return "\n".join(f"- {table}" for table in tables)
    except Exception as e:
        return f"Error: {str(e)}"

Wrapping Up

FastMCP makes it incredibly easy to build MCP servers. With just a few decorators, you can expose tools, resources, and prompts that AI agents can use to interact with your systems.

The key advantages of FastMCP:

Simple API: Decorator-based design that's intuitive to use
FastAPI Foundation: Built on FastAPI, so you get async support and great performance
Type Safety: Leverages Python type hints for better validation
Active Development: Regularly updated with the latest MCP protocol features

Next Steps

Build your own MCP server for a specific use case (database, API, file system, etc.)
Test it with Claude Desktop or another MCP client
Share your server with the community or use it in your own agent workflows

Thanks for reading! 🎉

If you build something cool with FastMCP, I'd love to hear about it. Happy building!

Multitenancy in AWS with account per tenant model

Wed, 08 Nov 2023 06:21:01 GMT

Multi-tenancy in the cloud is akin to running multiple, separate offices within a shared workspace. In technical terms, it means deploying a single software application or service that serves numerous users or clients, all isolated from one another. This approach maximises resource utilisation and cost-efficiency by sharing infrastructure and codebase.

MLT Architectures

Single Database, Separate Schemas: In this approach, a single database is used to store data for all tenants, but each tenant has its own schema within the database. This architecture provides strong isolation between tenants while allowing efficient data management. It's suitable for scenarios where tenants have similar data structures but need data separation.

Database-per-Tenant: Each tenant has its own dedicated database instance. This approach offers the highest level of data isolation but can be resource-intensive and costly. It's suitable for scenarios where strict data separation is required.

Micro-services with API Gateway: Tenants are served by separate microservices, each with its own database. An API gateway manages tenant-specific routing and authentication. This approach provides flexibility, scalability, and strong isolation.

Account level isolation: A multi-tenancy architecture with separate AWS accounts involves assigning each tenant or customer their own dedicated AWS account. This architecture ensures robust isolation, security, and scalability between tenants. Each tenant manages their AWS resources independently, with IAM policies enforcing access control. Network isolation is achieved using separate VPCs. Monitoring, billing, and disaster recovery are handled individually for each tenant, allowing for customised solutions. This approach is ideal for scenarios where data separation and security are paramount, such as SaaS applications or managed service providers.

The Approach we will be looking today is Account Level Isolation.

Multi-tenancy using separate AWS accounts is a reliable method that ensures strong isolation and security between tenants. Here’s a high-level overview of a multi-tenancy architecture with separate AWS accounts:

Tenant Isolation

We assign each tenant its own AWS account. Tenant resources, data, and configurations remain completely isolated from each other due to this separation.

Identity and Access Management (IAM)

AWS Identity and Access Management (IAM) controls access and permissions to each tenant’s AWS resources. IAM policies ensure granular access control, allowing tenants to access only their own resources.

Networking – Virtual Private Cloud (VPC) isolation

Each tenant typically has its own VPC, which provides network isolation. This prevents tenants from interacting directly with each other’s resources. Each tenant receives outbound internet access through Network Address Translation (NAT) gateways or instances.

Resource Scaling

Each tenant’s AWS account can independently scale its resources based on its own requirements and usage patterns. This ensures that one tenant’s resource demands do not impact others.

Monitoring and Logging

We use AWS CloudWatch and CloudTrail to monitor, log, and audit tenant-specific activities. This facilitates troubleshooting and security monitoring.

Backup and Disaster Recovery

Each tenant can implement backup and disaster recovery strategies based on their specific needs and compliance requirements.

Billing and Cost Allocation

You can use AWS Organizations to track individual tenant usage by providing cost allocation tags and consolidating billing across all tenant accounts.

Tenant Onboarding and Offboarding

You can set up automated processes for tenant onboarding and offboarding. When adding or removing tenants, you can automate the provisioning and de-provisioning of resources and permissions.

Cross-Tenant Functionality

You can set up a shared services or central admin account to manage functions that need to span across tenants, such as global authentication or reporting.

This architecture offers a strong separation between tenants, making it ideal for scenarios requiring security and data isolation, such as SaaS applications or managed service providers. It allows tenants to control their resources and configurations while centralizing administrative tasks.

When properly implemented, it offers a scalable, secure, and easily manageable multi-tenancy solution in the AWS cloud.

Guide to AWS Certified SysOps Administrator – Associate

Tue, 07 Nov 2023 09:04:41 GMT

Becoming an AWS Certified SysOps Administrator – Associate is a significant step in showcasing your expertise in managing and operating systems on the AWS platform. This guide will provide you with a roadmap to navigate through the preparation process successfully.

First of all you will need about 6 months hands-on experience in AWS to get an idea on its services. Next take a look at the AWS whitepapers to see what is covered in the exams. To start with your learning you have lots of courses available in udemy. The course that I found helpful was the one from Stephane Maarek. Here are the links to the resources that I used :

Course and Practice Tests

Embarking on the journey to become an AWS Certified SysOps Administrator – Associate requires a combination of hands-on experience, learning, and practice tests. The resources mentioned in this guide are meant to guide you through this process efficiently.

Best of luck on your certification journey!! 🤞