---
title: "Observability and Monitoring"
description: "Set up observability for AI features in production. Use AI Gateway analytics to track usage and costs, add structured logging for debugging, and set a budget with spend alerts before costs become incidents."
canonical_url: "https://vercel.com/academy/ai-summary-app-with-nextjs/observability-monitoring"
md_url: "https://vercel.com/academy/ai-summary-app-with-nextjs/observability-monitoring.md"
docset_id: "vercel-academy"
doc_version: "1.0"
last_updated: "2026-10-30T14:12:48.711Z"
content_type: "lesson"
course: "ai-summary-app-with-nextjs"
course_title: "Creating an AI Summary App with Next.js"
prerequisites:  []
---

<agent-instructions>
Vercel Academy — structured learning, not reference docs.
Lessons are sequenced.
Adapt commands to the human's actual environment (OS, package manager, shell, editor) — detect from project context or ask, don't assume.
The lesson shows one path; if the human's project diverges, adapt concepts to their setup.
Preserve the learning goal over literal steps.
Quizzes are pedagogical — engage, don't spoil.
Quiz answers are included for your reference.
</agent-instructions>

# Observability and Monitoring

# Observability and monitoring

Your AI features work. Users are happy. But something will break eventually—and you want to know before your users tell you. Observability means understanding what's happening in production without staring at logs all day.

## Outcome

Set up monitoring for your AI features using AI Gateway analytics, structured logging, and a budget with spend alerts.

## Fast track

1. Vercel → AI Gateway Overview → **Usage** and **Requests** to review requests, tokens, and cost
2. Add `console.log(JSON.stringify({ event, requestId, productSlug, duration, tokens }))` to AI functions
3. AI Gateway → API Keys → edit `ai-review-summary-key`: add a daily budget ($10 works) and turn on email alerts at 50%, 75%, and 100%

## Hands-on exercise 3.3

Add observability to your AI features:

**Requirements:**

1. Review AI Gateway analytics (requests, tokens, costs, errors)
2. Add structured logging to `summarizeReviews` and `getReviewInsights`
3. Add a budget with spend alerts to your AI Gateway API key
4. Test logging by generating some AI requests

**Implementation hints:**

- AI Gateway dashboard shows real-time and historical data
- Log request metadata (product slug, token count, duration)
- Budget spend alerts arrive by email
- Start with conservative thresholds and adjust based on real usage

## AI Gateway analytics

AI Gateway tracks everything automatically. No code changes needed.

**Find your analytics:**

1. Vercel Dashboard → **AI Gateway**
2. Open the **Overview**, which has **Usage** charts and **Requests** summaries

**What you'll see:**

```
Overview (Last 7 days)
────────────────────────────────────────
Total requests:     2,847
Success rate:       99.2%
Avg latency:        1,847ms
Total tokens:       2.1M
Estimated cost:     $6.32

Requests by model:
├─ anthropic/claude-sonnet-4.5    2,614 (91.8%)
├─ anthropic/claude-haiku-4.5      198 (7.0%)
└─ openai/gpt-5-mini                35 (1.2%)

Errors:
├─ 429 Rate Limited:    18
├─ 503 Service Error:    4
└─ Timeout:              1
```

**Key metrics to watch:**

| Metric       | Healthy       | Investigate | Alert     |
| ------------ | ------------- | ----------- | --------- |
| Success rate | Above 99%     | 95-99%      | Below 95% |
| Avg latency  | Under 2s      | 2-5s        | Over 5s   |
| Error rate   | Under 1%      | 1-5%        | Over 5%   |
| Daily cost   | Within budget | 2x budget   | 5x budget |

## Understanding the dashboard

**Requests by model:**
Shows how many requests each model handles over time. Look for:

- Unexpected spikes (bot traffic? viral post?)
- Sudden drops (deployment broke something?)
- Patterns (peak hours, quiet periods)

**Latency:**
The **Time to First Token** chart shows how long the first token takes on average, and the **Requests** summaries report P75 duration and P75 TTFT. Look for:

- P75 duration that jumps right after a deploy (something changed in the prompt or the model)
- Time to first token that climbs while traffic stays flat (a slower model or provider)

**Input/output token counts:**
Shows input vs output tokens. Look for:

- Input tokens >> Output tokens (normal for summarization)
- Unexpected token growth (prompts getting longer?)
- Spikes correlating with specific products (long reviews?)

**Spend:**
Shows what you spend over time. Look for:

- Steady growth (normal with traffic)
- Sudden jumps (fallbacks triggering? new feature?)
- Model distribution (are fallbacks firing more than expected?)

## Adding structured logging

AI Gateway tracks aggregate metrics. For debugging specific requests, add your own logging.

Update `lib/ai-summary.ts` to add logging while keeping the `"use cache"` directive from Lesson 3.1:

```typescript title="lib/ai-summary.ts"
import { generateText, Output } from "ai";
import { cacheLife, cacheTag } from "next/cache";
import { Product, ReviewInsights, ReviewInsightsSchema } from "./types";

export async function summarizeReviews(product: Product): Promise<string> {
  "use cache";
  cacheLife("hours");
  cacheTag(`product-summary-${product.slug}`);

  const startTime = Date.now();
  const requestId = crypto.randomUUID();

  console.log(JSON.stringify({
    event: "ai_request_start",
    requestId,
    function: "summarizeReviews",
    productSlug: product.slug,
    reviewCount: product.reviews.length,
    timestamp: new Date().toISOString(),
  }));

  const averageRating =
    product.reviews.reduce((acc, review) => acc + review.stars, 0) /
    product.reviews.length;

  const prompt = `Write a summary of the reviews for the ${product.name} product...`; // Your existing prompt

  try {
    const { text, usage } = await generateText({
      model: "anthropic/claude-sonnet-4.5",
      prompt,
      maxOutputTokens: 1000,
      temperature: 0.75,
    });

    const duration = Date.now() - startTime;

    console.log(JSON.stringify({
      event: "ai_request_success",
      requestId,
      function: "summarizeReviews",
      productSlug: product.slug,
      duration,
      inputTokens: usage?.inputTokens,
      outputTokens: usage?.outputTokens,
      totalTokens: usage?.totalTokens,
      timestamp: new Date().toISOString(),
    }));

    return text.trim();
  } catch (error) {
    const duration = Date.now() - startTime;

    console.error(JSON.stringify({
      event: "ai_request_error",
      requestId,
      function: "summarizeReviews",
      productSlug: product.slug,
      duration,
      error: error instanceof Error ? error.message : "Unknown error",
      timestamp: new Date().toISOString(),
    }));

    throw new Error("Unable to generate review summary. Please try again.");
  }
}
```

**What this logs:**

**Request start:**

```json
{
  "event": "ai_request_start",
  "requestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "function": "summarizeReviews",
  "productSlug": "mower",
  "reviewCount": 12,
  "timestamp": "2024-01-15T14:32:01.234Z"
}
```

**Request success:**

```json
{
  "event": "ai_request_success",
  "requestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "function": "summarizeReviews",
  "productSlug": "mower",
  "duration": 2341,
  "inputTokens": 847,
  "outputTokens": 89,
  "totalTokens": 936,
  "timestamp": "2024-01-15T14:32:03.575Z"
}
```

**Request error:**

```json
{
  "event": "ai_request_error",
  "requestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "function": "summarizeReviews",
  "productSlug": "mower",
  "duration": 5023,
  "error": "Rate limit exceeded",
  "timestamp": "2024-01-15T14:32:06.257Z"
}
```

**Why structured logging?**

- **Searchable** — Find all errors for a specific product
- **Parseable** — Tools like Vercel Logs, Datadog, or Axiom can parse JSON
- **Correlatable** — Request IDs link start → success/error
- **Measurable** — Track duration, tokens, and patterns over time

## Viewing logs in Vercel

**Find your logs:**

1. Vercel Dashboard → Your project
2. Click **Logs** tab
3. Filter by:
   - Level: **Error** (requests that logged with `console.error`)
   - Timeline: the past hour, a few days, or a custom range, depending on your plan's [log retention](https://vercel.com/docs/logs/runtime#limits)
   - Search: `ai_request_error` or `aquaHeat`

**Example log search:**

```
// Find all AI errors in the selected timeline
ai_request_error

// Find all requests for a specific product
aquaHeat
```

The search box matches text in the log message and the request path. It doesn't parse your JSON, so a query like `duration > 3000` searches for that literal text and finds nothing. To query fields like `duration`, send your logs to a tool like Datadog or Axiom with a [log drain](https://vercel.com/docs/drains).

## Setting up budgets and spend alerts

Picture a scraper that finds `/api/summary/mower` and hits it all weekend. Every request is a real Claude call. You find out Monday, from the invoice.

AI Gateway handles this with [budgets](https://vercel.com/docs/ai-gateway/observability-and-spend/budgets). A budget caps how much a team, project, API key, or team member can spend in a refresh period, and it can email you on the way to that cap.

Our app authenticates with `AI_GATEWAY_API_KEY`, so let's put the budget on that key. (A project budget only counts requests that use the project's OIDC token, and an API key's spend never counts toward a project.)

1. Vercel Dashboard → **AI Gateway** → **API Keys**
2. Find `ai-review-summary-key`, open its **···** menu, and select **Edit key**
3. Turn on the budget, enter a limit ($10 works for this course, any amount does), and choose a **daily** refresh period
4. Under **Email ... when usage reaches**, select **50%**, **75%**, and **100%**
5. Save

The key now shows its spend against the limit on the API Keys page, something like **$0.04 / $10 spent**. Here's what each threshold does:

```
Daily budget: $10 on ai-review-summary-key
├─ $5.00  (50%)  → Email to the key's creator
├─ $7.50  (75%)  → Email to the key's creator
└─ $10.00 (100%) → Email, then AI Gateway rejects further requests with a 402 until midnight UTC
```

The 50% and 75% emails are warnings only. At 100%, requests start failing with a `402` and `quota_for_entity_exceeded`, and that's when the fallback from 3.2 earns its keep: shoppers see the plain "Customer Reviews" card instead of an error page. The budget is a soft cap, so the request that crosses the line still completes and you can land slightly over.

**Error and deployment notifications (Vercel):**

Spend alerts only cover money. For "something broke" emails, each team member picks their own notifications:

1. Select your team, then open **Settings** in the sidebar
2. Under **Account**, select [**My Notifications**](https://vercel.com/docs/notifications#managing-notifications)
3. Turn on:
   - Deployment failures
   - Error anomalies (a high rate of `5xx` function invocations)
   - Usage alerts

These settings only change your own notifications, so teammates who want them need to turn them on too. If you're on Pro or Enterprise with Observability Plus, [Vercel Alerts](https://vercel.com/docs/alerts) can also send error and usage anomalies to Slack or a webhook.

**Start conservative:**
It's better to get too many alerts initially and tune them down than to miss something critical.

## Debugging production issues

When something goes wrong, here's how to investigate:

**1. Check AI Gateway dashboard**

- Error spike? What time did it start?
- Which model? (primary or fallback?)
- What error codes? (429, 503, timeout?)

**2. Check Vercel logs**

- Search for `ai_request_error`
- Filter to the timeframe
- Look for patterns (same product? same error?)

**3. Correlate with deployments**

- Did a deployment happen right before the errors?
- Check deployment logs for build issues

**4. Check provider status**

- [Anthropic Status](https://status.anthropic.com)
- [OpenAI Status](https://status.openai.com)
- If provider is down, your fallbacks should be handling it

**Common issues and causes:**

| Symptom              | Likely cause         | Fix                                                  |
| -------------------- | -------------------- | ---------------------------------------------------- |
| Sudden 429 spike     | Rate limit hit       | Add fallback model, implement backoff                |
| All requests failing | Bad API key          | Check env vars in Vercel                             |
| Slow responses       | Provider degradation | Fallbacks should kick in                             |
| Cost spike           | Cache not working    | Check `"use cache"` directive and `cacheLife` config |
| Token overflow       | Long reviews         | Truncate input or paginate                           |

## Production monitoring checklist

Before going live, verify:

- [ ] **AI Gateway analytics accessible**: Can you see requests, costs, errors?
- [ ] **Structured logging added**: JSON logs with request IDs and metadata
- [ ] **Budget with spend alerts set**: Get notified before bills surprise you
- [ ] **Error notifications on**: Know when things break
- [ ] **Fallbacks working**: Verified backup models are configured
- [ ] **Logs searchable**: Can find specific requests when debugging

## Try it

1. **Explore your AI Gateway dashboard:**
   - How many requests have you made?
   - What's your average latency?
   - Any errors?

2. **Add structured logging:**
   - Update `summarizeReviews` with the logging code
   - Generate a few summaries
   - Check Vercel logs for the JSON output

3. **Set up a budget with spend alerts:**
   - AI Gateway → API Keys → `ai-review-summary-key` → **Edit key**
   - Add a daily budget ($1 makes a quick test, and $1 is the minimum) with alerts at 50%, 75%, and 100%
   - Run `rm -rf .next`, restart `pnpm dev`, and open a product page so the cache misses, then check that the key shows its spend against the limit
   - Raise the limit back up when you're done testing, or a full day of traffic will hit the `402`

4. **Simulate an error (optional):**
   - Temporarily break your API key
   - Visit a product page
   - Check that error logs appear correctly
   - Fix the API key

## Commit

```bash
git add lib/ai-summary.ts
git commit -m "feat(observability): add structured logging to AI functions"
git push
```

## Done-when

- [ ] Explored AI Gateway analytics dashboard
- [ ] Understand key metrics (requests, latency, tokens, costs)
- [ ] Added structured logging to AI functions
- [ ] `ai-review-summary-key` has a budget with at least one spend alert threshold
- [ ] **My Notifications** has deployment failures and error anomalies turned on
- [ ] Know how to search logs in Vercel
- [ ] Understand debugging workflow for production issues

## What's next

Your AI features are observable. You'll know when things break, why they broke, and how much it's costing. Time to wrap up the course and review everything you've built.

***

**Sources:**

- [AI Gateway observability](https://vercel.com/docs/ai-gateway/observability-and-spend/observability)
- [AI Gateway budgets](https://vercel.com/docs/ai-gateway/observability-and-spend/budgets)
- [Notifications](https://vercel.com/docs/notifications)
- [Vercel Logs](https://vercel.com/docs/logs/runtime)
- [Logging Best Practices: An Engineer's Checklist](https://www.honeycomb.io/blog/engineers-checklist-logging-best-practices)


---

[Full course index](/academy/llms.txt) · [Sitemap](/academy/sitemap.md)