Observability
Volume 5 — Platform Engineering
Observability is the ability to understand what's happening inside a running system from its external outputs — without observability, debugging production means guessing. This chapter covers the standard model and how it's implemented across Traxs.
1. The Three Pillars
| Pillar | Answers | Traxs example |
|---|---|---|
| Logs | What happened, in detail, at a specific point in time | Seq, App Service Log Stream |
| Metrics | How is the system performing, in aggregate, over time | Request counts, response times, error rates |
| Traces | How did a single request flow through multiple components | A request spanning API → database → external service, correlated end to end |
They're complementary, not redundant — a metric can tell you error rates spiked at 2:14 PM, a trace can show which specific downstream call was slow for a given request, and a log gives you the exact exception and stack trace behind it. Relying on only one pillar means always missing part of the picture.
2. Structured vs. Unstructured Logging
Unstructured:
Error processing leave request 42 for user 123: connection timeout
Structured:
{
"timestamp": "2026-07-23T14:02:11Z",
"level": "Error",
"message": "Error processing leave request",
"leaveRequestId": 42,
"userId": 123,
"exceptionType": "TimeoutException",
"service": "roundtrip-api"
}
A structured log is queryable directly — "show me every error for userId: 123" is a real query against real fields, not a regex hoping the message format never changes. An unstructured log requires string-matching over unpredictable free text, which is the difference between a five-second search during an incident and a much longer one.
3. Seq
Seq is the structured logging platform used across Traxs — application logs are sent to it as structured events rather than plain text, giving searchable, filterable access to production log data well beyond what raw log files allow.
Log.Information("Leave request {LeaveRequestId} approved by {ApproverId}", request.Id, approverId);
The {LeaveRequestId} and {ApproverId} placeholders aren't just string interpolation — they become distinct, queryable fields in Seq, exactly like the structured JSON example above, without the developer needing to construct the JSON shape by hand.
Seq is configured conditionally on a valid URI in the Traxs setup — if the Seq endpoint isn't configured or isn't reachable, logging falls back rather than crashing the application over a missing observability dependency. Worth checking specifically if structured logs seem to be missing entirely rather than just incomplete — confirm the Seq connection itself is actually configured and reachable from the environment in question before assuming the application simply isn't logging.
4. App Service Log Stream
az webapp log tail --name roundtrip-api --resource-group traxs-prod
Log Stream shows an App Service's live stdout/stderr and platform-level diagnostic events — this is the source of truth for server-side exceptions, and specifically the tool that resolves the CORS-masking-a-500 pattern covered next.
5. The CORS-Masks-500s Pattern
This is one of the most consistently misleading debugging patterns in the whole stack, and it's worth internalizing precisely because the symptom points investigation in the wrong direction by default:
Browser console shows:
Access to fetch at 'https://api.roundtrips.app/...' from origin 'https://roundtrips.app'
has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present
Actual root cause:
The API returned a 500 Internal Server Error, and because it crashed before
reaching the middleware that adds CORS headers, the browser never received
those headers at all — and reports the *absence of CORS headers* as a CORS
error, rather than surfacing the underlying 500.
The browser's CORS error is real in the sense that the headers genuinely weren't present — but the reason they weren't present is an unhandled server exception, not a CORS misconfiguration. Chasing CORS configuration in response to this error, when the API is actually throwing before it gets anywhere near the CORS middleware, is a dead end.
The fix in practice: whenever a CORS error appears in the browser console, check App Service Log Stream first, before touching any CORS configuration at all. If there's a 500 in the server logs at the same timestamp, that's the actual problem, and fixing it will very likely make the "CORS error" disappear on its own without a single CORS setting having changed.
6. Correlation IDs and Basic Tracing
A correlation ID is a unique identifier generated at the start of a request and threaded through every log line, every downstream call, and every component that touches that request — the mechanism that turns a pile of unrelated log lines into one traceable story.
{ "correlationId": "a1b2c3d4", "message": "Leave request received", "step": "validation" }
{ "correlationId": "a1b2c3d4", "message": "Checking PTO balance", "step": "business-rule" }
{ "correlationId": "a1b2c3d4", "message": "Leave request approved", "step": "persistence" }
Searching Seq (or any structured log store) for one correlation ID reconstructs the entire request's path through the system — without it, correlating "the validation log at 2:14:11" with "the persistence log at 2:14:12" as the same request is guesswork, especially under any real concurrent load where many requests interleave in the raw log stream.
ASP.NET Core generates a TraceIdentifier per request automatically — including it explicitly in structured log statements (rather than relying on it existing only in framework-internal diagnostics) is what makes it actually usable for this kind of reconstruction in Seq.
7. Metrics and Alerting Basics
| Metric type | Example |
|---|---|
| Rate | Requests per minute, errors per minute |
| Duration | Response time (often as percentiles — p50, p95, p99, not just an average) |
| Saturation | CPU/memory utilization, queue depth |
Why percentiles matter over a plain average: an average response time of 200ms can hide a p99 of 8 seconds affecting one in a hundred users — an average alone can look completely healthy while a meaningful subset of real users have a genuinely broken experience. p95/p99 tracking surfaces exactly the tail-latency problems an average quietly buries.
A basic alerting posture worth having regardless of platform: alert on error rate (a sudden spike, not just "any error exists"), response time degradation (p95/p99 crossing a threshold), and resource saturation (CPU/memory approaching limits) — the three categories that reliably precede a full outage if left unaddressed.
8. Troubleshooting Playbook
| Symptom | Likely cause | What to check |
|---|---|---|
| Browser shows a CORS error | Frequently a masked 500, not an actual CORS misconfiguration | Check App Service Log Stream at the same timestamp before touching CORS config at all |
| Structured logs missing entirely for a service | Seq endpoint not configured, or unreachable from that environment | Confirm the Seq connection URI is actually set and reachable, since logging is configured to fall back silently rather than crash without it |
| Can't tell which downstream call was slow for a specific request | No correlation ID threaded through the logs | Add/confirm TraceIdentifier (or an equivalent) is included explicitly in every structured log statement, not just available in framework internals |
| Average response time looks fine, users still complain of slowness | Averages hiding tail latency | Check p95/p99, not just the average |
| Alert fired, unclear what actually happened | Alert only signals "something crossed a threshold," no correlated log/trace attached | Pair every alert with a direct link or query into the corresponding logs/traces for that time window, not just the raw metric |
| Log volume too high to find anything useful during an incident | No structured fields to filter on, or overly verbose logging at the wrong level | Filter by structured fields (service, correlation ID, level) rather than scrolling; review log level configuration if genuinely too noisy |
9. Quick Reference
| Category | Item | Detail |
|---|---|---|
| Pillar | Logs | What happened, in detail |
| Pillar | Metrics | Aggregate performance over time |
| Pillar | Traces | A single request's path through the system |
| Logging | Structured logging | Queryable fields, not free text |
| Tool | Seq | Structured log platform, conditional on valid URI config |
| Tool | App Service Log Stream | Live server-side stdout/stderr, source of truth for 500s |
| Pattern | CORS masking a 500 | Check server logs before touching CORS config |
| Tracing | Correlation ID | Threads one request's logs together across components |
| Metrics | p95 / p99 | Reveals tail latency an average hides |
Part of the Traxs Engineering Handbook — Volume 5: Platform Engineering. Companion chapters in this volume: Linux Fundamentals, Linux Administration, Kubernetes Fundamentals, CI/CD, Secrets Management.