Xiber NetOS — Admin & Operations Manual

Development setup, services, configuration, migrations, roles, and troubleshooting.

Tech Stack

Prerequisites

• Docker Engine 24+
• Docker Compose v1 (the standalone docker-compose binary, e.g. 1.29.2). The production NetOS server runs Compose v1; the Compose v2 plugin (docker compose, no hyphen) is not installed for the root/sudo context on that host, so all server operations use the hyphenated docker-compose command. For consistency with the server, every command in this guide uses the hyphenated docker-compose (v1) form.
• 4 GB RAM minimum
• Ports 3000, 5432, 6379, 8000 available
• (Optional) Python 3.12+ and Node.js 20+ for local development outside Docker

Repository Layout

circuitos/
├── apps/
│   ├── api/                FastAPI backend
│   │   ├── app/
│   │   │   ├── main.py         Entry point
│   │   │   ├── api/v1/         Route handlers
│   │   │   ├── core/           Auth, config
│   │   │   ├── db/             Session management
│   │   │   ├── models/         SQLAlchemy entities + enums
│   │   │   ├── schemas/        Pydantic request/response models
│   │   │   └── services/       Business logic (import, extraction, scheduler)
│   │   ├── alembic/            Migration versions
│   │   ├── scripts/            Seed data, utilities
│   │   ├── pyproject.toml      Python dependencies
│   │   └── Dockerfile
│   ├── web/                Next.js frontend
│   │   ├── app/                App Router pages
│   │   ├── components/         React components
│   │   ├── lib/                API client, utilities
│   │   ├── package.json
│   │   └── Dockerfile
│   └── mcp/                MCP server skeleton
│       ├── app/server.py
│       └── pyproject.toml
├── infra/
│   └── docker/
│       ├── docker-compose.yml          Local dev stack
│       ├── docker-compose.public.yml   Production profile (SSO + tunnel)
│       ├── nginx/                      Public reverse proxy config
│       └── postgres/                   Custom PostgreSQL Dockerfile
├── docs/                   Documentation (you are here)
├── examples/               Sample CSV data
└── .env.example            Environment variable template

Local Development

Start Everything

cd infra/docker
docker-compose up --build

Run Migrations

docker-compose exec api alembic upgrade head

Seed Sample Data

docker-compose exec -T -e PYTHONPATH=/app api python scripts/seed_sample_data.py

Creates: 4 carriers (Lumen, Zayo, Cogent, Hurricane Electric), 10 endpoints with coordinates, 8 circuits with contracts, and lifecycle events. The script is idempotent — safe to run multiple times.

Restart Services

docker-compose restart api web

Rebuild After Dependency Changes

docker-compose up --build api web

View Logs

docker-compose logs -f api        # API logs
docker-compose logs -f web        # Frontend logs
docker-compose logs -f postgres   # Database logs

Services

Server Deployment

The current demo deployment runs on the Xiber NetOS server.

Server path:

/home/stephen/netos

Server compose file:

infra/docker/docker-compose.server.yml

The server runs Docker Compose v1 (docker-compose 1.29.2). Always use the hyphenated docker-compose binary with the -p netos project name and the server compose file:

docker-compose -p netos -f docker-compose.server.yml ps
docker-compose -p netos -f docker-compose.server.yml logs -f api
docker-compose -p netos -f docker-compose.server.yml logs -f web

Containers on this project are named netos_<service>_1 — e.g. netos_api_1, netos_web_1, netos_postgres_1, netos_redis_1.

Compose v1 recreate bug (KeyError: 'ContainerConfig')

Compose v1 (1.29.x) frequently fails when asked to recreate an already-running service in place, aborting with:

ERROR: for netos_api_1  'ContainerConfig'
KeyError: 'ContainerConfig'

The safe pattern the checked-in deploy scripts use is to stop and remove the old container first, then bring it up fresh:

cd ~/netos/infra/docker
docker-compose -p netos -f docker-compose.server.yml build api web

# Remove the existing containers BEFORE `up` to avoid the ContainerConfig bug
docker stop netos_api_1 netos_web_1 || true
docker rm   netos_api_1 netos_web_1 || true

docker-compose -p netos -f docker-compose.server.yml up -d api web

Do this per service you are recreating. docker-compose ... up -d <svc> on a service whose container was already stopped+removed creates it cleanly; up -d on a live container is what triggers the bug.

The server web service runs next start from a built image. Do not mount apps/web or .next over the production container; only docs/ is mounted read-only at /app/content/docs.

Releases that include migrations

When a release ships new Alembic migrations, apply them with a one-off container BEFORE swapping the running `api` container, so the schema is already at head when the new code starts:

cd ~/netos/infra/docker
docker-compose -p netos -f docker-compose.server.yml build api
docker-compose -p netos -f docker-compose.server.yml run --rm api alembic upgrade head
# then recreate api (and web) using the stop+rm+up sequence above

Rollback

Tag the currently-good images before deploying so you can revert quickly if a new build misbehaves:

# Before deploy: retag the known-good images
docker tag netos_api:latest netos_api:rollback
docker tag netos_web:latest netos_web:rollback

# To roll back: point :latest back at the saved images and recreate
docker tag netos_api:rollback netos_api:latest
docker tag netos_web:rollback netos_web:latest
docker stop netos_api_1 netos_web_1 || true
docker rm   netos_api_1 netos_web_1 || true
docker-compose -p netos -f docker-compose.server.yml up -d api web

Note that a rollback only reverts application code/images, not the database. If the failed release ran migrations, plan the corresponding downgrade separately.

For fast frontend iteration, run a separate local preview service instead of putting the public web container into development mode:

cd ~/netos
scripts/start-web-preview.sh

The preview service runs next dev with the web source bind-mounted. It is available locally at http://127.0.0.1:3011/preview and through the protected public proxy at https://netos.xiberian.net/preview. When a set of changes is ready for the public app, promote it with:

cd ~/netos
scripts/deploy-prod-web.sh

Run migrations on the server:

cd ~/netos/infra/docker
docker-compose -p netos -f docker-compose.server.yml run --rm api alembic upgrade head

Configuration

RF System Standards

RF system standards are maintained from the Admin page. They define the selectable defaults used when creating RF Links.

Use PtMP Base/Sector for equipment installed on an infrastructure asset as a PtMP node, such as Tarana BN, Siklu Terragraph DN, or Cambium ePMP AP. Use PtMP Subscriber for customer-side radios, such as Tarana RN/RNv, Siklu T280/T265, or Cambium Force 4600c. When a user creates a PtMP RF Link from a selected PtMP source node, NetOS shows only subscriber radios whose compatible base list includes that source node's base system.

RF Links no longer require a manually entered link name. NetOS generates the display name from the selected A facility, Z facility or customer endpoint, and RF system/equipment. Links between two infrastructure assets are classified as backhaul. Links from an infrastructure asset to a customer endpoint are classified as customer endlinks.

Environment Variables

The API reads its configuration from environment variables via apps/api/app/core/config.py (Settings). Names below are the environment-variable form (Settings maps them case-insensitively). Defaults are the application defaults from Settings; the server compose file (docker-compose.server.yml) overrides some of them (noted where it matters). Copy .env.example to .env for local dev; the production stack is configured through docker-compose.server.yml and infra/docker/.env.public.

Database and Redis

Authentication and proxy trust (see Authorization & RBAC)

SMTP and notifications (bootstrap defaults; Admin > Notifications overrides these once saved)

AI providers

Geocoding

Scheduling (weekly renewal report and activity digest; times are UTC, Monday = 0)

Frontend (web container) — these are read by Next.js, not by Settings:

Copy .env.example to .env and adjust as needed. For production SSO, see Public Hostname & SSO.

AI Assistant Provider

The NetOS Assistant uses /api/v1/ai/extract. With no provider configured it uses the deterministic parser. Admin users should normally configure the live provider from Admin > AI, where API keys are encrypted server-side and only the last four characters are shown after save.

Supported AI providers:

• OpenAI: default base URL https://api.openai.com/v1, using the Responses API with strict JSON output.
• MiniMax: default base URL https://api.minimax.io/v1, default model MiniMax-M3, using the OpenAI-compatible Chat Completions API for text/image tasks.
• Fallback parser: deterministic extraction with no external model.

Use Test in Admin > AI after entering a key. The test calls the provider's OpenAI-compatible /models endpoint and verifies that the selected model is advertised before users rely on the provider.

The assistant never writes inventory directly. It returns draft payloads that users apply to forms and save through the normal NetOS APIs. Durable assistant memory is stored in ai_memory_entries and is limited to explicit user or organization preferences/rules, such as lines beginning with "remember..." or "always...".

Site photo analysis uses the same configured provider. Uploaded images are stored under /app/storage and the Docker deployment mounts that path from apps/api/storage so documents survive API container rebuilds. If an image analysis fails because the provider returned an empty or invalid response, users can open the site document gallery and choose Re-run AI after settings are corrected.

The Admin > AI page also includes a Customer Research Connector for Sonar/MCP-style enrichment. Configure the MCP URL, tool name, optional authorization token, and enable the connector. NetOS calls this connector only from the backend, then converts the research result into a normal customer import preview row. Users must still review, edit, select, and commit the staged row through the standard import workflow.

The connector supports either an MCP SSE URL such as http://mcp.xiberian.net:8081/sse?agent_id=netos-customer-research or a direct HTTP endpoint that accepts MCP JSON-RPC tools/call requests. For SSE, NetOS opens the stream, reads the server-provided message endpoint, initializes the MCP session, and then calls the configured tool. The configured tool should accept a query argument and return either structured JSON with draft, citations, warnings, and summary, or text containing that JSON. Browser clients never receive the connector credential. If the MCP server expects Authorization: <token> rather than Authorization: Bearer <token>, paste only the raw token in the API key field; NetOS sends the value exactly as entered.

Use the Test button after entering connector values. The test initializes the MCP session and runs tools/list; it does not perform customer research or create import rows. A passing test means NetOS can reach the MCP server and the configured tool name appears in the advertised tool list.

Email Notifications

Admin users with settings_view / settings_edit can manage delivery from Admin > Notifications. This page controls:

• SMTP host, port, username, password, and sender
• public NetOS base URL used in email links
• welcome emails for newly created users
• weekly report enablement, day, UTC hour, and lookahead window
• weekly activity digest enablement, day, UTC hour, and lookback window
• test email delivery before saving or sending reports

Saved Admin notification settings override environment variables. Environment variables remain useful as bootstrap defaults and for deployments that have not yet saved a database-backed notification setting. SMTP passwords are encrypted server-side and are never returned to the browser; the UI only shows whether a password exists and its last four characters. The Xiber gateway at mgw.xiberian.net refuses plaintext authentication; NetOS uses STARTTLS on port 587 and implicit TLS on port 465.

The Timeline page includes a Weekly Renewal Email Report section. It previews and manually sends an HTML report for circuits and infrastructure agreements whose renewal decision deadline is overdue or within the configured lookahead window. Each report includes:

• urgency state (watch, active, critical, overdue)
• agreement/vendor/type/status details
• MRC, term end, renewal notice, and renewal term
• links back to the circuit or infrastructure detail
• links to available agreement documents
• an HTML timeline-style snapshot for the report items

The API process also runs a lightweight scheduler. When SMTP is configured and the renewal report is enabled, it checks every 30 minutes and sends once on the configured UTC weekday/hour to all active registered users. Sends are recorded in user_activity_logs with event_type=report, so a container restart does not send the same report date twice.

The same scheduler sends the weekly activity digest when enabled. The digest summarizes write events, authorization changes, API-token usage, report sends, and renewal notice counts for the configured lookback window. Admins can also send the digest immediately from Admin > Notifications.

Docker Network

• Server-side web calls (Next.js SSR) use API_BASE_URL=http://api:8000 (Docker internal DNS)
• Browser-side calls use the UI host origin.
• Machine clients and MCP connectors should use http://10.10.67.118:8010

with Authorization: Bearer ntos_live_.... The API is also bound to 127.0.0.1:8010 for local VM checks and is not bound to the public NIC.

• The VM has persistent netplan routes for 10.10.75.0/24 and 10.10.43.0/24

via 10.10.67.1 dev ens19. These keep replies to internal API clients on the protected LAN path instead of returning through the public ens18 default route.

Authentication & RBAC

Authentication (who you are) and authorization (what you may do) are separate. Microsoft SSO / Entra proves identity; NetOS decides authorization with its own database-driven roles, permissions, and field-level filtering. The authoritative reference is Authorization & RBAC; this section is the operator summary.

Identity: production vs. dev auth

Two mutually exclusive identity paths, both selected in apps/api/app/core/config.py:

• Production (proxy-trusted headers). With TRUST_PROXY_AUTH_HEADERS=true, the API trusts SSO identity headers (X-Auth-Request-Email, X-Forwarded-Email, CF-Access-Authenticated-User-Email, X-Ms-Client-Principal-Name, etc.) — but only when the request's client IP is inside `TRUSTED_AUTH_PROXY_CIDRS`. This is how browser SSO flows through the reverse proxy. See Public Hostname & SSO.
• Dev auth (header shim). With ALLOW_DEV_AUTH=true, the API accepts x-user-email / x-user-role with no identity verification:

``bash curl -H "x-user-role: exec" http://localhost:8000/api/v1/circuits

When no email header is present it falls back to dev@xiber.com, which auto-bootstraps as Super Admin. For that reason `ALLOW_DEV_AUTH` must be `false` in production. The server compose file already defaults it to false.

If neither path yields an identity, the API returns 401.

Roles

Real authorization uses the DB-seeded RBAC roles — Super Admin, Executive, Finance Admin, Project Manager, Network Engineer, Field Technician, Read-Only User — each composed of granular permissions. See Authorization & RBAC for the full model, field-level financial masking, and API tokens.

The lowercase identity roles below are the legacy `user.role` values and the values accepted by the dev-auth x-user-role header. They gate a few coarse write checks but are not the primary authorization mechanism:

Database

Custom PostgreSQL Image

Built from pgvector/pgvector:pg16 with PostGIS installed.

Dockerfile: infra/docker/postgres/Dockerfile

Core Tables

Uploaded document files are persisted outside the API container at apps/api/storage, mounted into the container as /app/storage. Do not remove this mount in production; otherwise database document records can outlive the uploaded files after a container rebuild.

Materialized View

• circuit_financial_mv — pre-computed financial rollups for dashboard performance

Documentation PDFs

Markdown documentation is the source of truth. Downloadable PDFs are generated from the Markdown files and published from apps/web/public/docs.

Run the exporter after material documentation changes:

python3 scripts/build_doc_pdfs.py

Generated files include:

The exporter embeds public documentation images such as the infrastructure attribution and waterfall diagrams.

Extensions

• PostGIS — geometry columns for endpoint coordinates, GeoJSON queries
• pgvector — vector embeddings (installed, not yet used)

Migrations

Migration Files

Located in apps/api/alembic/versions/, numbered sequentially (0001 … 0054 and growing). Always deploy schema with alembic upgrade head rather than tracking individual files. Notable milestones:

Run docker-compose -p netos -f docker-compose.server.yml run --rm api alembic current to see the applied revision.

Commands

Apply all migrations:

# Via Docker
docker-compose exec api alembic upgrade head

# Or standalone
cd apps/api
alembic upgrade head

Check current version:

docker-compose exec api alembic current

Create a new migration:

docker-compose exec api alembic revision --autogenerate -m "description"

Review auto-generated migrations before applying — they may need manual adjustments for PostGIS columns or custom types.

Seed Data

Script: apps/api/scripts/seed_sample_data.py

What it creates:

The script uses upsert logic on natural keys — safe to run repeatedly without creating duplicates.

Audit Trail

Audit data is available in the Admin view at /admin.

What Is Logged

Timestamps are stored in UTC in PostgreSQL and displayed in the UI as Eastern time with the timezone label.

Soft Deletes

Circuit and infrastructure bulk deletes set deleted_at. They do not physically remove rows. Normal list/map/dashboard views filter out soft-deleted rows.

Bug & Feature Queue

The Admin view includes a feedback form and queue.

Admins can add progress comments to each request. Public comments attempt to email the requester when SMTP is configured. SMTP failures are non-blocking, so feedback updates still save if email is unavailable.

SMTP configuration is normally managed from Admin > Notifications. The variables below are bootstrap fallbacks before Admin settings are saved:

Verification Commands

Check API Health

curl -fsS http://localhost:8000/healthz
# → {"status": "ok"}

Check Map Data

curl -fsS http://localhost:8000/api/v1/circuits/map | python3 -m json.tool | head -20

Check Dashboard

curl -fsS http://localhost:8000/api/v1/dashboard/summary | python3 -m json.tool | head -20

API Tests and Lint (pytest + httpx)

The API has an integration test harness under apps/api/tests/ (pytest, pytest-asyncio, and an in-process httpx client against the ASGI app). Run it from the API venv:

cd apps/api
.venv/bin/pytest tests/ -v
.venv/bin/ruff check app tests

Key facts for operators:

• Tests run against a dedicated `circuitos_test` database, never the live circuitos database. The default TEST_DATABASE_URL is postgresql+asyncpg://circuitos:circuitos@127.0.0.1:55432/circuitos_test (override via the TEST_DATABASE_URL env var).
• conftest.py hard-refuses any target database whose name does not end in _test, and it auto-creates the test database and runs migrations on first use.
• The harness forces ALLOW_DEV_AUTH=true; with no identity headers, requests resolve to dev@xiber.com (auto-bootstrapped Super Admin), so tests exercise authorized paths by default.

Compile Check (Python)

python3 -m compileall apps/api/app apps/api/scripts apps/mcp/app

Web Gates (Next.js)

cd apps/web
npm run typecheck   # tsc --noEmit
npm run lint        # next lint
npm run build       # next build

Troubleshooting

Map Doesn't Load

Frontend 500 on Server-Rendered Pages

• Verify API_BASE_URL=http://api:8000 is set in Docker Compose
• Verify the api container is running: docker-compose logs api
• Check for Python exceptions in API logs

No Sample Circuits Showing

• Run the seed script (see above)
• Verify: curl http://localhost:8000/api/v1/circuits returns data
• Check migrations: docker-compose exec api alembic current

PostGIS Extension Errors

• Rebuild the custom PostgreSQL image: docker-compose build postgres
• Verify infra/docker/postgres/Dockerfile is referenced in docker-compose.yml
• Check: docker-compose exec postgres psql -U circuitos -c "SELECT PostGIS_Version();"

Container Won't Start

docker-compose logs <service>     # Check for errors
docker-compose down -v            # Nuclear option: remove volumes and rebuild
docker-compose up --build

Warning: down -v destroys all data. Re-run migrations and seed script after.