Xiber NetOS — Admin & Operations Manual

Development setup, services, configuration, migrations, roles, and troubleshooting.

Tech Stack

Prerequisites

• Docker Engine 24+
• Docker Compose v1 (the standalone docker-compose binary, e.g. 1.29.2). The production NetOS server runs Compose v1; the Compose v2 plugin (docker compose, no hyphen) is not installed for the root/sudo context on that host, so all server operations use the hyphenated docker-compose command. For consistency with the server, every command in this guide uses the hyphenated docker-compose (v1) form.
• 4 GB RAM minimum
• Ports 3000, 5432, 6379, 8000 available
• (Optional) Python 3.12+ and Node.js 20+ for local development outside Docker

Repository Layout

circuitos/
├── apps/
│   ├── api/                FastAPI backend
│   │   ├── app/
│   │   │   ├── main.py         Entry point
│   │   │   ├── api/v1/         Route handlers
│   │   │   ├── core/           Auth, config
│   │   │   ├── db/             Session management
│   │   │   ├── models/         SQLAlchemy entities + enums
│   │   │   ├── schemas/        Pydantic request/response models
│   │   │   └── services/       Business logic (import, extraction, scheduler)
│   │   ├── alembic/            Migration versions
│   │   ├── scripts/            Seed data, utilities
│   │   ├── pyproject.toml      Python dependencies
│   │   └── Dockerfile
│   ├── web/                Next.js frontend
│   │   ├── app/                App Router pages
│   │   ├── components/         React components
│   │   ├── lib/                API client, utilities
│   │   ├── package.json
│   │   └── Dockerfile
│   └── mcp/                MCP server skeleton
│       ├── app/server.py
│       └── pyproject.toml
├── infra/
│   └── docker/
│       ├── docker-compose.yml          Local dev stack
│       ├── docker-compose.public.yml   Production profile (SSO + tunnel)
│       ├── nginx/                      Public reverse proxy config
│       └── postgres/                   Custom PostgreSQL Dockerfile
├── docs/                   Documentation (you are here)
├── examples/               Sample CSV data
└── .env.example            Environment variable template

Local Development

Start Everything

cd infra/docker
docker-compose up --build

Run Migrations

docker-compose exec api alembic upgrade head

Seed Sample Data

docker-compose exec -T -e PYTHONPATH=/app api python scripts/seed_sample_data.py

Creates: 4 carriers (Lumen, Zayo, Cogent, Hurricane Electric), 10 endpoints with coordinates, 8 circuits with contracts, and lifecycle events. The script is idempotent — safe to run multiple times.

Restart Services

docker-compose restart api web

Rebuild After Dependency Changes

docker-compose up --build api web

View Logs

docker-compose logs -f api        # API logs
docker-compose logs -f web        # Frontend logs
docker-compose logs -f postgres   # Database logs

Services

Server Deployment

The current demo deployment runs on the Xiber NetOS server.

Server path:

/home/stephen/netos

Server compose file:

infra/docker/docker-compose.server.yml

The server runs Docker Compose v1 (docker-compose 1.29.2). Always use the hyphenated docker-compose binary with the -p netos project name and the server compose file:

docker-compose -p netos -f docker-compose.server.yml ps
docker-compose -p netos -f docker-compose.server.yml logs -f api
docker-compose -p netos -f docker-compose.server.yml logs -f web

Containers on this project are named netos_<service>_1 — e.g. netos_api_1, netos_web_1, netos_postgres_1, netos_redis_1.

Compose v1 recreate bug (KeyError: 'ContainerConfig')

Compose v1 (1.29.x) frequently fails when asked to recreate an already-running service in place, aborting with:

ERROR: for netos_api_1  'ContainerConfig'
KeyError: 'ContainerConfig'

The safe pattern the checked-in deploy scripts use is to stop and remove the old container first, then bring it up fresh:

cd ~/netos/infra/docker
docker-compose -p netos -f docker-compose.server.yml build api web

# Remove the existing containers BEFORE `up` to avoid the ContainerConfig bug
docker stop netos_api_1 netos_web_1 || true
docker rm   netos_api_1 netos_web_1 || true

docker-compose -p netos -f docker-compose.server.yml up -d api web

Do this per service you are recreating. docker-compose ... up -d <svc> on a service whose container was already stopped+removed creates it cleanly; up -d on a live container is what triggers the bug.

The server web service runs next start from a built image. Do not mount apps/web or .next over the production container; only docs/ is mounted read-only at /app/content/docs.

Releases that include migrations

When a release ships new Alembic migrations, apply them with a one-off container BEFORE swapping the running `api` container, so the schema is already at head when the new code starts:

cd ~/netos/infra/docker
docker-compose -p netos -f docker-compose.server.yml build api
docker-compose -p netos -f docker-compose.server.yml run --rm api alembic upgrade head
# then recreate api (and web) using the stop+rm+up sequence above

Rollback

Tag the currently-good images before deploying so you can revert quickly if a new build misbehaves:

# Before deploy: retag the known-good images
docker tag netos_api:latest netos_api:rollback
docker tag netos_web:latest netos_web:rollback

# To roll back: point :latest back at the saved images and recreate
docker tag netos_api:rollback netos_api:latest
docker tag netos_web:rollback netos_web:latest
docker stop netos_api_1 netos_web_1 || true
docker rm   netos_api_1 netos_web_1 || true
docker-compose -p netos -f docker-compose.server.yml up -d api web

Note that a rollback only reverts application code/images, not the database. If the failed release ran migrations, plan the corresponding downgrade separately.

For fast frontend iteration, run a separate local preview service instead of putting the public web container into development mode:

cd ~/netos
scripts/start-web-preview.sh

The preview service runs next dev with the web source bind-mounted. It is available locally at http://127.0.0.1:3011/preview and through the protected public proxy at https://netos.xiberian.net/preview. When a set of changes is ready for the public app, promote it with:

cd ~/netos
scripts/deploy-prod-web.sh

Run migrations on the server:

cd ~/netos/infra/docker
docker-compose -p netos -f docker-compose.server.yml run --rm api alembic upgrade head

Configuration

RF System Standards

RF system standards are maintained from the Admin page. They define the selectable defaults used when creating RF Links.

Use PtMP Base/Sector for equipment installed on an infrastructure asset as a PtMP node, such as Tarana BN, Siklu Terragraph DN, or Cambium ePMP AP. Use PtMP Subscriber for customer-side radios, such as Tarana RN/RNv, Siklu T280/T265, or Cambium Force 4600c. When a user creates a PtMP RF Link from a selected PtMP source node, NetOS shows only subscriber radios whose compatible base list includes that source node's base system.

RF Links no longer require a manually entered link name. NetOS generates the display name from the selected A facility, Z facility or customer endpoint, and RF system/equipment. Links between two infrastructure assets are classified as backhaul. Links from an infrastructure asset to a customer endpoint are classified as customer endlinks.

Environment Variables

The API reads its configuration from environment variables via apps/api/app/core/config.py (Settings). Names below are the environment-variable form (Settings maps them case-insensitively). Defaults are the application defaults from Settings; the server compose file (docker-compose.server.yml) overrides some of them (noted where it matters). Copy .env.example to .env for local dev; the production stack is configured through docker-compose.server.yml and infra/docker/.env.public.

Database and Redis

Authentication and proxy trust (see Authorization & RBAC)

SMTP and notifications (bootstrap defaults; Admin > Notifications overrides these once saved)

AI providers

Storage and uploads

Site photos: AI downscaling and HEIC support

Before a photo is sent to the AI vision model, app.services.site_images.prepare_image_for_analysis downscales it (longest edge capped at 1600px, re-encoded as JPEG under a ~2.5MB cap, stepping quality/edge down further if needed) — no photo is skipped for being large anymore, and the original file on disk is never touched. HEIC/HEIF support (iPhone photos) comes from the pillow-heif dependency (apps/api/pyproject.toml), registered at import time behind a try/except in site_images.py (register_heif_opener()) so a missing or broken wheel on an unusual platform never breaks API startup — it just means HEIC uploads keep their metadata but can't be opened for AI analysis/thumbnails on that host. Confirm it installed with docker compose exec api python -c "import pillow_heif" after a deploy that touches this dependency.

Integrations

Geocoding

Photo GPS backfill. Photos uploaded before GPS enrichment shipped have gps_latitude/gps_longitude (and any EXIF heading/altitude) but no reverse-geocoded place or site distance/bearing. POST /api/v1/admin/photos/backfill-geo (requires settings_edit; optional ?limit= query param, default 200) runs app.services.site_images.backfill_photo_geo once across InfrastructureDocument and CustomerDocument rows, rate-limited to ~1 row/second to respect Nominatim's usage policy, and returns {infrastructure_documents, customer_documents, examined, left_without_place} counts. Call it again (it skips rows that already have a metadata_json.gps.place) to page through more than one call's limit. Run it once after this feature ships to backfill existing photos.

Scheduling (weekly renewal report and activity digest; times are UTC, Monday = 0)

Frontend (web container) — these are read by Next.js, not by Settings:

Copy .env.example to .env and adjust as needed. For production SSO, see Public Hostname & SSO.

AI Assistant Providers

The NetOS Assistant uses /api/v1/ai/extract. With no provider configured it uses the deterministic parser. Admin users configure one or more provider keys from Admin > AI, where API keys are encrypted server-side and only the last four characters are shown after save.

Admin > AI manages a table of provider keys (ai_provider_settings), not a single row. Each provider has:

• Label — a human-readable name for the key (e.g. "Primary OpenAI", "Backup DeepSeek").
• Provider — a free-form vendor slug (e.g. openai, deepseek, groq, ollama, custom). This is informational only (a label/preset key) and never affects request routing.
• API style — the field that actually controls how requests are shaped: openai_responses (OpenAI's own /responses API, input_image parts, strict json_schema output — OpenAI only) or openai_chat (the OpenAI-compatible /chat/completions API with image_url parts — DeepSeek, MiniMax, Groq, OpenRouter, Together, Mistral, xAI, Ollama, LM Studio, vLLM, and effectively any other OpenAI-compatible endpoint). Any OpenAI-compatible endpoint works as an openai_chat provider, not just the presets listed below.
• JSON mode — how each openai_chat call asks for structured JSON output:
• auto (default, recommended): try response_format: json_schema first; on an HTTP 400/422 (or any 4xx whose body mentions response_format/json_schema/unsupported), retry once with json_object.
• json_schema: always send strict response_format: json_schema. Requires provider support for structured outputs.
• json_object: always embed the JSON schema as instructions in the system prompt and send response_format: json_object. Needed by DeepSeek and many OpenAI-compatible servers that don't support json_schema.
• openai_responses providers always use strict json_schema output and ignore this setting.
• No API key required — check this for a local/self-hosted server (e.g. Ollama) that doesn't check an API key at all; the row is then usable with no stored key and NetOS sends the request with no Authorization header.
• Priority — an integer, lower wins. Used to order candidates within the same group (see resolution below).
• Uses — an optional set of feature areas this key should serve: site_photos (image analysis), documents (uploaded document analysis), extraction (/ai/extract, /ai/extract-pdf), import_preview (bulk import transform/extraction), quotes (vendor quote extraction), findings (site finding narrative drafting from photo/document AI analyses plus technician notes — text-only, never re-sends image bytes, so a non-vision model is fine here). Leaving Uses empty makes the key General — eligible for any use. Site photo analysis needs a vision-capable model — text-only models (e.g. deepseek-flash) will fail or return nonsense if assigned to site_photos.

Resolution: for a given use, NetOS picks the best enabled candidate: a row that lists the use explicitly is preferred over a General row; within the same group, lower priority wins, then most recently updated. Only enabled rows with a known api_style and either a decryptable API key or "No API key required" set are candidates. If the table has no rows at all, NetOS falls back to the OPENAI_API_KEY/AI_PROVIDER environment variables (legacy single-key behavior); if the table has rows but none match, the deterministic fallback parser is used for that call. There is no automatic failover between multiple candidates on a request error.

Presets (prefill only — every field stays editable, and NetOS never verifies these URLs against the network before saving):

The web admin fetches these from GET /api/v1/ai/providers/options, which also returns the API style/JSON mode descriptions and the Uses list — so this table is generated from a single Python source (app/services/ai_providers.py), never hardcoded in the frontend.

Fallback parser: deterministic extraction with no external model (used automatically whenever no enabled provider key matches a given use).

Anthropic's Messages API is not yet a supported api_style. Only openai_responses and openai_chat exist today; a Claude/Anthropic provider would need a new api_style implementation, not just a new preset.

Use Test on a provider row in Admin > AI after entering a key (or against the already-saved key). The test calls the provider's OpenAI-compatible /models endpoint and verifies that the selected model is advertised before users rely on the provider. For a row with "No API key required" set, the probe is sent with no Authorization header. If the server is reachable but rejects the request as unauthorized (HTTP 401/403), the test reports ok: false with a message that distinguishes "reached the server, key was rejected" from a network/DNS failure.

The Model field has a Load models button that pulls the provider's live /models list (once a key is stored or typed, or "No API key required" is checked) so the model can be picked from what the provider actually offers instead of typed blind. Free text is still allowed for a model the provider doesn't list — some providers return very different list sizes: OpenRouter returns hundreds of models, DeepSeek currently returns two (deepseek-flash, deepseek-v4-pro), and Ollama returns whatever models have been locally pulled on that host.

The assistant never writes inventory directly. It returns draft payloads that users apply to forms and save through the normal NetOS APIs. Durable assistant memory is stored in ai_memory_entries and is limited to explicit user or organization preferences/rules, such as lines beginning with "remember..." or "always...".

Site photo analysis (site_photos) and document analysis (documents) resolve independently, so an admin can route image analysis and document analysis to different provider keys. Uploaded images are stored under /app/storage and the Docker deployment mounts that path from apps/api/storage so documents survive API container rebuilds. If an image analysis fails because the provider returned an empty or invalid response, users can open the site document gallery and choose Re-run AI after settings are corrected.

The Admin > AI page also includes a Customer Research Connector for Sonar/MCP-style enrichment. Configure the MCP URL, tool name, optional authorization token, and enable the connector. NetOS calls this connector only from the backend, then converts the research result into a normal customer import preview row. Users must still review, edit, select, and commit the staged row through the standard import workflow.

The connector supports either an MCP SSE URL such as http://mcp.xiberian.net:8081/sse?agent_id=netos-customer-research or a direct HTTP endpoint that accepts MCP JSON-RPC tools/call requests. For SSE, NetOS opens the stream, reads the server-provided message endpoint, initializes the MCP session, and then calls the configured tool. The configured tool should accept a query argument and return either structured JSON with draft, citations, warnings, and summary, or text containing that JSON. Browser clients never receive the connector credential. If the MCP server expects Authorization: <token> rather than Authorization: Bearer <token>, paste only the raw token in the API key field; NetOS sends the value exactly as entered.

Use the Test button after entering connector values. The test initializes the MCP session and runs tools/list; it does not perform customer research or create import rows. A passing test means NetOS can reach the MCP server and the configured tool name appears in the advertised tool list.

Email Notifications

Admin users with settings_view / settings_edit can manage delivery from Admin > Notifications. This page controls:

• SMTP host, port, username, password, and sender
• public NetOS base URL used in email links
• welcome emails for newly created users
• weekly report enablement, day, UTC hour, and lookahead window
• weekly activity digest enablement, day, UTC hour, and lookback window
• test email delivery before saving or sending reports

Saved Admin notification settings override environment variables. Environment variables remain useful as bootstrap defaults and for deployments that have not yet saved a database-backed notification setting. SMTP passwords are encrypted server-side and are never returned to the browser; the UI only shows whether a password exists and its last four characters. The Xiber gateway at mgw.xiberian.net refuses plaintext authentication; NetOS uses STARTTLS on port 587 and implicit TLS on port 465.

The Timeline page includes a Weekly Renewal Email Report section. It previews and manually sends an HTML report for circuits and infrastructure agreements whose renewal decision deadline is overdue or within the configured lookahead window. Each report includes:

• urgency state (watch, active, critical, overdue)
• agreement/vendor/type/status details
• MRC, term end, renewal notice, and renewal term
• links back to the circuit or infrastructure detail
• links to available agreement documents
• an HTML timeline-style snapshot for the report items

The API process also runs a lightweight scheduler. When SMTP is configured and the renewal report is enabled, it checks every 30 minutes and sends once on the configured UTC weekday/hour to all active registered users. Sends are recorded in user_activity_logs with event_type=report, so a container restart does not send the same report date twice.

The same scheduler sends the weekly activity digest when enabled. The digest summarizes write events, authorization changes, API-token usage, report sends, and renewal notice counts for the configured lookback window. Admins can also send the digest immediately from Admin > Notifications.

Recognition Email, Missions & Department Sync

Admin > Notifications also has a Monthly Recognition Email section: enabled toggle, day of month (1–28), and UTC hour, plus a Send Now (previous month) button that always sends (bypassing the once-per-month guard) for the previous calendar month, or for ?month=YYYY-MM if called directly. The email lists top contributors overall and per category, verifier of the month, most improved, badges earned, missions completed, and team totals for that month — names, categories, and counts only, same privacy rule as the leaderboard. Disabled by default; the same scheduler loop that handles renewal reports and the activity digest checks it every 30 minutes and sends once per month, recorded in user_activity_logs (action=monthly_recognition_sent) so a restart never double-sends.

Admin > Recognition > Missions (recognition_admin, Super Admin by default) lists active/completed missions with overall and per-mission progress, and offers:

• Generate now — creates a mission for any of the 8 built-in gap types (infra_missing_coordinates, customer_missing_coordinates, customer_missing_contact, provider_missing_noc_phone, provider_missing_account_manager, infra_no_photo_12mo, infra_stale_180d, records_unverified_top) that has at least 5 candidate records and no existing active mission of that type. The same daily scheduler tick calls this automatically, recorded in user_activity_logs (action=missions_generated) so it only runs once per day.
• Add custom mission — pick a gap type and, for infrastructure/customer gap types, optionally narrow the candidates to one state.
• Archive — retires a mission early (its multiplier stops applying to new events; already-recorded progress and points are untouched).
• Sync departments — pulls department from Microsoft 365 via the Xiber Gateway connector's msgraph.list_users tool and writes it onto matching users rows by email (case-insensitive), so the leaderboard's department filter and Teams view have real data. Requires the xiber_gateway connector (see Integrations below) to be configured and enabled — otherwise the button returns a 422 explaining what's missing. The same weekly scheduler tick runs this automatically, recorded in user_activity_logs (action=departments_synced).

Reward Settings & Bonusly Connector (Rewards, Phase 3)

Admin > Recognition > Rewards (recognition_admin) is the reward program's control panel. A Settings form covers every assumption in Contributions & Rewards §7: milestone size (points), reward units per milestone, per-user monthly cap (milestones), company monthly budget (units), fulfillment mode (auto via Bonusly or batch via CSV export), the giver account email, the hashtag appended to each bonus reason, and the Payouts enabled toggle. Saving with payouts_enabled=true is rejected (422) unless fulfillment mode is batch, or the bonusly connector (below) is enabled *and* has passed a Test within the last 30 days — this stops an admin from flipping payouts on against an unconfigured or long-stale connector.

A budget meter shows the current calendar month's committed/delivered units against the configured cap. A redemptions table (filterable by status: eligible, processing, delivered, held, failed) offers Hold (with a reason), Release (back to eligible), Retry (failed → eligible, clears the attempt counter and backoff), and Mark delivered (records an external Bonusly bonus id and an optional note — the manual completion step for batch mode) on each row. Export CSV downloads the current status filter (or eligible by default) as idempotency_key, user_email, reward_units, milestone_points, month_key, reason for pasting into Bonusly's own bulk-bonus UI or a script. Sweep now runs the nightly job immediately: converts eligible reward balances into milestone redemptions (respecting each user's monthly cap and any open flags, which hold instead of pay), then — if payouts are enabled and mode is auto — attempts to post the eligible queue to Bonusly, respecting the company budget.

Setting up Bonusly for real payouts:

1. In Bonusly, create (or designate) a dedicated giver account — its monthly giving allowance *is* the reward budget, since Bonusly has no separate "company points" concept; every bonus draws from the giver's own balance.
2. As a Global or Tech admin, go to Company → Integrations → API & Tokens and create a Personal Access Token scoped at minimum recognition:write (add recognition:administer if this token should also set giver_email on each bonus rather than defaulting to the token owner). PATs expire after at most a year — calendar a renewal.
3. In Admin > Integrations, open the Bonusly card, paste the giver account's email and the token, enable the connector, and click Test — it calls GET /users/me and, since a giver email is set, also looks up that account's giving_balance and reports it in the Test result. A successful Test stamps last_test_ok_at on the connector, which is what the Rewards settings validation above checks.
4. Back in Admin > Recognition > Rewards, set the budget/cap/milestone numbers to match the giver account's real monthly allowance, set fulfillment mode to auto, enable payouts, and save.
5. The nightly scheduler (or Sweep now) takes it from there: eligible balances become redemptions, and eligible redemptions get posted to Bonusly with a reason like "NetOS data steward reward: 500 contribution points #dataquality", idempotent per milestone (reward:<user_id>:<milestone_index>) so a retry or a scheduler restart never double-pays.

If a Bonusly plan/token without programmatic bonus creation is all that's available, leave fulfillment mode on batch (or leave payouts off) — the CSV export + Mark delivered flow covers the same states without ever needing the connector configured.

Historical Contribution Replay

The Contributions engine only scores writes made *after* it went live — every user_activity_logs audit row from before that (record creates/updates, photo/document uploads, findings, links, comments) was never scored. POST /api/v1/contributions/admin/replay-history (recognition_admin) backfills points for a window of that history, with each event's timestamp set to the *original* activity's time rather than "now," so reputation, levels, and the leaderboard reflect the real history instead of a single spike on the day the backfill ran. See Contributions & Rewards §14 for the full implementation details.

Body: {"since": "2026-01-01T00:00:00Z", "until": "2026-09-01T00:00:00Z", "reward_eligible": false, "dry_run": true, "limit": 5000} — until defaults to now, reward_eligible and dry_run default to false/true.

• Always dry-run first. The response's per_user list (sorted by points, highest first) previews exactly what a real run would award/withhold per person, with a by_rule breakdown — nothing is written in dry-run mode.
• `reward_eligible` stays `false` unless you mean it. By default, replayed points count toward reputation, levels, and the leaderboard immediately, but are excluded from the reward ledger (reward_accounts) entirely — nobody gets a surprise Bonusly payout for a photo they uploaded eight months before the reward program existed. Only set reward_eligible: true for a window where back-paying rewards is the explicit intent.
• Re-running is safe. Rows already scored (their activity_id shows up in contribution_events.source_activity_id) are automatically excluded from consideration on every subsequent call, so replaying an overlapping or repeated window never double-scores anything.
• `limit` bounds how many matching rows (oldest first) a single call considers — useful for a smaller preview or to replay a huge window in batches.
• A real (non-dry-run) call recomputes reputation and the reward ledger for every affected user at the end, and leaves its own summary row in user_activity_logs (action=replay_contribution_history) — which is never itself a replay candidate.
• Not covered: confirm_record/"Confirm this record" history (its reciprocal-guard and corroboration bookkeeping depends on *live* state at verification time, so it can't be safely recomputed after the fact) and import.commit scoring (imports don't log a user_activity_logs row in the shape this replay reads).

Integrations

Admin users with settings_view / settings_edit can manage outbound integration connectors from Admin > Integrations (GET/PUT/POST .../test under /api/v1/integrations/connectors/{connector_type}, backed by the integration_connector_settings table — one row per connector type, upserted by type). Three connectors exist today:

Microsoft Graph — the SharePoint browse/ingest connector for the "site media" feature. Since the delegated-user-token change, Graph calls act as the signed-in user by default — see Public Hostname & SSO → Delegated Microsoft Graph token for the full flow. This card's own tenant/client/secret config is only needed for two things: (a) the tenant-wide application-token fallback, and (b) the "Test" button's client-credentials check.

Access modes:

Neither is available: the API returns 428 Precondition Required with detail "Sign in through the NetOS SSO to access SharePoint as yourself" (see API & Integrations).

`allow_app_fallback` (boolean, default false, part of the connector's config JSON): explicitly opts a deployment into the old tenant-wide behavior for callers with no delegated token (e.g. dev/test environments without the SSO proxy in front, or a temporary gap while rolling out delegated auth). Leave it off in production once delegated auth is verified working, so a misconfigured proxy fails closed (428) rather than silently widening every user's access to the whole tenant.

To configure:

1. In Entra ID, register an application (or reuse the existing SSO app registration).
2. For the delegated (user) path: add Delegated permissions Sites.Read.All and Files.Read.All (plus offline_access) on the *same app registration oauth2-proxy uses for SSO login*, grant admin consent, and apply the public-stack sso-proxy change (see the SSO doc linked above) — this is a one-time infra change, not something done from this card.
3. For the optional application (fallback) path: add Application permissions (not delegated) Sites.Read.All (or Sites.Selected plus a per-site grant for a narrower blast radius) and Files.Read.All, grant admin consent, then paste the Tenant ID, Client ID, and Client Secret here, enable the connector, and check Allow application-token fallback.
4. Click Test to check the application path (acquires a token via client-credentials, calls GET /sites?search=*&$top=1, and decodes the token's roles claim) or Test as me to check your own delegated path (calls GET /me using your forwarded token and decodes its scp claim) — either flags a missing permission/scope even when the underlying call succeeds (a common gap: admin consent was skipped, or Sites.Selected was granted without a site-level consent).

The connector list (GET /api/v1/integrations/connectors) also reports user_token_present for the current request, so the card can show a "Proxy is passing your Microsoft token: yes/no" status line — useful for confirming the sso-proxy change actually took effect after a deploy.

The client secret is Fernet-encrypted the same way AI provider keys are (see AI_SECRET_KEY above) and is never returned to the browser — only whether a secret is saved and its last four characters.

Xiber Gateway — the shared MCP tool gateway (~150 tools across xiber-* servers: BookStack, Sonar, Slack, Monday, Teams, ...) used by later phases (Sonar research, Teams/Slack notifications, and the site-findings sharing/Sonar-ticket actions below). Configure the base URL, agent ID, and token, then Test, which calls GET {base_url}/tools with X-Agent-ID/X-Agent-Token headers and reports the tool count. app/services/xiber_gateway.py exposes a generic call_tool(tool_name, arguments) helper (POST {base_url}/call) that later phases call directly — this phase only wires up settings and the connectivity test.

Bonusly (Contributions & Rewards, Phase 3) — fulfills reward-point milestones as Bonusly bonuses, drawn from a dedicated giver account's own monthly allowance (Bonusly has no separate company-points concept). Configure the giver account's email and a personal access token, enable the connector, and Test, which calls GET /users/me and, since a giver email is set, GET /users?email= to report that account's giving_balance in the result message. See "Reward Settings & Bonusly Connector" above for the full setup flow and how a successful Test gates turning payouts on.

Site findings actions (Phase 3) — sharing a finding by email/Teams/Slack or creating a Sonar ticket (app/api/v1/findings_actions.py, docs/api-integrations.md's "Site Findings — Sharing & Sonar Tickets" section) reuses the two connectors above rather than adding a new one:

GET /api/v1/findings/directory/capabilities (used by the web share dialog to grey out unavailable options, e.g. the Outlook-draft button when the gateway isn't configured) reports {gateway, graph, smtp} booleans computed from the same connector/SMTP configuration described above.

Docker Network

• Server-side web calls (Next.js SSR) use API_BASE_URL=http://api:8000 (Docker internal DNS)
• Browser-side calls use the UI host origin.
• Machine clients and MCP connectors should use http://10.10.67.118:8010

with Authorization: Bearer ntos_live_.... The API is also bound to 127.0.0.1:8010 for local VM checks and is not bound to the public NIC.

• The VM has persistent netplan routes for 10.10.75.0/24 and 10.10.43.0/24

via 10.10.67.1 dev ens19. These keep replies to internal API clients on the protected LAN path instead of returning through the public ens18 default route.

Authentication & RBAC

Authentication (who you are) and authorization (what you may do) are separate. Microsoft SSO / Entra proves identity; NetOS decides authorization with its own database-driven roles, permissions, and field-level filtering. The authoritative reference is Authorization & RBAC; this section is the operator summary.

Identity: production vs. dev auth

Two mutually exclusive identity paths, both selected in apps/api/app/core/config.py:

• Production (proxy-trusted headers). With TRUST_PROXY_AUTH_HEADERS=true, the API trusts SSO identity headers (X-Auth-Request-Email, X-Forwarded-Email, CF-Access-Authenticated-User-Email, X-Ms-Client-Principal-Name, etc.) — but only when the request's client IP is inside `TRUSTED_AUTH_PROXY_CIDRS`. This is how browser SSO flows through the reverse proxy. See Public Hostname & SSO.
• Dev auth (header shim). With ALLOW_DEV_AUTH=true, the API accepts x-user-email / x-user-role with no identity verification:

``bash curl -H "x-user-role: exec" http://localhost:8000/api/v1/circuits

When no email header is present it falls back to dev@xiber.com, which auto-bootstraps as Super Admin. For that reason `ALLOW_DEV_AUTH` must be `false` in production. The server compose file already defaults it to false.

If neither path yields an identity, the API returns 401.

Roles

Real authorization uses the DB-seeded RBAC roles — Super Admin, Executive, Finance Admin, Project Manager, Network Engineer, Field Technician, Read-Only User — each composed of granular permissions. See Authorization & RBAC for the full model, field-level financial masking, and API tokens.

The lowercase identity roles below are the legacy `user.role` values and the values accepted by the dev-auth x-user-role header. They gate a few coarse write checks but are not the primary authorization mechanism:

Database

Custom PostgreSQL Image

Built from pgvector/pgvector:pg16 with PostGIS installed.

Dockerfile: infra/docker/postgres/Dockerfile

Core Tables

Uploaded document files are persisted outside the API container at apps/api/storage, mounted into the container as /app/storage. Do not remove this mount in production; otherwise database document records can outlive the uploaded files after a container rebuild.

Materialized View

• circuit_financial_mv — pre-computed financial rollups for dashboard performance

Documentation PDFs

Markdown documentation is the source of truth. Downloadable PDFs are generated from the Markdown files and published from apps/web/public/docs.

Run the exporter after material documentation changes:

python3 scripts/build_doc_pdfs.py

Generated files include:

The exporter embeds public documentation images such as the infrastructure attribution and waterfall diagrams.

Extensions

• PostGIS — geometry columns for endpoint coordinates, GeoJSON queries
• pgvector — vector embeddings (installed, not yet used)

Migrations

Migration Files

Located in apps/api/alembic/versions/, numbered sequentially (0001 … 0054 and growing). Always deploy schema with alembic upgrade head rather than tracking individual files. Notable milestones:

Run docker-compose -p netos -f docker-compose.server.yml run --rm api alembic current to see the applied revision.

Commands

Apply all migrations:

# Via Docker
docker-compose exec api alembic upgrade head

# Or standalone
cd apps/api
alembic upgrade head

Check current version:

docker-compose exec api alembic current

Create a new migration:

docker-compose exec api alembic revision --autogenerate -m "description"

Review auto-generated migrations before applying — they may need manual adjustments for PostGIS columns or custom types.

Seed Data

Script: apps/api/scripts/seed_sample_data.py

What it creates:

The script uses upsert logic on natural keys — safe to run repeatedly without creating duplicates.

Audit Trail

Audit data is available in the Admin view at /admin.

What Is Logged

Timestamps are stored in UTC in PostgreSQL and displayed in the UI as Eastern time with the timezone label.

Soft Deletes

Circuit and infrastructure bulk deletes set deleted_at. They do not physically remove rows. Normal list/map/dashboard views filter out soft-deleted rows.

Property grouping scrub (address review)

URL: /properties/review. This is the reviewed address scrub described in the Properties section of the user manual: it scans existing commercial customer addresses, proposes building-level groupings, and lets an admin confirm each one by hand before any customer is linked. Nothing links automatically — only Apply (or Apply all confirmed) writes to customers.property_id.

Who can run it. Every route under /api/v1/property-grouping requires properties_edit (Super Admin, Project Manager, Network Engineer by default) except GET /property-grouping/suggest, which only requires customers_edit — it backs the "other customers share this address" hint on the customer add/edit form, not a grouping-review action.

How candidates are built

Candidates are commercial customers only, not soft-deleted; MFC customers are never scanned or grouped. By default only customers with no property_id are considered — pass include_linked=true on a run to also re-scan already-linked customers (useful for spotting a linked customer that drifted from its group, but it never re-links anyone by itself).

1. Canonical address key. Each candidate's address is reduced to a key like 4201 millersville rd|in by app.services.address_normalization.canonical_address_key: collapse whitespace, split off an embedded city/state/ZIP if address_line_1 contains one (e.g. "123 Main St, Indianapolis, IN 46220"), lowercase and strip punctuation, strip unit/suite/floor tokens (ste, suite, unit, apt, #, bldg, fl, rm, …), require a leading house number (rows with none — blank, a PO Box, a bare street name — are reported back as unaddressable and never clustered), then map directionals (north→n, …) and roughly three dozen USPS street suffixes (street/str→st, avenue/av→ave, road→rd, boulevard/boul→blvd, drive→dr, lane→ln, court→ct, place→pl, parkway/pky→pkwy, circle→cir, highway→hwy, terrace→ter, trail→trl, square→sq, plaza→plz, pike, alley→aly, expressway→expy, freeway→fwy, turnpike→tpke, crossing→xing, center/centre→ctr, commons→cmns, cove→cv, point→pt, ridge→rdg, and several already-canonical forms (loop, run, bend→bnd, path, pass, row, walk, way)). Two addresses that differ only in suffix spelling, directional wording, unit, or an embedded city/ZIP land on the same key.
2. Exact-key clusters. Candidates sharing a key are bucketed together first.
3. Guarded proximity merge. Two different-key buckets are merged into one cluster only when their centroids are within the run's radius (default 60 m) and they share the same house number and state — this is deliberately conservative so 4210 Foo St never merges into 4201 Foo St. Two nearby buckets that fail the house-number/state check are left separate and each gets a nearby_cluster warning pointing at the other, for a human to merge manually if appropriate.
4. Existing-property attach. A cluster within the radius of an already-created Property with the same house number (exact key match, or a proximity match under the same guard) is proposed as an addition to that property instead of a new one (existing_property_nearby).
5. Singletons. A cluster below min_members (default 2) is dropped and counted in the run's singletons_dropped total — *unless* it matched an existing property, in which case even a lone new candidate is kept as a proposal to add it to that property.

No geocoding provider is ever called during a run — clustering uses each customer's already-stored latitude/longitude (ST_Y/ST_X on customers.geom) and plain Python haversine distance, not PostGIS ST_DWithin (degrees are anisotropic at this latitude and the candidate set is small). Nominatim is only called on demand, once, when a reviewer clicks "Use Nominatim suggestion" on a single proposal (POST /property-grouping/proposals/{id}/suggest-address) — that call never persists anything by itself.

Warning codes

Proposal status machine and re-run rules

Statuses: proposed → confirmed → applied (terminal), or proposed/confirmed → skipped, or proposed → stale and back. A proposal can also become merged (terminal, folded into a survivor) via Merge into…. Proposals are matched across runs by canonical key: first the proposal's own group_key, then its absorbed_group_keys (accumulated by proximity merges and manual merges over time), following a merged_into_proposal_id chain if the resolved proposal has since been merged into another.

Re-running a scan does this to each existing proposal, depending on its status when the run starts:

A customer a reviewer explicitly removed from a proposal ("Remove selected" → soft-removed member row, removed_at set) stays removed across every subsequent re-run; a run never re-adds a row that already exists for that customer+proposal pair, removed or not. Two clusters resolving to the same already-processed proposal in one run (e.g. after a manual merge) only refresh once, from whichever cluster is encountered first.

What Apply writes

Applying a confirmed proposal (or a proposed one, passing confirm: true in the same call):

1. Creates a new Property via services.properties.create_property (using the proposal's name/nickname/address/tenant_count, status active, geocoded from the cluster centroid) — or, if the proposal targets an existing property (existing_property_id, set by the existing_property_nearby match or a manual override), reuses that property and creates nothing.
2. Links every active (non-removed) member via services.properties.link_customers_to_property(..., source="property_grouping", score=False) — this writes one link_customer_property audit row per customer (with extra.skip_scoring: true) plus a single unscored link_property_customers summary row on the property. This is the same writer every other property-linking path in NetOS uses, so the audit shape is identical to a manual "Add tenants" link.
3. Writes two more audit rows: apply_property_group_proposal on the proposal (before/after proposal payload) and apply_property_group_proposal on the property ({"proposal_id", "linked_customer_ids"}).
4. Marks the proposal applied (applied_by, applied_at, applied_property_id).
5. Pays one property.group_apply contribution award for the whole proposal (12 base points + 1 per member beyond 2, capped at +8 — Milkyway's 19 members pays the full +8, i.e. 20 points) — the per-customer link rows from step 2 are explicitly unscored, so a 19-tenant apply never pays 19 × 3 points. This award is not in the engine's replayable-action allowlist, so it can never be paid twice for the same proposal, including by a future historical replay.

Nominatim is never called as part of generating or applying proposals — only the reviewer-triggered, per-proposal suggest-address endpoint calls it, and that result is never persisted until the reviewer accepts it into the proposal's address fields and saves.

Safety

• Generation is non-destructive. A run (POST /property-grouping/runs) only ever writes property_group_runs / property_group_proposals / property_group_proposal_members rows, plus one system-authored sentinel audit row (event_type=report, action=property_grouping_run, entity_type=property_group_run) recording the run's counts. It never touches customers or creates a Property.
• Apply is the only path that writes `customers.property_id` for this workflow (via link_customers_to_property, the same function every other property-linking UI uses).
• Stale members block the whole apply. apply_proposal re-validates every active member under a row lock at apply time (still exists, not deleted, still commercial, and either unlinked or already linked to this same property). If any member fails, the call raises before writing anything and the API returns 409 with:

``json {"detail": {"detail": "2 members are no longer eligible", "conflicts": [ {"customer_id": "<uuid>", "name": "Acme Co", "reason": "already_linked"} ]}}

Reasons seen here: not_found, deleted, not_commercial, already_linked. Remove the flagged members (or fix the underlying record) and retry.

• Apply-all-confirmed is transactionally isolated per proposal. POST /property-grouping/proposals/apply-all-confirmed wraps each confirmed proposal's apply in its own SAVEPOINT (session.begin_nested()); one proposal's conflict or error rolls back only that proposal (its result lands in failed, others still land in applied), and the whole batch commits once at the end.

Undo

There is no single "undo" button — reverse the writes in order:

1. Unlink each affected customer: DELETE /api/v1/properties/{property_id}/customers/{customer_id} (writes an unlink_customer_property audit row per customer; the customer record and its history are otherwise untouched).
2. If the property itself should go away, DELETE /api/v1/properties/{property_id}?force=true once it has no remaining linked customers or infrastructure (soft-delete).
3. POST /api/v1/property-grouping/proposals/{proposal_id}/reopen puts the proposal back to proposed so it can be re-reviewed or re-applied. This only succeeds once the applied property is fully unlinked or soft-deleted first — reopening while the property still has live linked customers returns 409 ("Cannot reopen: the applied property still has linked customers"), by design, so this can't silently orphan a property with real tenants still pointing at it.

Production runbook

This is the scrub Stephen runs (assisted) against the live circuitos database, per the Properties plan's Verification §3. Follow it in order; nothing links until the explicit Apply / Apply all confirmed steps.

1. Preflight. GET /api/v1/property-grouping/runs/latest should return null (no prior run). As of 2026-09-20 there are 319 commercial customers and 75 MFC customers in production — MFC is never touched by any step below.
2. Run with defaults. POST /api/v1/property-grouping/runs with {"min_members": 2, "radius_m": 60, "include_linked": false} (also the UI's default params). Expect proposals including: 4201 Millersville Rd (19 members, "Milkyway"), 3500 Depauw Blvd (6), 280 E 96th St (7, split city_mismatch — 5 Carmel / 2 Indianapolis), 5858 N College Ave (4), 733 W Henry St (3), 1030 E Washington St (3), several 2-member pairs, and a small unaddressable list of customers whose address needs fixing directly in the customer editor before they can ever be scanned.
3. Review Milkyway first (the largest, highest-value group): open its proposal card, rename it "Milkyway", set a suite count, remove any member flagged far_from_centroid that isn't actually in the building, optionally click "Use Nominatim suggestion" to fill in a postal code, Save, then Confirm, then Apply. Verify: the new property's rollups show 19 active tenants and the expected penetration/MRR sum, and one of its linked customers' own audit history shows a link_customer_property row.
4. Review the rest of the proposals the same way — confirm groups that look right, Skip (with a reason) any that shouldn't become properties, use Merge into… for any pair flagged nearby_cluster that Stephen confirms is really one site.
5. Apply all confirmed (POST /property-grouping/proposals/apply-all-confirmed). Inspect the failed list in the response — expect only stale-member conflicts (a customer edited or reassigned between confirm and apply); fix and re-apply those proposals individually.
6. Verify the results:

• GET /api/v1/property-grouping/proposals?status=applied — the count should equal the number confirmed.
• GET /api/v1/audit/activity?entity_type=customer&limit=500 — filter the response for action=link_customer_property rows and confirm the count equals the total number of customers linked across every applied proposal. (This endpoint filters by entity_type/entity_id/user_email, not by action directly, so filter the returned rows client-side.)
• GET /api/v1/contributions/events?rule_key=property.group_apply (recognition_admin) — one event per applied proposal, no duplicates.

1. Re-run once more with the same defaults and confirm the run's counts.proposals_created is 0 — every real group is already represented by an existing proposal.
2. No contributions replay is needed. property.group_apply and the per-customer link_customer_property rows (unscored) are written going forward only; there's no historical backlog to backfill for this feature the way Historical Contribution Replay exists for other write types.

Bugs & Requests

Bugs & Requests (/requests, sidebar) replaced Admin → Feedback on 2026-09-30 (migration 0105). Admin → Bugs & Requests now just links to it.

• Reporting. Anyone reports from the top bar's Report a problem, which takes the current page along, plus the browser, screen size and page title. The report can include screenshots (paste, drop or pick), a category (guessed from the page), an urgency, and who should look at it.
• Triage. People with feedback_manage (Executives, Project Managers, Super Admins) set status, urgency and category, assign someone, write internal notes, and remove tests or duplicates.
• Tasks. An assigned request is also a task in that person's queue. Finishing the task closes the request and emails the reporter. Handing the task over, or putting it back, moves the request too.
• Visibility. The reporter and the assignee see their request in full; internal notes are only for triagers and the assignee. Everyone else sees a one-line summary, because screenshots can show anything that was on screen.
• Dashboard. Triagers get Bugs & requests to triage under Needs review.

Files are stored under <STORAGE_ROOT>/feedback-attachments/<request id>/ and are served only to people who may see the request in full.

Email goes out on these events:

• the admin address, for each new report;
• the assignee, through the task;
• the reporter, on status changes and public replies;
• the assignee, on replies and notes.

SMTP failures don't block saving.

SMTP configuration is normally managed from Admin > Notifications. The variables below are bootstrap fallbacks before Admin settings are saved:

Verification Commands

Check API Health

curl -fsS http://localhost:8000/healthz
# → {"status": "ok"}

Check Map Data

curl -fsS http://localhost:8000/api/v1/circuits/map | python3 -m json.tool | head -20

Check Dashboard

curl -fsS http://localhost:8000/api/v1/dashboard/summary | python3 -m json.tool | head -20

API Tests and Lint (pytest + httpx)

The API has an integration test harness under apps/api/tests/ (pytest, pytest-asyncio, and an in-process httpx client against the ASGI app). Run it from the API venv:

cd apps/api
.venv/bin/pytest tests/ -v
.venv/bin/ruff check app tests

Key facts for operators:

• Tests run against a dedicated `circuitos_test` database, never the live circuitos database. The default TEST_DATABASE_URL is postgresql+asyncpg://circuitos:circuitos@127.0.0.1:55432/circuitos_test (override via the TEST_DATABASE_URL env var).
• conftest.py hard-refuses any target database whose name does not end in _test, and it auto-creates the test database and runs migrations on first use.
• The harness forces ALLOW_DEV_AUTH=true; with no identity headers, requests resolve to dev@xiber.com (auto-bootstrapped Super Admin), so tests exercise authorized paths by default.

Compile Check (Python)

python3 -m compileall apps/api/app apps/api/scripts apps/mcp/app

Web Gates (Next.js)

cd apps/web
npm run typecheck   # tsc --noEmit
npm run lint        # next lint
npm run build       # next build

Troubleshooting

Map Doesn't Load

Frontend 500 on Server-Rendered Pages

• Verify API_BASE_URL=http://api:8000 is set in Docker Compose
• Verify the api container is running: docker-compose logs api
• Check for Python exceptions in API logs

No Sample Circuits Showing

• Run the seed script (see above)
• Verify: curl http://localhost:8000/api/v1/circuits returns data
• Check migrations: docker-compose exec api alembic current

PostGIS Extension Errors

• Rebuild the custom PostgreSQL image: docker-compose build postgres
• Verify infra/docker/postgres/Dockerfile is referenced in docker-compose.yml
• Check: docker-compose exec postgres psql -U circuitos -c "SELECT PostGIS_Version();"

Container Won't Start

docker-compose logs <service>     # Check for errors
docker-compose down -v            # Nuclear option: remove volumes and rebuild
docker-compose up --build

Warning: down -v destroys all data. Re-run migrations and seed script after.