Found an interesting open-source all-in-one comms + crm + agent OSS project @macrodotcom yesterday.

I was very impressed but they don’t have a documentation site available.

Introducing the new grok-wiki 0 to 1 documentation feature. Automatically generate a beautiful documentation by providing a GitHub URL or folder path.

It’s live in the latest v0.0.16 release.

You can also point your agent directly to https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e… to gain immediate context.

Macro Engineering Documentation

Source: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e Agent-readable docs

Technical documentation for the Macro monorepo: SolidJS app, Rust services, sync worker, AI tooling, infrastructure, local development, APIs, configuration, and operations.

Full Markdownllms.txtMarkdown aliasGitHub## Pages

1. OverviewOverview: Public entry points, repository surfaces, runtime assumptions, and the shortest successful path through the docs.
2. InstallationInstall: Required tools, Nix option, encrypted environment setup, Docker resources, LocalStack, FusionAuth, and first setup command.
3. Local development quickstartQuickstart: Bring up local services with just recipes, shared Compose resources, database initialization, and expected health signals.
4. Frontend quickstartQuickstart: Run the SolidJS app, choose local or remote services, understand app routing, and start Tauri development.
5. Local E2E smoke testsGuide: Run deterministic Playwright and ignored Rust smoke tests against the local-only stack and shared seed fixtures.
6. Monorepo mapConcept: Workspace boundaries, package managers, major runtime folders, generated documentation, and where source-of-truth manifests live.
7. Runtime environments and service URLsConcept: Environment values, local versus dev versus production URL resolution, CORS rules, and frontend service selection.
8. Service topologyConcept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks.
9. Entities and blocksConcept: Item types, document-backed and non-document blocks, aliases, load sources, nesting rules, and file type resolution.
10. Split layout and navigationConcept: Route encoding, component registry entries, split history, navigation causes, popovers, and desktop versus mobile split behavior.
11. Real-time document syncConcept: Sync-service worker routes, Durable Object sessions, permission tokens, Bebop messages, snapshot lifecycle, and client reconnect behavior.
12. Run backend services locallyGuide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks.

Complete Markdown

# Macro Engineering Documentation

> Technical documentation for the Macro monorepo: SolidJS app, Rust services, sync worker, AI tooling, infrastructure, local development, APIs, configuration, and operations.

## Context Links

- [Agent index](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/llms.txt)
- [Human interactive docs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e)
- [GitHub repository](https://github.com/macro-inc/macro)

## Repository Metadata

- Repository: macro-inc/macro

- Generated: 2026-06-01T01:07:41.847Z
- Updated: 2026-06-01T02:14:57.236Z
- Runtime: Pi · Codex · gpt-5.5
- Format: Documentation
- Pages: 29

## Page Index

- 01. [Overview](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/01-overview.md) - Overview: Public entry points, repository surfaces, runtime assumptions, and the shortest successful path through the docs.
- 02. [Installation](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/02-installation.md) - Install: Required tools, Nix option, encrypted environment setup, Docker resources, LocalStack, FusionAuth, and first setup command.
- 03. [Local development quickstart](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/03-local-development-quickstart.md) - Quickstart: Bring up local services with just recipes, shared Compose resources, database initialization, and expected health signals.
- 04. [Frontend quickstart](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/04-frontend-quickstart.md) - Quickstart: Run the SolidJS app, choose local or remote services, understand app routing, and start Tauri development.
- 05. [Local E2E smoke tests](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/05-local-e2e-smoke-tests.md) - Guide: Run deterministic Playwright and ignored Rust smoke tests against the local-only stack and shared seed fixtures.
- 06. [Monorepo map](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/06-monorepo-map.md) - Concept: Workspace boundaries, package managers, major runtime folders, generated documentation, and where source-of-truth manifests live.
- 07. [Runtime environments and service URLs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/07-runtime-environments-and-service-urls.md) - Concept: Environment values, local versus dev versus production URL resolution, CORS rules, and frontend service selection.
- 08. [Service topology](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/08-service-topology.md) - Concept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks.
- 09. [Entities and blocks](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/09-entities-and-blocks.md) - Concept: Item types, document-backed and non-document blocks, aliases, load sources, nesting rules, and file type resolution.
- 10. [Split layout and navigation](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/10-split-layout-and-navigation.md) - Concept: Route encoding, component registry entries, split history, navigation causes, popovers, and desktop versus mobile split behavior.
- 11. [Real-time document sync](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/11-real-time-document-sync.md) - Concept: Sync-service worker routes, Durable Object sessions, permission tokens, Bebop messages, snapshot lifecycle, and client reconnect behavior.
- 12. [Run backend services locally](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/12-run-backend-services-locally.md) - Guide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks.
- 13. [Develop SolidJS and Tauri apps](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/13-develop-solidjs-and-tauri-apps.md) - Guide: Use Bun and Vite, run local service overrides, build app bundles, start Tauri targets, and avoid iOS worker deadlocks.
- 14. [Change a Rust service API](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/14-change-a-rust-service-api.md) - Guide: Update Axum handlers, OpenAPI emitters, generated client specs, Orval output, and CI checks for service API changes.
- 15. [Generate service clients and tool schemas](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/15-generate-service-clients-and-tool-schemas.md) - Guide: Regenerate OpenAPI JSON, TypeScript clients, DCS model types, AI tool schemas, and MCP documentation pages from Rust sources.
- 16. [Work with local seed data](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/16-work-with-local-seed-data.md) - Guide: Use seed_cli, local_e2e fixtures, manifest aliases, reset SQL, and shared Playwright and Rust fixture loaders.
- 17. [Storage and workspace APIs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/17-storage-and-workspace-apis.md) - Reference: Document, channel, project, call, CRM, soup, pin, history, permissions, properties, and search endpoints.
- 18. [Auth and team APIs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/18-auth-and-team-apis.md) - Reference: Login, OAuth callbacks, JWT refresh, account links, team membership, invites, permissions, billing, and user endpoints.
- 19. [Email and notification APIs](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/19-email-and-notification-apis.md) - Reference: Gmail init and sync, drafts, threads, labels, attachments, notification preferences, unread state, and unsubscribe routes.
- 20. [AI chat streaming API](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/20-ai-chat-streaming-api.md) - Reference: DCS chat endpoints, stream payloads, toolset selection, model identifiers, extraction statuses, structured completion, and stop semantics.
- 21. [MCP server and tool registry](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/21-mcp-server-and-tool-registry.md) - Reference: Remote MCP endpoint, OAuth-backed server, toolset composition, SendEmail boundary, generated schemas, and docs tool pages.
- 22. [Environment variables reference](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/22-environment-variables-reference.md) - Reference: Required and optional service environment variables, defaults, secrets, queue names, ports, and provider-specific keys.
- 23. [Database migrations and SQLx cache](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/23-database-migrations-and-sqlx-cache.md) - Operations: MacroDB migrations, local database just recipes, SQLx offline cache, fixture data, and migration validation workflow.
- 24. [Feature flags and server selection](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/24-feature-flags-and-server-selection.md) - Reference: Vite environment overrides, feature flag defaults, local service selection syntax, sync-service host selection, and runtime host helpers.
- 25. [Infrastructure stacks reference](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/25-infrastructure-stacks-reference.md) - Reference: Pulumi stack layout, reusable resources, service stacks, buckets, queues, Datadog sidecars, FusionAuth, and deployment inputs.
- 26. [Build, test, and quality gates](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/26-build-test-and-quality-gates.md) - Operations: Rust checks, clippy, SQLx preparation, TypeScript checks, Biome, Tailwind hygiene, Vitest, Playwright, and CI path filters.
- 27. [Deploy services and web app](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/27-deploy-services-and-web-app.md) - Operations: Generic service deployment, web app deployment, production release workflow, database migrations, Pulumi stacks, and deployment concurrency.
- 28. [Observability and debugging](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/28-observability-and-debugging.md) - Operations: Backend tracing initialization, Datadog ECS sidecars, browser RUM and logs, sourcemaps, local debug signals, and iOS WebView freeze diagnosis.
- 29. [Documentation site maintenance](https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/29-documentation-site-maintenance.md) - Contribution: Maintain the Mintlify docs site, navigation manifest, generated MCP pages, local preview, broken-link checks, and writing conventions.

## Source File Index

- `.github/workflows/code-check-cloud-storage.yml`
- `.github/workflows/deploy-service-generic.yml`
- `.github/workflows/deploy-web-app.yml`
- `.github/workflows/migrate-macro-db.yml`
- `.github/workflows/release-production.yml`
- `.github/workflows/reusable-deploy-service.yml`
- `.github/workflows/web-app-check-main.yml`
- `docker-compose-databases.yml`
- `docker-compose.local-e2e.yml`
- `docker-compose.yml`
- `docs/AGENTS.md`
- `docs/AI/mcp/overview.mdx`
- `docs/AI/mcp/tools/index.mdx`
- `docs/config/tool-pages.json`
- `docs/CONTRIBUTING.md`
- `docs/docs.json`
- `docs/package.json`
- `docs/README.md`
- `docs/scripts/generate-mcp-tool-pages.ts`
- `flake.nix`
- `infra/packages/lambda/src/index.ts`
- `infra/packages/resources/src/index.ts`
- `infra/packages/resources/src/resources/datadog.ts`
- `infra/packages/service/src/index.ts`
- `infra/packages/shared/src/ai_tools.ts`
- `infra/README.md`
- `infra/stacks/document-storage/index.ts`
- `infra/stacks/fusionauth-instance/README.md`
- `infra/stacks/mcp-server/index.ts`
- `js/app/AGENTS.md`
- `js/app/justfile`
- `js/app/package.json`
- `js/app/packages/app/component/Root.tsx`
- `js/app/packages/app/component/split-layout/componentRegistry.tsx`
- `js/app/packages/app/component/split-layout/layout.ts`
- `js/app/packages/app/component/split-layout/layoutManager.ts`
- `js/app/packages/app/component/split-layout/SplitLayoutRoute.tsx`
- `js/app/packages/app/component/split-layout/tests/layoutManager.test.ts`
- `js/app/packages/app/index.tsx`
- `js/app/packages/app/vite.config.ts`
- `js/app/packages/block-md/definition.ts`
- `js/app/packages/block-project/definition.ts`
- `js/app/packages/core/auth/logout.ts`
- `js/app/packages/core/auth/sso.ts`
- `js/app/packages/core/block.ts`
- `js/app/packages/core/component/AI/types/index.ts`
- `js/app/packages/core/constant/allBlocks.ts`
- `js/app/packages/core/constant/featureFlags.ts`
- `js/app/packages/core/constant/servers.ts`
- `js/app/packages/core/tests/featureFlags.test.ts`
- `js/app/packages/observability/src/index.ts`
- `js/app/packages/service-clients/orval.config.ts`
- `js/app/packages/service-clients/service-auth/client.ts`
- `js/app/packages/service-clients/service-auth/openapi.json`
- `js/app/packages/service-clients/service-cognition/client.ts`
- `js/app/packages/service-clients/service-cognition/openapi.json`
- `js/app/packages/service-clients/service-email/client.ts`
- `js/app/packages/service-clients/service-email/openapi.json`
- `js/app/packages/service-clients/service-notification/client.ts`
- `js/app/packages/service-clients/service-notification/openapi.json`
- `js/app/packages/service-clients/service-properties/client.ts`
- `js/app/packages/service-clients/service-properties/openapi.json`
- `js/app/packages/service-clients/service-search/client.ts`
- `js/app/packages/service-clients/service-search/openapi.json`
- `js/app/packages/service-clients/service-storage/client.ts`
- `js/app/packages/service-clients/service-storage/openapi.json`
- `js/app/packages/service-clients/service-sync/client.ts`
- `js/app/packages/service-clients/service-sync/source.ts`
- `js/app/README.md`
- `js/app/scripts/generate-api-schema.ts`
- `js/app/scripts/generate-dcs-tools.ts`
- `js/app/scripts/services.ts`
- `js/app/tauri/src-tauri/README.md`
- `js/app/tests/e2e/fixtures/local-e2e-seed.ts`
- `js/package.json`
- `justfile`
- `local_stack.just`
- `README.md`
- `RUNNING_LOCALLY.md`
- `rust/cloud-storage/agent/src/model.rs`
- `rust/cloud-storage/AGENTS.md`
- `rust/cloud-storage/ai_tools/src/bin/gen_tool_schemas.rs`
- `rust/cloud-storage/ai_tools/src/lib.rs`
- `rust/cloud-storage/authentication_service/src/config.rs`
- `rust/cloud-storage/authentication_service/src/openapi.rs`
- `rust/cloud-storage/Cargo.toml`
- `rust/cloud-storage/chat/.sqlx/query-57fc603adf83fd7c07968425c98f9a57924573692cf0529ad021b6eef00ce99e.json`
- `rust/cloud-storage/database.just`
- `rust/cloud-storage/document_cognition_service/src/api/stream/chat_message/mod.rs`
- `rust/cloud-storage/document_cognition_service/src/api/stream/stop.rs`
- `rust/cloud-storage/document_cognition_service/src/config.rs`
- `rust/cloud-storage/document_cognition_service/src/model/stream.rs`
- `rust/cloud-storage/document_cognition_service/src/openapi.rs`
- `rust/cloud-storage/document_storage_service/src/config.rs`
- `rust/cloud-storage/document_storage_service/src/openapi.rs`
- `rust/cloud-storage/email_service/src/config.rs`
- `rust/cloud-storage/email_service/src/openapi.rs`
- `rust/cloud-storage/integration_tests/local_e2e/README.md`
- `rust/cloud-storage/justfile`
- `rust/cloud-storage/local_e2e_test_support/README.md`
- `rust/cloud-storage/macro_cors/src/lib.rs`
- `rust/cloud-storage/macro_db_client/migrations/0001_baseline.sql`
- `rust/cloud-storage/macro_db_client/README.md`
- `rust/cloud-storage/macro_entrypoint/src/lib.rs`
- `rust/cloud-storage/macro_env/src/lib.rs`
- `rust/cloud-storage/macro_service_urls/src/lib.rs`
- `rust/cloud-storage/mcp_service/src/main.rs`
- `rust/cloud-storage/mcp_service/src/tool_service.rs`
- `rust/cloud-storage/models_soup/src/lib.rs`
- `rust/cloud-storage/notification_service/src/config.rs`
- `rust/cloud-storage/notification_service/src/openapi.rs`
- `rust/cloud-storage/README.md`
- `rust/cloud-storage/search_service/src/config.rs`
- `rust/cloud-storage/seed_cli/README.md`
- `rust/cloud-storage/seed_cli/seed/local_e2e/manifest.json`
- `rust/cloud-storage/seed_cli/seed/local_e2e/reset.sql`
- `rust/cloud-storage/seed_cli/seed/local_e2e/users.json`
- `rust/cloud-storage/seed_cli/src/main.rs`
- `rust/cloud-storage/sqlx.just`
- `rust/sync-service/Cargo.toml`
- `rust/sync-service/src/cf_worker.rs`
- `rust/sync-service/src/durable_object.rs`
- `rust/sync-service/src/generated/schema.rs`
- `rust/sync-service/src/websocket.rs`

---

## 01. Overview

> Overview: Public entry points, repository surfaces, runtime assumptions, and the shortest successful path through the docs.

- Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/01-overview.md
- Generated: 2026-06-01T00:46:58.329Z

### Source Files

- `README.md`
- `RUNNING_LOCALLY.md`
- `js/app/README.md`
- `rust/cloud-storage/AGENTS.md`
- `docker-compose.yml`
- `docs/docs.json`

---
title: "Overview"
description: "Overview: Public entry points, repository surfaces, runtime assumptions, and the shortest successful path through the docs."
---

Macro is a SolidJS and Rust monorepo for the Macro workspace: a static SPA frontend, Rust backend services, generated MCP tool documentation, Pulumi infrastructure, Docker Compose local services, and deterministic local smoke-test fixtures.

## Public entry points

| Surface | Entry point | Backing repository area |
| --- | --- | --- |
| Web application | `https://macro.com/app` | `js/app`, `infra/stacks/web-app` |
| Product docs | `docs.macro.com` site config | `docs/docs.json`, `docs/index.mdx`, `docs/product/*` |
| MCP server | `https://mcp-server.macro.com/mcp` | `rust/cloud-storage/mcp_service`, `rust/cloud-storage/ai_tools`, `docs/AI/mcp/*` |
| Local backend stack | `just run_local` | `justfile`, `docker-compose.yml`, `docker-compose-databases.yml` |
| Local E2E smoke stack | `just local-e2e` | `docker-compose.local-e2e.yml`, `rust/cloud-storage/seed_cli`, `js/app/tests/e2e` |

<Note>
The local setup is explicitly single-instance. The root `justfile` exports `COMPOSE_PROJECT_NAME=macro`, and the Compose files use the fixed project name `macro`, so multiple checkouts share the same local containers, volumes, networks, LocalStack, and FusionAuth resources.
</Note>

## Repository surfaces

:::files
```text
.
├── README.md                         # product, security, license, community
├── RUNNING_LOCALLY.md                # supported local setup and E2E path
├── justfile                          # root setup, local stack, E2E orchestration
├── docker-compose*.yml               # local services, databases, E2E overrides
├── local_stack.just                  # LocalStack SQS, DynamoDB, and S3 setup
├── docs/                             # public docs site and generated MCP docs
├── js/
│   ├── app/                          # SolidJS/Vite frontend workspace
│   ├── lexical-service/              # Lexical conversion service
│   └── websocket-service/            # Bun websocket service
├── rust/
│   ├── cloud-storage/                # Rust service workspace
│   └── sync-service/                 # sync service container
├── infra/                            # Pulumi stacks and shared service package
└── agents/transcription/             # transcription agent service

:::

Frontend

The frontend lives in js/app and uses Bun, Vite, SolidJS, Tailwind, TypeScript, and workspace packages under js/app/packages/*. The app package builds a static SPA with base path /app; development serves from / on port 3000 by default.

Common commands:

cd js/app
bun i
just local
bun run check
bun run local:e2e

The Vite config exposes build-time values such as import.meta.env.__APP_VERSION__, ASSETS_PATH, __LOCAL_DOCKER__, __LOCAL_JWT__, and __GIT_BRANCH__.

Rust services

The main backend surface is the Cargo workspace in rust/cloud-storage. It contains service binaries, worker binaries, Lambda handlers, database clients, domain crates, generated schemas, and integration tests.

Representative service binaries include:

rust/cloud-storage/Dockerfile.dev builds the standard service bundle into the macro-local-rust-services:dev image. search_processing_service uses a separate Dockerfile and is behind the processors Compose profile.

Docs and MCP reference

The docs site is configured by docs/docs.json. Current navigation contains:

• Getting Started: docs/index.mdx
• Product pages: AI Chat, Email, Channels, Docs, Canvas, Tasks, Search, Files
• MCP setup: docs/AI/mcp/overview.mdx
• Generated MCP tool pages: docs/AI/mcp/tools/*
• Changelog: docs/changelog/introduction

MCP tool pages are generated from the Rust tool registry:

cd docs
bun install
bun run generate:tools
bun run dev

The generator builds gen_tool_schemas from rust/cloud-storage, reads rust/cloud-storage/ai_tools/schemas/tools.json, writes MDX pages under docs/AI/mcp/tools, and updates docs/config/tool-pages.json.

Infrastructure

Infrastructure lives under infra/stacks/* and shared Pulumi packages under infra/packages/*.

Important stacks include:

Runtime assumptions

Required local tools

RUNNING_LOCALLY.md lists these local prerequisites:

• sops
• Docker
• AWS CLI/configuration
• SQLx / sqlx-cli
• Pulumi
• Bun
• Node
• just

If Nix is available, the repo supports nix develop for toolchain management.

Local data plane

The default local stack uses:

Local E2E overrides force services onto local Postgres, Redis, LocalStack queues, LocalStack buckets, and deterministic seed data.

Environment and secrets

The root setup path decrypts .env-local*.enc into .env with sops, patches local FusionAuth values when available, creates shared Docker networks and volumes, initializes local AWS resources, initializes databases, and builds the Rust dev service image.

just setup

Use just get_environment when only the .env file is needed.

Shortest successful path

just setup

just run_local

just run_local --build

cd js/app
bun i
just local

just local-e2e

just local-e2e-rust

just local-e2e-all

Verification signals

Failure modes to check first

Related pages

02. Installation

Install: Required tools, Nix option, encrypted environment setup, Docker resources, LocalStack, FusionAuth, and first setup command.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/02-installation.md
• Generated: 2026-06-01T00:46:44.840Z

Source Files

• RUNNING_LOCALLY.md
• flake.nix
• justfile
• local_stack.just
• docker-compose-databases.yml
• rust/cloud-storage/README.md

Macro local installation is driven by just setup at the repository root. The setup path decrypts .env, creates shared Docker networks and volumes, provisions LocalStack AWS resources, initializes the local Postgres database, configures a local FusionAuth stack, and builds local Rust service images.

Prerequisites

Nix option

If Nix is available, use the repository flake instead of installing most tools manually:

nix develop

The default shell provides the Rust toolchain, just, bun, pulumi, sops, sqlx-cli, Node.js 24, Docker Compose tooling, jq, and related backend/frontend utilities. It also sets SOPS_KMS_ARN for encrypted environment decryption.

For Tauri/frontend platform work, use the JavaScript app shell:

nix develop .#js-app

Encrypted environment setup

The root justfile decrypts encrypted dotenv bundles into a plain root .env file.

If you are not inside nix develop, export the KMS ARNs before decrypting:

export SOPS_KMS_ARN="arn:aws:kms:us-east-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93,arn:aws:kms:us-west-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93"

When just get_environment dev is used and ~/.aws/credentials exists, the recipe replaces AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in .env from the [default] profile.

First setup command

Run from the repository root:

just setup

just setup performs this sequence:

Expected final output:

Setup complete.

Docker resources

Local Docker resources are intentionally frozen to the macro Compose project. Multiple checkouts and worktrees share the same containers, volumes, networks, LocalStack instance, and FusionAuth instance.

Shared networks and volumes

just create_networks creates:

Database Compose services

docker-compose-databases.yml defines:

The local database URL used by cloud-storage justfiles is:

postgres://user:password@localhost:5432/macrodb

LocalStack

Local AWS emulation is handled by local_stack.just.

just setup_localstack

The setup starts localstack/localstack:4 with:

The recipe provisions SQS queues used by notification, email, contacts, conversion, document deletion, document upload finalization, document text extraction, search events, and static-file events. It also creates these DynamoDB tables:

S3 buckets created locally:

All buckets receive CORS rules for http://localhost:3000 through http://localhost:3009. The doc-storage bucket is also wired so s3:ObjectCreated:* events send messages to document-upload-finalizer-queue.

Backend AWS clients switch to LocalStack when LOCAL_AWS_URL is set. In that mode, the Rust AWS config uses us-east-1, test credentials, the configured endpoint URL, and path-style S3 behavior.

FusionAuth

The local FusionAuth stack lives under infra/stacks/fusionauth-instance.

just setup_fusionauth

The root setup command calls the same stack setup through:

just infra/stacks/fusionauth-instance/setup

Local FusionAuth uses:

Local admin credentials documented by the stack:

username: [email protected]
password: macroIsGreat!
api-key: bf69486b-4733-4954-a44e-2e1b5f2c8a91

During setup, the FusionAuth recipe patches root .env with local values including:

just run_local also calls just patch_local_fusionauth_env, which patches .env again if the local Pulumi stack exists. If FusionAuth is not running, that recipe starts it temporarily to read the OAuth client secret, then stops it.

Running after installation

Start backend services:

just run_local

If service images changed, rebuild selected images while starting:

just run_local --build

By default, convert_service and search_processing_service are not required for the frontend dev path. search_processing_service is behind the processors Compose profile and is pinned to linux/amd64 because its local PDFium library is amd64-only.

Start the frontend against local services:

cd js/app
bun i
just local

Local E2E smoke setup

After installation:

just local-e2e

This uses docker-compose.local-e2e.yml overrides so services use local Postgres and LocalStack instead of shared dev assets, seeds deterministic fixture data, launches the frontend with local bearer-token auth, and runs Playwright.

Additional E2E commands:

just local-e2e-rust
just local-e2e-all
just local-e2e-ui

The local E2E seed path is guarded: it requires LOCAL_E2E_SEED=true and rejects database URLs that are not the local Docker database shape postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb.

Cleanup and reset

Troubleshooting

Related pages

03. Local development quickstart

Quickstart: Bring up local services with just recipes, shared Compose resources, database initialization, and expected health signals.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/03-local-development-quickstart.md
• Generated: 2026-06-01T00:46:27.751Z

Source Files

• RUNNING_LOCALLY.md
• justfile
• docker-compose.yml
• docker-compose-databases.yml
• local_stack.just
• infra/stacks/fusionauth-instance/README.md

Local development is orchestrated from the root justfile: just setup prepares .env, shared Docker resources, LocalStack, Postgres migrations, FusionAuth, and cached Rust service images; just run_local starts the Docker Compose service stack under the fixed macro Compose project.

Prerequisites

Install the tools expected by RUNNING_LOCALLY.md and the recipes:

If not using the repository Nix shell, export the configured SOPS KMS ARN before decrypting local environment files.

export SOPS_KMS_ARN="arn:aws:kms:us-east-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93,arn:aws:kms:us-west-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93"

Quickstart

```bash
just setup
```

This decrypts `.env`, creates shared Docker networks and volumes, initializes LocalStack, creates and migrates the local database, configures FusionAuth, and prebuilds development service images.

```bash
just run_local
```

To rebuild service containers after local service changes, pass the build flag.

```bash
just run_local --build
```

```bash
cd js/app
bun i
just local
```

`just local` runs the app with `VITE_LOCAL_SERVERS=ALL` and defaults to `PORT=3000`.

Shared Docker resources

create_networks creates the shared resources used across the root Compose file, the database Compose file, and the FusionAuth Compose file.

The root docker-compose.yml includes docker-compose-databases.yml and infra/stacks/fusionauth-instance/docker-compose.yml, so just run_local can start application services together with Postgres, Redis, OpenSearch, and FusionAuth.

What just setup initializes

The local Macro database URL used by the Rust database recipes is:

postgres://user:password@localhost:5432/macrodb

LocalStack resources

local_stack.just starts LocalStack as a standalone Docker container named localstack on the shared databases network.

All buckets receive a local CORS policy for http://localhost:3000 through http://localhost:3009.

Service ports and health signals

Most Rust services expose container port 8080 and map it to a distinct host port.

Use Docker Compose waiting when starting a subset or detached stack:

just run_local -d --wait

Frontend local service selection

js/app/justfile provides local frontend shortcuts. The default local command selects all local service hosts.

cd js/app
just local

The underlying app config reads VITE_LOCAL_SERVERS:

List recognized local service names with:

cd js/app
just local-services

Local E2E smoke path

The local smoke harness runs the backend with local-only service overrides, seeds deterministic fixtures, starts the frontend with local bearer-token auth, and runs Playwright.

just local-e2e

Related harnesses:

just local-e2e-rust
just local-e2e-all
just local-e2e-ui

The E2E path uses docker-compose.local-e2e.yml to override database, Redis, LocalStack, bucket, table, and queue environment variables so the smoke suite does not mutate shared dev assets. The seed command is guarded by LOCAL_E2E_SEED=true and rejects any DATABASE_URL outside the local Docker database shape:

postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb

Stopping and resetting

Troubleshooting

.env not found

just run_local patches local FusionAuth values into .env. If .env is missing, run:

just get_environment

For a fresh checkout, prefer the full setup:

just setup

FusionAuth local stack not found

If patch_local_fusionauth_env reports that the Pulumi local stack is missing, run:

just setup

FusionAuth local admin defaults are:

username: [email protected]
password: macroIsGreat!
api-key: bf69486b-4733-4954-a44e-2e1b5f2c8a91

Local E2E token generation fails

Prefer the repository-level harness:

just local-e2e

If running Playwright directly, seed first and ensure .env exists:

just local-e2e-seed
cd js/app
LOCAL_E2E=true bunx playwright test

You can bypass token generation by exporting LOCAL_JWT.

Search processing on Apple Silicon

search_processing_service is pinned to linux/amd64 because its local pdfium library is AMD64-only. On Apple Silicon, enabling the processor profile uses QEMU emulation for build and runtime.

Related pages

04. Frontend quickstart

Quickstart: Run the SolidJS app, choose local or remote services, understand app routing, and start Tauri development.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/04-frontend-quickstart.md
• Generated: 2026-06-01T00:46:48.071Z

Source Files

• js/app/README.md
• js/app/package.json
• js/app/justfile
• js/app/packages/app/index.tsx
• js/app/packages/app/component/Root.tsx
• js/app/AGENTS.md

The Macro frontend is a Bun-managed Vite/SolidJS SPA in js/app; the web app serves under /app, while Tauri loads the same frontend bundle through native desktop and mobile shells.

Prerequisites

• Bun
• just
• Optional: nix develop from the repository root to enter a shell with the expected toolchain
• For Tauri: Rust, Tauri CLI, and platform SDKs for Android or iOS

Run the web app

bun run dev runs cd packages/app && bun run --bun dev, which starts Vite with packages/app/vite.config.ts. The dev server defaults to port 3000, binds to 0.0.0.0, uses strict port mode, and enables HMR over WebSocket.

Choose remote or local services

In development mode, service endpoints come from packages/core/constant/servers.ts.

Useful shortcuts from js/app/justfile:

cd js/app

just local-services
just local
just local-dss
just local-search
just local-email

App routing model

The runtime selects the router by platform:

The canonical authenticated landing route is:

/component/inbox

The root route / preloads user info and history, stores known campaign query parameters as short-lived cookies, normalizes short IDs in the URL, then redirects:

The split-layout route decodes URL segments as type/id pairs:

/component/inbox
/<block-or-alias>/<id>
/component/inbox/<block-or-alias>/<id>

If no valid pair is present, the layout falls back to the inbox component.

Top-level auth and utility routes include:

Runtime initialization

packages/app/index.tsx performs the browser entrypoint work:

• Imports global CSS and font packages.
• Initializes Lexical markdown support.
• Initializes monochrome icon behavior.
• Sets document.documentElement.dataset.platform to web, desktop, ios, or android.
• Tracks current input modality as keyboard, mouse, or touch.
• In Tauri, proxies non-localhost fetch calls through @tauri-apps/plugin-http for native compatibility.
• In development mode, wraps Root in a Solid ErrorBoundary.
• Outside Vite HMR sessions, lazy-loads observability and listens for vite:preloadError to prompt a refresh after deployments.

Root mounts global providers for analytics, PostHog, entities, user context, query sync, undo, channels, calls, quick access, search, chat attachments, notifications, split layout routing, onboarding, and toasts.

Build and preview

cd js/app

bun run build
bun run preview

Build output is emitted from packages/app into packages/app/dist. Production-style builds use /app as the Vite base.

Additional build recipes:

cd js/app

just build-dev
just build-staging
just build-prod

Local smoke tests

Prefer the repository-level harness for deterministic local E2E:

just local-e2e

That command starts the local service subset with compose overrides, seeds deterministic data, and runs Playwright with LOCAL_E2E=true.

For Playwright UI mode:

just local-e2e-ui

Start Tauri development

The Tauri shell lives under js/app/tauri and packages the same packages/app frontend.

cd js/app/tauri

cargo tauri dev
cargo tauri android dev
cargo tauri ios dev

Tauri config uses:

Set TAURI_DEV_HOST when a device or emulator needs HMR to connect to a host other than localhost.

TAURI_DEV_HOST=192.168.1.20 cargo tauri android dev

Contributor checks

Common frontend checks from js/app/AGENTS.md:

cd js/app

bun run test
bun run check
bun run lint
bun run format
bun run knip

Next

• Use RUNNING_LOCALLY.md for backend stack setup.
• Use js/app/AGENTS.md for frontend contribution conventions.
• Use js/app/tauri/src-tauri/README.md for native shell development notes.

05. Local E2E smoke tests

Guide: Run deterministic Playwright and ignored Rust smoke tests against the local-only stack and shared seed fixtures.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/05-local-e2e-smoke-tests.md
• Generated: 2026-06-01T00:49:10.463Z

Source Files

• RUNNING_LOCALLY.md
• justfile
• docker-compose.local-e2e.yml
• rust/cloud-storage/seed_cli/README.md
• js/app/tests/e2e/fixtures/local-e2e-seed.ts
• rust/cloud-storage/integration_tests/local_e2e/README.md
• rust/cloud-storage/local_e2e_test_support/README.md

just local-e2e, just local-e2e-rust, and just local-e2e-all start the Macro local stack with docker-compose.local-e2e.yml overrides, reset and seed deterministic fixtures from rust/cloud-storage/seed_cli/seed, then run Playwright or ignored Rust integration smoke suites against localhost services.

Prerequisites

Run the normal local setup first:

just setup

The harness expects the same local toolchain used by the repository: Docker, Bun, Rust/Cargo, AWS CLI, SQLx tooling, Pulumi, and a decrypted .env. Local Docker resources are intentionally frozen to the macro Compose project, so do not run two local stacks from different checkouts at the same time.

Command reference

Forward Playwright file filters or flags through the just target:

just local-e2e tests/e2e/local-smoke.spec.ts
just local-e2e --project=local-chromium

For Rust filters, pass the test name after the target:

just local-e2e-rust channel_message_posts_to_http_and_delivers_to_websocket

Runtime shape

just local-e2e*
  ├─ setup_localstack
  │  ├─ LocalStack on localhost:4566
  │  ├─ SQS queues
  │  ├─ DynamoDB tables
  │  └─ S3 buckets + doc-storage notification wiring
  ├─ COMPOSE_FILE=docker-compose.yml:docker-compose.local-e2e.yml just run_local -d --wait ...
  │  ├─ Postgres localhost:5432
  │  ├─ Redis localhost:6379
  │  └─ Macro service subset on local ports
  ├─ just local-e2e-seed
  └─ Playwright and/or ignored Rust tests

The compose override pins application services to local Postgres, Redis, and LocalStack instead of shared dev assets. It also supplies deterministic bucket, queue, table, AWS credential, and region values for the local smoke stack.

The startup service subset is defined in the root justfile as:

authentication-service
connection_gateway
contacts_service
document_storage_service
email_service
notification_service
static_file_service
static_file_cdn
sync_service
websocket_service

Common local ports used by the tests include:

Seed fixtures

The shared seed contract lives under rust/cloud-storage/seed_cli/seed.

The seed CLI scenario resets fixture ranges, inserts users and verification rows, then seeds documents, channels, and channel messages. A successful seed prints:

Local e2e smoke seed data ready for macro|[email protected]

Playwright local smoke suite

When LOCAL_E2E=true, js/app/playwright.config.ts switches to a local-only browser project named local-chromium. The Playwright web server command sets:

PORT=${PORT:-3000}
VITE_LOCAL_SERVERS=ALL
VITE_ENABLE_BEARER_TOKEN_AUTH=true
LOCAL_JWT=<generated token>
bun run dev

If LOCAL_JWT is not already exported, Playwright runs js/app/scripts/generate-local-e2e-token.ts. That script calls the Rust local_e2e_test_support binary generate_local_e2e_token, which loads the shared seed user and signs a Macro API token using .env or process environment values. The default local token lifetime is eight hours.

The local Playwright specs skip unless LOCAL_E2E=true:

Tests import localE2ESeed from js/app/tests/e2e/fixtures/local-e2e-seed.ts. That fixture loader reads the Rust seed JSON files directly and exposes raw arrays, lookup maps, and smoke aliases.

Rust local E2E suite

The Rust suite lives in rust/cloud-storage/integration_tests/local_e2e and is intentionally marked #[ignore] so normal workspace test runs do not require Docker services. Run it through:

just local-e2e-rust

The crate uses local_e2e_test_support to load the same seed files, generate local JWTs, and resolve local service endpoints. The support crate rejects non-local service URLs for mutating tests.

Supported local overrides include:

The ignored Rust tests cover channel mutations, persistence checks, notifications/contacts side effects where workers are available, and connection gateway WebSocket delivery.

Running Playwright directly

Prefer the repository-level harness. If you need a manual loop, start and seed the stack first:

AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 just setup_localstack
COMPOSE_FILE=docker-compose.yml:docker-compose.local-e2e.yml just run_local -d --wait authentication-service connection_gateway contacts_service document_storage_service email_service notification_service static_file_service static_file_cdn sync_service websocket_service
just local-e2e-seed
cd js/app
LOCAL_E2E=true bunx playwright test

You can bypass token generation by exporting LOCAL_JWT, but the generated token path is the default and keeps Playwright aligned with the Rust helper.

Troubleshooting

Failed to generate LOCAL_JWT for LOCAL_E2E Playwright

Ensure .env exists from just get_environment or just setup, then rerun the repo-level harness:

just local-e2e

If running Playwright directly, seed first with just local-e2e-seed. As a temporary bypass, export LOCAL_JWT.

Seed refuses to run

Use just local-e2e-seed rather than invoking the seed CLI manually. The recipe sets LOCAL_E2E_SEED=true, local AWS values, and the expected local database URL.

Tests refuse a service URL

Rust helpers reject non-local service URLs. Check LOCAL_E2E_DOCUMENT_STORAGE_URL, LOCAL_E2E_CONNECTION_GATEWAY_WS_URL, and LOCAL_E2E_NOTIFICATION_URL; they must point at localhost or 127.0.0.1.

Local stack conflicts across checkouts

The Compose project name, networks, and volumes are shared as macro. Stop the current stack before starting another checkout:

just stop-local
just stop-databases

Adding local E2E coverage

1. Put stable shared fixture rows in rust/cloud-storage/seed_cli/seed.
2. Add only aliases to local_e2e/manifest.json; keep the source data in the fixture files.
3. Use localE2ESeed in Playwright and LocalE2eSeed in Rust instead of duplicating IDs.
4. Keep Playwright specs gated by LOCAL_E2E=true.
5. Keep Rust integration tests marked #[ignore].
6. Use generated unique text or IDs for mutating assertions so repeated smoke runs can coexist with the deterministic base seed.

Related pages

• RUNNING_LOCALLY.md
• rust/cloud-storage/seed_cli/README.md
• rust/cloud-storage/integration_tests/local_e2e/README.md
• rust/cloud-storage/local_e2e_test_support/README.md

06. Monorepo map

Concept: Workspace boundaries, package managers, major runtime folders, generated documentation, and where source-of-truth manifests live.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/06-monorepo-map.md
• Generated: 2026-06-01T00:50:29.936Z

Source Files

• rust/cloud-storage/Cargo.toml
• rust/sync-service/Cargo.toml
• js/package.json
• js/app/package.json
• infra/README.md
• docs/docs.json

Macro is organized as several adjacent workspaces rather than one repository-wide package workspace: Rust backend services live under rust/cloud-storage, the Cloudflare Durable Object sync worker lives under rust/sync-service, the Solid/Tauri app and JS services live under js, Pulumi infrastructure lives under infra, and the Mintlify documentation site lives under docs.

Repository boundary map

.
├── justfile                         # Local orchestration entrypoint
├── docker-compose*.yml              # Local service, database, and E2E topology
├── flake.nix                        # Nix shells and cached Rust/API build jobs
├── rust/
│   ├── rust-toolchain.toml          # Rust toolchain source of truth
│   ├── cloud-storage/               # Main Rust Cargo workspace
│   └── sync-service/                # Cloudflare Worker/Durable Object sync runtime
├── js/
│   ├── package.json                 # Bun workspace for app + selected JS packages
│   ├── app/                         # Solid app, package libraries, tests, Tauri frontend
│   ├── lexical-core/                # Shared Lexical package
│   ├── lexical-service/             # Wrangler service for Lexical conversion
│   ├── websocket-service/           # Standalone Bun WebSocket service
│   └── analytics-proxy/             # Standalone Wrangler proxy
├── infra/
│   ├── package.json                 # Bun workspace for Pulumi stacks/packages
│   ├── stacks/                      # Deployable Pulumi projects
│   └── packages/                    # Shared Pulumi resource libraries
└── docs/
    ├── docs.json                    # Rendered docs navigation/config
    ├── package.json                 # Mintlify scripts
    └── AI/mcp/tools/                # Generated MCP tool reference pages

Package managers and workspace manifests

Nested package-manager exceptions

• js/loro-mirror/package.json has pnpm scripts and pnpm overrides. It is included by the js/package.json workspace list, but its own scripts call pnpm.
• js/websocket-service and js/analytics-proxy have package manifests but are not listed in the js/package.json workspace array.
• infra/stacks/web-app has a Pulumi project, but infra/tsconfig.json explicitly excludes stacks/web-app from the infra TypeScript check.

Major runtime folders

Backend Cargo workspace

rust/cloud-storage/Cargo.toml owns the backend Cargo workspace membership and shared dependency versions. Most backend crates set publish = false and depend on sibling crates by relative path.

Common commands run from rust/cloud-storage:

just check          # SQLX_OFFLINE=true cargo check with warnings denied
just clippy         # workspace clippy with all features
just format         # cargo fmt
just prepare_db     # refresh SQLx offline cache through sqlx.just
just build          # SQLX_OFFLINE=true cargo build

Database lifecycle commands are routed through rust/cloud-storage/macro_db_client/justfile:

just macro_db_client/create_db
just macro_db_client/migrate_db
just macro_db_client/drop_db -y -f

MacroDB schema migrations live under rust/cloud-storage/macro_db_client/migrations. Several worker/test crates also carry local migrations for their own test schemas.

Sync worker boundary

rust/sync-service is a separate Cargo project compiled to WebAssembly for Cloudflare Workers. Its wrangler.toml defines:

The Dockerfile installs Node 22, worker-build, Wrangler dependencies, builds Bebop TypeScript bindings from bebop/schema.bop, applies local D1 migrations, and starts Wrangler on port 8787.

Frontend and app packages

js/package.json declares the Bun workspace boundary:

[
  "app",
  "app/packages/*",
  "app/infra/*",
  "loro-mirror",
  "lexical-core",
  "lexical-service"
]

js/app/package.json is the app-level command surface:

cd js/app
bun run dev          # Vite app through packages/app
bun run build        # Vite production-style build
bun run test         # Vitest
bun run local:e2e    # Playwright with LOCAL_E2E=true
bun run check        # TypeScript + Biome
bun run gen-api      # Rust OpenAPI binaries -> service-clients
bun run gen-tools    # Rust AI tool schemas -> DCS tool types

js/app/tsconfig.json includes packages/**/*, ../loro-mirror/**/*, and ../lexical-core/**/*, and defines path aliases for generated service clients such as @service-storage/*, @service-cognition/*, @service-email/*, and @service-sync/*.

Tauri packaging

The Tauri workspace lives in js/app/tauri. src-tauri/tauri.conf.json points frontendDist at ../../packages/app/dist, uses http://localhost:3000 for development, and delegates build hooks back to the app justfile:

{
  "beforeDevCommand": { "script": "just dev-tauri", "cwd": "../.." },
  "beforeBuildCommand": { "script": "just build-tauri", "cwd": "../.." }
}

Use js/app/justfile for app-local run modes:

cd js/app
just local                    # Vite against local services
just local-dcs                # Local cognition service only
just local-dss                # Local document storage service only
just build-prod               # Production-mode frontend bundle
just dist-archive             # Zip dist for native_app_server tests

Infrastructure workspace

Infrastructure code is TypeScript Pulumi under infra. The root infra/package.json declares Bun workspaces for stacks/* and packages/*, requires Node >=21, and enforces Bun through a preinstall script.

Run Pulumi from inside a stack directory:

cd infra/stacks/document-storage
pulumi up --stack dev
pulumi up --stack prod

Operational defaults from the infra README:

Shared stack constants and exported helpers live in infra/packages/shared/src/index.ts; reusable resource constructors live in infra/packages/resources/src/index.ts.

Local orchestration

The root justfile is the local runtime entrypoint. It freezes Docker Compose resources under the macro project name so multiple checkouts share the same local containers, networks, and volumes.

Primary commands:

just setup          # Environment, networks, LocalStack, DBs, FusionAuth, Rust images
just run_dbs -d     # Postgres + Redis
just run_local      # Docker Compose services
just run_local --build
just stop-local
just local-e2e
just local-e2e-rust
just local-e2e-all

Local service topology is defined by:

The local E2E harness seeds deterministic data through rust/cloud-storage/seed_cli and runs Playwright from js/app.

Generated code and documentation

Source-of-truth checklist

When changing a boundary, update the owning manifest first:

Related pages

07. Runtime environments and service URLs

Concept: Environment values, local versus dev versus production URL resolution, CORS rules, and frontend service selection.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/07-runtime-environments-and-service-urls.md
• Generated: 2026-06-01T00:49:31.354Z

Source Files

• rust/cloud-storage/macro_env/src/lib.rs
• rust/cloud-storage/macro_service_urls/src/lib.rs
• rust/cloud-storage/macro_cors/src/lib.rs
• js/app/packages/core/constant/servers.ts
• js/app/scripts/services.ts
• docker-compose.local-e2e.yml

Macro resolves runtime environments through separate backend, frontend, and tooling selectors: Rust services read ENVIRONMENT, the Vite app reads import.meta.env.MODE plus VITE_LOCAL_SERVERS, and generation scripts read Node process variables such as MODE and LOCAL_BACKEND.

Runtime selectors

Backend environment model

The backend environment enum has three variants:

macro_entrypoint loads .env when present, reads the environment, and uses it for logging/tracing behavior. Local uses pretty tracing without the OpenTelemetry sidecar; dev and prod initialize OTLP tracing to the Datadog agent endpoint on 127.0.0.1:4317.

Backend service URL resolver

Rust code uses EnvExtMacroServiceUrls on macro_env::Environment for canonical backend URLs.

The resolver is used by backend features such as auth, GitHub redirects, notification preferences, and email formatting. URL parse tests verify every resolver method returns a valid Url for all three environments.

Frontend service selection

js/app/packages/core/constant/servers.ts owns browser-facing runtime hosts.

Remote host selection

Remote hosts use a suffix based on Vite mode:

auth-logout is special: development uses a FusionAuth dev logout URL, while production uses https://auth.macro.com/oauth2/logout....

Local host selection

Local hosts are only selected when MODE === 'development'.

Example:

MODE=development VITE_LOCAL_SERVERS=document-storage-service:9000,email-service bun run dev

Frontend local host keys include the backend services plus auth-logout and scheduled-action. Runtime code imports SERVER_HOSTS directly in generated service clients, auth flows, WebSocket clients, image proxying, observability, and static-file helpers.

Sync service host selection

The sync service has a separate host map:

Local sync is selected when the frontend is in development mode and VITE_LOCAL_SERVERS is ALL or contains sync-service.

SYNC_PERMISSION_TOKEN_DSS_HOST intentionally follows the sync service. If sync is remote, permission tokens come from remote document storage so the token is signed with the JWT secret that matches the remote sync service. If sync is local, it uses the selected document storage host from SERVER_HOSTS.

Local Docker and local E2E

docker-compose.yml exposes most Rust services on stable localhost ports while containers listen internally on 8080. Important browser-facing ports include:

Local E2E uses docker-compose.local-e2e.yml to override shared dev infrastructure with local Docker dependencies such as Postgres, Redis, and LocalStack. The Playwright local E2E web server starts Vite with:

VITE_LOCAL_SERVERS=ALL VITE_ENABLE_BEARER_TOKEN_AUTH=true bun run dev

The repository-level harness is:

just local-e2e

CORS rules

Backend Axum services generally install macro_cors::cors_layer(); authentication adds the refresh-token header through cors_layer_with_headers(...).

Default CORS behavior:

The sync service has its own Worker-side CORS implementation. It allows a similar but not identical origin set and accepts preview origins shaped like https://{subdomain}.preview.macro.com.

Service-client and schema generation URLs

js/app/scripts/services.ts defines service metadata for generated clients and tooling:

• services[] maps each service name to dev/prod/local OpenAPI URLs, output directories, and Orval project keys.
• serviceUrl(service) selects local when MODE=local or LOCAL_BACKEND=true, production when MODE=production, and dev otherwise.
• generate-api-schema.ts uses the service list for names, output directories, and Orval keys, but generates OpenAPI JSON by building and running local Rust OpenAPI binaries.
• generate-dcs-models.ts can fetch Document Cognition models from the selected service URL, or read from MODELS_JSON when supplied.

Contribution checklist

When adding or changing a service URL:

Troubleshooting

08. Service topology

Concept: Rust services, workers, queues, storage dependencies, ports, and deployment boundaries used by the local and cloud stacks.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/08-service-topology.md
• Generated: 2026-06-01T00:50:24.621Z

Source Files

• docker-compose.yml
• rust/cloud-storage/Cargo.toml
• rust/cloud-storage/AGENTS.md
• infra/stacks/document-storage/index.ts
• infra/packages/resources/src/index.ts
• infra/packages/shared/src/ai_tools.ts

macro-inc/macro runs the cloud-storage surface as a Rust Cargo workspace deployed either as local Docker Compose containers or as Pulumi-managed AWS ECS/Fargate services, Lambda handlers, SQS queues, S3 buckets, DynamoDB tables, Redis caches, OpenSearch, and Postgres databases.

Runtime boundaries

The repository has two primary topology descriptions:

Local Docker topology

docker-compose.yml freezes the Compose project name to macro, includes the database and FusionAuth compose files, and builds macro-local-rust-services:dev from rust/cloud-storage/Dockerfile.dev.

The dev Dockerfile builds a bundle of standard Rust binaries into /app/out/* and each service chooses a command from that bundle. search_processing_service uses a separate Dockerfile and is pinned to linux/amd64 because its Pdfium library is amd64-only.

Local infrastructure containers

docker-compose-databases.yml provides:

Local networks separate service traffic:

Cloud service boundary

Cloud services are provisioned with Pulumi stacks under infra/stacks. Public Rust HTTP services typically follow the same pattern:

1. Build a service-specific ECR image from rust/cloud-storage/Dockerfile with SERVICE_NAME.
2. Create an ECS/Fargate service in private subnets.
3. Put an Application Load Balancer in public subnets when isPrivate: false.
4. Terminate HTTPS on port 443; redirect HTTP 80 to HTTPS.
5. Forward ALB traffic to container port 8080.
6. Attach service-specific IAM policies for Secrets Manager, SQS, S3, DynamoDB, SNS, Lambda, or OpenSearch.

flowchart LR
  subgraph Local["Local Compose"]
    BrowserLocal["Host/browser"]
    ComposePorts["Host port mappings"]
    RustBundle["macro-local-rust-services:dev"]
    LocalDB["Postgres + Redis + OpenSearch"]
    Fusion["FusionAuth"]
    LocalS3["LocalStack-compatible AWS endpoint"]
  end

  subgraph Cloud["Pulumi AWS stacks"]
    Route53["Route53 + HTTPS ALB"]
    ECS["ECS/Fargate Rust services"]
    Workers["ECS workers + Lambda handlers"]
    Queues["SQS queues + DLQs"]
    Storage["S3 buckets + DynamoDB"]
    Data["RDS MacroDB + Redis + OpenSearch"]
  end

  BrowserLocal --> ComposePorts --> RustBundle
  RustBundle --> LocalDB
  RustBundle --> Fusion
  RustBundle --> LocalS3

  Route53 --> ECS
  ECS --> Data
  ECS --> Storage
  ECS --> Queues
  Queues --> Workers
  Storage --> Workers

Core cloud stacks

Service URLs and stack routing

Cloud service URL values are centralized in ServiceUrl and resolved by stack (dev or prod). Pulumi stacks inject these values as environment variables instead of hard-coding peer endpoints inside Rust binaries.

Storage dependencies

Queues, workers, and event flows

Document upload and processing path

sequenceDiagram
  participant Client
  participant DSS as document_storage_service
  participant S3 as document-storage S3 bucket
  participant EB as EventBridge/SQS
  participant Finalizer as document_upload_finalizer
  participant Extractor as document_text_extractor
  participant Search as search_processing_service
  participant OS as OpenSearch

  Client->>DSS: Request document upload/update
  DSS->>S3: Write object or issue presigned upload path
  S3-->>EB: Object Created
  EB-->>Finalizer: Finalize versioned object
  EB-->>Extractor: Extract document text
  DSS-->>EB: Publish search event when applicable
  EB-->>Search: Search event queue message
  Search->>S3: Read source content
  Search->>OS: Index searchable representation

Local development replaces the EventBridge Lambda path with the document_upload_finalizer_local_worker polling a LocalStack SQS queue. Cloud uses EventBridge rules and Lambda targets with DLQs.

AI tool hosting topology

getAiToolsInfra() defines the shared infrastructure contract for services that host the Rust ai_tools crate. It returns:

getAiToolsServiceRoleArns() centralizes the IAM role ARNs for tool-hosting services such as MCP server, document cognition, and agent schedule service. Resource-side policies can grant access to that set without coupling the topology to one model provider.

Health checks and verification signals

Local Compose health checks use the container port, usually 8080, even when the host port differs. Cloud ALB target groups use the configured healthCheckPath and forward to the ECS task container port.

Operations notes

Local startup constraints

• Create the external Docker networks and volumes expected by Compose before starting the full stack.
• Use the processors profile only when the local machine can run the amd64 search processing image or QEMU emulation.
• Use COMPOSE_FILE=docker-compose.yml:docker-compose.local-e2e.yml for deterministic local E2E environment overrides.
• Ensure a LocalStack-compatible endpoint exists when using S3/SQS-backed local flows.

Cloud deployment constraints

• New service environment variables must be added to the matching Pulumi stack so ECS tasks or Lambda handlers receive them.
• Services that call AWS APIs need both runtime environment values and IAM policy coverage.
• Queue-backed workflows should define DLQs and alarms; the shared Queue component creates a queue, DLQ, and DLQ alarm by default.
• Rust services use SQLX_OFFLINE=true during Docker builds; SQLx query metadata must be prepared from the Rust workspace when database queries change.

Related pages

09. Entities and blocks

Concept: Item types, document-backed and non-document blocks, aliases, load sources, nesting rules, and file type resolution.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/09-entities-and-blocks.md
• Generated: 2026-06-01T00:51:49.790Z

Source Files

• js/app/packages/core/block.ts
• js/app/packages/core/constant/allBlocks.ts
• js/app/packages/block-md/definition.ts
• js/app/packages/block-project/definition.ts
• js/app/packages/service-clients/service-storage/client.ts
• rust/cloud-storage/models_soup/src/lib.rs

Macro maps storage-service items and repository-defined block definitions into Solid block instances. The frontend resolves an entity-like item to a BlockName or BlockAlias, creates a Source, runs the selected block definition’s load(source, intent), and exposes the loaded data through block-scoped signals.

Core identifiers

Block definitions

Block definitions are collected with import.meta.glob('../../block-*/definition.ts'). Each definition provides:

defineBlock({
  name,
  description,
  component,
  accepted,
  load,
  defaultFilename?,
  aliases?,
  liveTrackingEnabled?,
  syncServiceEnabled?,
  editPermissionEnabled?,
})

Document-backed and non-document blocks

NonDocumentBlockTypes marks blocks that do not correspond to document file content:

[
  'call',
  'chat',
  'channel',
  'project',
  'email',
  'contact',
  'automation',
]

Everything else normally behaves as document-backed content. Aliases are not listed as non-document blocks: task resolves to md, and csv resolves to code, so both remain document-backed.

blockNameToItemType() converts blocks back to item types for history and storage operations:

Aliases

Aliases are declared inside block definitions, not in a separate routing table.

Alias handling has two layers:

1. fileTypeToBlockName() may return the alias when the input is an alias or when an item subType.type is an alias.
2. resolveBlockAlias() flattens aliases to the concrete block implementation before mounting or calling block methods.

Split-layout URLs preserve aliases with aliasContext, so a route can encode /task/:id while the mounted implementation is still the md block. Duplicate split checks compare resolved block names, preventing the same item from opening twice as both /md/:id and /task/:id.

File type resolution

allBlocks.ts builds lookup tables from every block definition’s accepted map:

Resolution order for fileTypeToBlockName(input, icon?):

1. Missing input returns unknown.
2. channel_message maps to channel.
3. With ENABLE_DOCX_TO_PDF, docx and write map to pdf for behavior, but icon: true returns write.
4. Alias names return the alias.
5. Registered block names return themselves.
6. Accepted file extensions map to their owning block.
7. Unrecognized input returns unknown.

itemToBlockName(item, icon?) gives document subtypes precedence over file types. If item.subType.type is a known alias, it returns that alias; otherwise it resolves item.fileType, then falls back to item.type.

Load sources and lifecycle

The orchestrator’s default source resolver returns sync-service when a block definition has syncServiceEnabled; otherwise it returns a dss source with the id and optional upload metadata from router state.

item or route
  -> block/file-type resolution
  -> createBlockInstance(type, id)
  -> Source: sync-service or dss
  -> definition.load(source, 'initial')
  -> BlockLoader signals

BlockLoader rejects nested preload results during initial load, records load errors as UNAUTHORIZED, MISSING, GONE, or INVALID, and publishes common fields when present:

Markdown is the main sync-service-backed block. It requires the storage location to be syncServiceContent; object-storage markdown content is treated as invalid because markdown initialization and persistence are backend-owned.

Backend entity shape

The Rust SoupItem enum is the backend list/search aggregate. Each variant can produce a canonical Entity with an entity type and string id. Documents with SoupDocumentSubType::Task report property entity type Task; ordinary documents report Document.

Property references are only available for documents, projects, email threads, and chats. Channels, calls, CRM companies, and foreign entities intentionally return no property reference in to_entity_reference().

Nesting rules

Nested blocks are gated by ValidNestingCombinations and checked through canNestBlock(name, parentName). The rule is keyed by child block and stores allowed parent blocks.

Alias-aware callers resolve aliases before checking nesting. For example, task resolves to md, and csv resolves to code; a CSV/code preview can be nested in markdown if the caller resolves it to code.

Creation and upload paths

Document creation and upload use storage-service client methods:

Upload file type inference first uses an explicit options.fileType, then MIME-to-extension lookup from block definitions, then the filename extension when the MIME type is empty. The backend owns object-created finalization and sync-service initialization after upload.

Related pages

10. Split layout and navigation

Concept: Route encoding, component registry entries, split history, navigation causes, popovers, and desktop versus mobile split behavior.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/10-split-layout-and-navigation.md
• Generated: 2026-06-01T00:52:49.230Z

Source Files

• js/app/packages/app/component/Root.tsx
• js/app/packages/app/component/split-layout/SplitLayoutRoute.tsx
• js/app/packages/app/component/split-layout/componentRegistry.tsx
• js/app/packages/app/component/split-layout/layoutManager.ts
• js/app/packages/app/component/split-layout/layout.ts
• js/app/packages/app/component/split-layout/tests/layoutManager.test.ts

The app renders workspace content through a Solid Router catch-all split route whose path segments encode the visible split contents as type/id pairs, while createSplitLayout owns mounted content, per-split history, focus events, popovers, URL synchronization, and desktop/mobile layout decisions.

Route encoding

A split URL is parsed as alternating path segments:

/component/inbox/md/doc_123
└─ type ─┘└ id ┘└type┘└ id  ┘
   split 1       split 2

decodePairs() consumes pairs until a missing type or id is found. Non-component types are resolved through the block alias registry; aliases keep aliasContext so the URL can re-encode the alias instead of only the resolved base block type.

Route mounting and URL sync

LAYOUT_ROUTE uses path: '/*splits' and passes props.params.splits?.split('/') ?? [] into SplitLayoutContainer. The root route table also maps top-level app paths such as /inbox, /agents, /mail, /documents, /tasks, /channels, /calls, and /files to the same layout component, while DEFAULT_ROUTE is /component/inbox.

SplitLayoutContainer performs two-way synchronization:

Split content model

The layout manager treats all mounted content as SplitContent:

type SplitContent =
  | {
      type: BlockName | BlockAlias;
      id: string;
      params?: BlockComponentProps[BlockName];
      aliasContext?: BlockAliasContext;
      state?: EntryState;
    }
  | {
      type: 'component';
      id: string;
      params?: Record<string, unknown>;
      state?: EntryState;
    };

Block content is mounted through the global block orchestrator. Component content is resolved through the split component registry.

Component registry entries

Component splits use { type: 'component', id }. The id must be registered in componentRegistry.tsx; otherwise resolveComponent() throws Component '<name>' not registered.

Registered app-level component ids include:

Opening and replacing splits

Most callers use useSplitLayout() rather than the manager directly.

openWithSplit() follows this order:

1. A registered navigation interceptor may consume the navigation. Mobile swipe layout uses this.
2. Unless allowDuplicate is true, an existing visible split with the same type and id is activated and returned.
3. If no explicit handle is provided, the active split becomes the target handle.
4. The target handle is replaced when preferNewSplit is false or the resize zone cannot fit another 400px panel.
5. Otherwise a new split is appended.

replaceWhenFull defaults to enabled. Setting replaceWhenFull: false allows callers to request a new split even when canAppendSplit() reports insufficient space, but desktop rendering still depends on the resize zone.

Split history and navigation causes

Each split owns an independent History<SplitContent> with push, merge, back, forward, replaceCurrent, goToIndex, and remove.

Components can read the latest cause with useNavigationCause(). This is intended for behavior that differs between fresh navigation and restoration, such as suppressing auto-focus after back/forward.

useEntryState(key, { default }) stores component-owned state on the current history entry. Registered captors run before navigation away, then the captured state is mirrored back onto split.content.state.

Referral metadata

referredFrom records where navigation originated. It is stored on SplitState and exposed through SplitHandle.referredFrom().

Allowed values are:

'list-view' | 'kommand-menu' | 'mention' | 'attachment' | 'launcher' |
'sidebar' | 'dock' | 'entity-actions-menu' | 'hotkey' | 'quick-access' |
'file-upload' | 'search' | null

Use the closest existing value when adding a navigation caller. If the source is not meaningful to downstream behavior or analytics, pass null.

Popover splits

Popover splits are temporary mounts managed by createPopoverSplit() and rendered by PopoverSplitRenderer.

A popover split:

• creates a popover-... id;
• acquires a focus lock before state updates;
• mounts the same block/component content model as normal splits;
• renders inside a Dialog and Panel;
• provides a stub SplitHandle with isPopover() === true;
• has no URL segments, no history navigation, and lastNavigationCause() === 'fresh';
• closes on escape or dialog close;
• releases the focus lock and removes its map entry after a short animation delay.

Popover content still receives SplitPanelContext, so components that depend on panel context can render inside the dialog. Do not expect URL persistence, back/forward history, or split resizing inside a popover.

Desktop layout behavior

Desktop split rendering uses a horizontal Resize.Zone with Resize.Panel children. Each panel has minSize={400} and renders a SplitPanel containing:

• a split-specific hotkey DOM scope;
• SoupContextProvider;
• split header and toolbar slots;
• the mounted block or registered component element;
• spotlight support;
• focus restoration and active split tracking.

Focus changes inside a panel activate that panel after a short debounce. Insert and remove events explicitly move focus to the inserted split or the nearest remaining split.

Relevant split hotkeys include:

Mobile layout behavior

There are two mobile-related checks:

The native mobile layout uses slot A and slot B. One slot is foreground and the other may hold a background split for swipe-back. The background split is excluded from URL encoding, duplicate detection, and content lookup.

Mobile navigation behavior differs from desktop:

• createMobileSwipeLayout() registers a navigation interceptor.
• Non-mergeHistory navigation is handled as forward navigation into the background slot, then promoted to foreground after animation.
• Swipe-back promotes the background slot, removes the old foreground split, and lazily mounts the promoted split’s previous history entry as the next background split.
• mergeHistory navigation bypasses the interceptor and uses normal replace behavior.
• MobileDock uses mergeHistory when switching between component/list views so dock tab switches do not create swipe-back entries.

Reconciliation contracts

reconcile(newSplits) is used for URL-driven changes. It compares the visible split key sequence with the decoded URL sequence, preserves excluded splits unchanged, and rebuilds changed visible positions. When a visible split exists at the same index, the new split reuses that split id to keep panel identity stable at that position.

Run the layout manager tests when changing reconciliation, history, or URL behavior. The test suite covers URL-state reconciliation, block-to-component replacement, and browser-back-like ordering scenarios.

Troubleshooting

11. Real-time document sync

Concept: Sync-service worker routes, Durable Object sessions, permission tokens, Bebop messages, snapshot lifecycle, and client reconnect behavior.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/11-real-time-document-sync.md
• Generated: 2026-06-01T00:52:59.543Z

Source Files

• rust/sync-service/src/cf_worker.rs
• rust/sync-service/src/durable_object.rs
• rust/sync-service/src/websocket.rs
• rust/sync-service/src/generated/schema.rs
• js/app/packages/service-clients/service-sync/client.ts
• js/app/packages/service-clients/service-sync/source.ts
• js/app/packages/block-md/definition.ts

The sync service is a Cloudflare Worker backed by one DocumentSyncSession Durable Object per document_id; it exposes HTTP document endpoints, upgrades /document/{document_id}/connect to a WebSocket session, serializes realtime messages with Bebop, and stores Loro snapshots plus pending operations for reconnect and recovery.

Runtime shape

flowchart LR
  subgraph Client
    MD["block-md load()"]
    Source["createSyncServiceSource()"]
    Rest["syncServiceClient"]
  end

  subgraph Worker["sync-service Worker"]
    Router["cf_worker::router"]
    Schema["/schema"]
    Copy["/document/{id}/copy"]
  end

  subgraph DO["DocumentSyncSession Durable Object"]
    WS["/connect WebSocket"]
    API["metadata/raw/snapshot/active_peers/initialize"]
    State["DocumentState(LoroDoc)"]
    Awareness["EphemeralStore"]
    Alarm["alarm()"]
  end

  subgraph Storage
    SQLKV["Durable Object SQL snapshot store"]
    KVFallback["SNAPSHOT_STORE_KV fallback"]
    DOKV["Durable Object KV pending ops"]
    D1["USER_PEER_MAPPING D1"]
  end

  MD --> Source
  Source --> WS
  Rest --> Router
  Router --> Schema
  Router --> Copy
  Router --> DO
  WS --> State
  WS --> Awareness
  API --> State
  State --> SQLKV
  SQLKV --> KVFallback
  DOKV --> State
  WS --> D1
  Alarm --> SQLKV
  Alarm --> DOKV

Worker routes

The top-level Worker routes health/schema traffic directly and forwards document traffic to the Durable Object namespace DOCUMENT_SYNC_SESSION using id_from_name(document_id).

pass_to_durable_object wraps Durable Object fetches with the service default timeout and returns status 408 if the RPC times out.

Durable Object routes

DocumentSyncSession owns the document session state. CORS validation runs before route handling. Allowed origins include local app ports, Macro production/dev/staging origins, Capacitor localhost, Apollo testing, and non-empty https://{subdomain}.preview.macro.com preview origins.

Permission tokens

The service accepts two token sources:

JWTs are HS256 tokens signed with DOCUMENT_PERMISSIONS_SECRET and contain:

type AuthToken = {
  user_id?: string;
  document_id: string;
  access_level: 'view' | 'comment' | 'edit' | 'owner' | 'admin';
};

Document-scoped requests must match document_id unless the token is internal admin. initialize requires edit or stronger. Debug routes require admin.

For WebSocket document updates, PeerUpdate is ignored when Wsm::can_edit() is false. The current runtime write gate treats comment, edit, owner, and admin as editable because AccessLevel::can_edit() checks for AccessLevel::Comment or stronger; view users cannot push document updates. Awareness updates are accepted for connected users independently of edit access.

WebSocket session lifecycle

1. connect_handler decodes the query token.
2. The Durable Object stores the document_id if not already set.
3. A Cloudflare WebSocketPair is created.
4. The server socket is accepted with a generated WebSocket tag.
5. WebSocketMetadata is stored in Durable Object storage and memory:
   • user_id
   • access_level
   • peer_ids
6. The current DocumentState is loaded or reused.
7. The server sends RemoteInitialSync with:
   • a shallow Loro snapshot
   • encoded awareness state
8. The client receives the paired WebSocket response.

String "ping" messages receive "pong". Binary messages are decoded as FromPeer Bebop messages.

Bebop message protocol

The schema lives in rust/sync-service/bebop/schema.bop and is generated into Rust and TypeScript clients.

Client to service: FromPeer

Service to client: FromRemote

Messages larger than 1 MB log a warning before deserialization; they are not rejected solely because of that size check.

Snapshot and operation lifecycle

DocumentState wraps a Loro document. It imports snapshots with the from_service tag, imports client updates with the from_client tag, and tracks whether a snapshot should be saved by comparing last_update and last_export.

Storage layers

The default Cargo feature set enables do-sqlite-snapshot-storage, so snapshots are written to Durable Object SQL. The combined storage backend reads SQL first and falls back to KV when SQL has no snapshot.

Save loop

After each binary WebSocket message, the Durable Object schedules an alarm roughly 5 seconds in the future if no later alarm is already scheduled.

When the alarm fires:

1. The Durable Object loads DocumentState.
2. If state.should_save() is true:
   • exports a full Loro snapshot
   • stores the snapshot
   • stores the Loro operation version vector
   • clears applied pending operation keys
   • marks the state exported
3. If sockets are still connected:
   • schedules the next alarm
   • broadcasts RemoteSnapshot with a shallow snapshot
4. If no sockets remain, it logs that the Durable Object reached zero connections.

wakeup calls warm the session storage and document state when the document exists, then use a JavaScript timeout keepalive with the default 60 second TTL. The TypeScript client’s safeWakeup debounces wakeups by 200 ms and suppresses repeat wakeups for 55 seconds per document.

Initialization and copy

initialize accepts a Bebop InitializeFromSnapshotRequest:

type InitializeFromSnapshotRequest = {
  snapshot: Uint8Array;
};

It fails if snapshot storage already contains a snapshot for the document. Otherwise it stores the snapshot, records the DOCUMENT_ID in Durable Object storage, and prepares SessionStorage.

copy is handled at the Worker layer because it touches two Durable Object instances:

1. POST /document/{source_id}/snapshot with optional version_id.
2. Wrap the returned binary snapshot in InitializeFromSnapshotRequest.
3. POST /document/{target_id}/initialize.

The JSON copy request is:

{
  "target_document_id": "new-document-id",
  "version_id": {
    "peer": "123",
    "counter": 10
  }
}

version_id is optional. When present, the snapshot handler exports state at the requested Loro frontier.

TypeScript client behavior

syncServiceClient exposes HTTP helpers against SYNC_SERVICE_HOSTS.worker:

Tauri requests add an explicit Origin header of https://dev.macro.com in development and https://macro.com otherwise.

createSyncServiceSource(documentId, token) builds the WebSocket URL:

{SYNC_SERVICE_HOSTS.ws}/document/{documentId}/connect?token={token}

Connection settings:

The first connection uses the provided token. Reconnects request a fresh permission token from storageServiceClient.permissionsTokens.createPermissionToken({ document_id }); if token refresh fails, the client falls back to the last known WebSocket URL.

Heartbeat is intentionally started only after the first RemoteInitialSync arrives. On reconnect, the client starts heartbeat, waits for another RemoteInitialSync, and emits a reconnect sync event containing the new snapshot and awareness. If reconnect sync times out, it logs the failure and lets heartbeat monitoring close/retry the socket.

Markdown block integration

The Markdown block only loads realtime collaboration when the source is sync-service.

Load flow:

1. Fetch document metadata, document location, and permission token in parallel.
2. If the location is a pending presigned URL, wait for syncServiceContent readiness.
3. Reject the load when the final location is not syncServiceContent.
4. Create the sync-service source with the permission token.
5. Create a Loro manager using MARKDOWN_LORO_SCHEMA.
6. Initialize local Loro state from initialSync.snapshot.
7. Return the block data with syncSource, loroManager, metadata, and user access level.

Markdown initialization and lifecycle persistence are backend-owned; the frontend does not repair an object-storage-backed Markdown document into sync-service content during block load.

Operational configuration

Cloudflare bindings in wrangler.toml:

Useful local commands:

cd rust/sync-service

# Build the Worker used by Miniflare tests
just worker-build

# Run e2e tests and Rust unit tests
just test

# Apply local D1 migrations and start Wrangler
just dev

# Regenerate TypeScript Bebop bindings used by tests
cd bebop && npx bebopc build

Error and verification signals

Related pages

12. Run backend services locally

Guide: Start databases, LocalStack, FusionAuth, Rust services, optional processor profiles, and local health checks.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/12-run-backend-services-locally.md
• Generated: 2026-06-01T00:53:22.606Z

Source Files

• justfile
• docker-compose.yml
• docker-compose-databases.yml
• docker-compose.local-e2e.yml
• local_stack.just
• rust/cloud-storage/justfile

The local backend runtime is owned by just recipes and Docker Compose files at the repository root. just setup prepares .env, external Docker networks and volumes, LocalStack AWS resources, the local Postgres database, FusionAuth configuration, and prebuilt Rust service images; just run_local then starts the Compose stack.

Prerequisites

Install the tools used directly by the local recipes:

If not using the repository shell environment, export SOPS_KMS_ARN before decrypting local environment files.

export SOPS_KMS_ARN="arn:aws:kms:us-east-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93,arn:aws:kms:us-west-1:569036502058:key/mrk-cab29bf948044eb79005a81f48d40e93"

One-command setup

just setup

just setup performs this sequence:

1. Decrypts the root local environment into .env.
2. Creates external Docker networks: databases and auth.
3. Creates persistent local volumes for Postgres, Redis, OpenSearch, and FusionAuth.
4. Starts and provisions LocalStack.
5. Creates and migrates the local Macro Postgres database.
6. Starts FusionAuth, deploys the local Pulumi stack, patches root .env, then stops FusionAuth.
7. Builds the development Rust service images.

Start only infrastructure

Use these commands when you want to rebuild or inspect dependencies before starting application services.

This targets only `postgres` and `redis` from `docker-compose-databases.yml`.

The local database URL is `postgres://user:password@localhost:5432/macrodb`.

Local infrastructure endpoints

LocalStack resources

just setup_localstack starts LocalStack with SERVICES=sqs,dynamodb,s3, waits for /_localstack/health, then creates queues, tables, buckets, and a document-upload S3 notification.

S3 buckets

All buckets receive a local CORS configuration allowing http://localhost:3000 through http://localhost:3009.

DynamoDB tables

SQS queues

Local setup creates queues for notifications, email jobs, contacts, conversion, document deletion, document text extraction, search events, static-file events, and document-upload finalization. The document-upload finalizer wires S3 ObjectCreated events from doc-storage to document-upload-finalizer-queue.

Inside Docker, services use http://localstack:4566. Host-side commands and generated browser-facing LocalStack URLs use http://localhost:4566.

FusionAuth local setup

FusionAuth local setup is split between Docker Compose and a Pulumi stack under infra/stacks/fusionauth-instance.

just setup_fusionauth

The setup recipe:

1. Downloads the FusionAuth container .env if missing.
2. Starts the FusionAuth Postgres database and app.
3. Waits for http://localhost:9011/api/status to report {"status":"Ok"}.
4. Initializes or updates the Pulumi local stack.
5. Writes local FusionAuth values into the root .env.
6. Stops FusionAuth after setup.

Useful local credentials:

just run_local calls patch_local_fusionauth_env before starting services. If FusionAuth is not already running, the patch recipe starts it temporarily, reads the client secret, updates .env, and stops the temporary container afterward.

Start backend services

just run_local

By default, this runs docker compose up in the foreground. Pass Docker Compose arguments through just when you want detached startup, waiting, or a subset of services.

just run_local -d --wait

For a clean rebuild after changing service code:

just run_local --build

run_local always builds the shared Rust services image target rust_services_image. With --build, it also rebuilds websocket_service, sync_service, and lexical_service.

Service ports and health endpoints

Check the stack:

docker compose ps
curl -fsS http://localhost:8080/health
curl -fsS http://localhost:8086/health
curl -fsS http://localhost:8094/api/health
curl -fsS http://localhost:8787/health
curl -fsS http://localhost:9011/api/status
curl -fsS http://localhost:4566/_localstack/health

Optional processor profile

search_processing_service is behind the Compose profile processors.

just run_local --profile processors

Rebuild it explicitly when changing processor code:

just run_local --build --profile processors

The processor image uses Dockerfile.search_processing_service.dev and is pinned to linux/amd64 because the bundled search_processing_service/pdfium-lib/linux/libpdfium.so is amd64-only. Apple Silicon hosts run this service through emulation.

Local E2E backend profile

The local E2E commands start LocalStack, start a narrowed backend service set with docker-compose.local-e2e.yml overrides, seed deterministic data, then run tests.

just local-e2e

Rust ignored integration tests use the same seeded stack:

just local-e2e-rust

Run Rust integration tests and Playwright after one stack startup and seed pass:

just local-e2e-all

Open Playwright UI mode:

just local-e2e-ui

The E2E override file forces services to use local dependencies instead of shared development infrastructure:

The deterministic seed command is guarded. It requires LOCAL_E2E_SEED=true and refuses database URLs outside the local Docker database shape postgres://user:...@(localhost|127.0.0.1|postgres):5432/macrodb.

Stop and reset

Troubleshooting

.env not found

Run:

just get_environment

For a fresh checkout, prefer:

just setup

FusionAuth local stack is missing

If run_local prints a warning that the Pulumi local stack was not found, run:

just setup_fusionauth

Then restart the backend stack.

LocalStack resources are missing

Recreate LocalStack resources:

AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1 just setup_localstack

The recipe uses localstack/localstack:4; it is intentionally not latest.

Processor builds are slow on Apple Silicon

search_processing_service runs as linux/amd64 because of the bundled PDFium library. Use the default stack unless you need the processors profile.

Local E2E seed refuses to run

Check that the seed command has LOCAL_E2E_SEED=true and that DATABASE_URL points at the local Compose database, not a shared development database.

Related pages

13. Develop SolidJS and Tauri apps

Guide: Use Bun and Vite, run local service overrides, build app bundles, start Tauri targets, and avoid iOS worker deadlocks.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/13-develop-solidjs-and-tauri-apps.md
• Generated: 2026-06-01T00:54:19.836Z

Source Files

• js/app/README.md
• js/app/package.json
• js/app/justfile
• js/app/packages/app/index.tsx
• js/app/packages/app/vite.config.ts
• js/app/tauri/src-tauri/README.md
• js/app/AGENTS.md

The Macro frontend is a SolidJS single-page app in js/app/packages/app built by Vite and run with Bun; the same Vite output is used for web, desktop, iOS, and Android through the Tauri wrapper in js/app/tauri.

Project layout

:::files js/app/ ├─ package.json # Bun scripts for dev, build, tests, codegen ├─ justfile # frontend build and local-service shortcuts ├─ packages/app/ │ ├─ index.tsx # Solid entry point and Tauri fetch override │ ├─ vite.config.ts # app Vite config entry │ └─ vite.base.ts # shared Vite server/build/env configuration ├─ packages/core/constant/ │ └─ servers.ts # remote/local service host selection ├─ packages/core/util/ │ ├─ platform.ts # web/desktop/iOS/Android runtime detection │ └─ platformFetch.ts # Tauri HTTP plugin fetch adapter └─ tauri/src-tauri/ ├─ tauri.conf.json # Tauri dev/build hooks and bundle settings └─ README.md # native target commands :::

Prerequisites

Use nix develop from the repository root when available. Otherwise install:

• Bun
• Node tooling used by Bun scripts
• Rust and Cargo
• Tauri CLI
• Xcode for iOS targets
• Android Studio and Android SDK tooling for Android targets

For local backend services, complete the repository-level local setup first:

just setup

The local stack uses the fixed Docker Compose project name macro; do not run two local stacks at the same time.

Run the web app with Vite

Run commands from js/app.

bun i
bun run dev

bun run dev changes into packages/app and starts:

bun run --bun dev

The package-level app script runs:

vite -c vite.config.ts

Vite serves on 0.0.0.0:${PORT:-3000} with strictPort: true, WebSocket HMR, CORS enabled, polling file watch, and filesystem access to the workspace root.

Vite build behavior

The shared config sets different paths for serve and build:

Build modes are selected with MODE:

just build-dev
just build-staging
just build-prod

Equivalent direct command shape:

cd js/app/packages/app
MODE=production NODE_ENV=production bun run --bun build

The app build also writes dist/semver.txt as:

<package-version>+<git-short-sha>

Use NO_MINIFY=true when a readable bundle is needed for debugging. The Vite config switches output names to stable assets/[name].js and disables minification.

Runtime environment values

vite.base.ts injects compile-time values through define.

Run against local service overrides

In development mode, service hosts come from packages/core/constant/servers.ts.

If VITE_LOCAL_SERVERS is unset or empty, the app uses remote development hosts. If it is ALL, all configured services use localhost. Otherwise, provide a comma-separated list of service names to override individually.

# All local services
just local

# Cognition HTTP + websocket services
just local-dcs

# Document storage only
just local-dss

# Search only
just local-search

# Email only
just local-email

Direct examples:

cd js/app/packages/app

# Use all local hosts
VITE_LOCAL_SERVERS=ALL bun run --bun dev

# Override one service
VITE_LOCAL_SERVERS=document-storage-service bun run --bun dev

# Override multiple services
VITE_LOCAL_SERVERS=document-storage-service,email-service bun run --bun dev

# Override a local port for one service
VITE_LOCAL_SERVERS=document-storage-service:18086 bun run --bun dev

Available service keys include:

Run the local smoke suite

From the repository root:

just local-e2e

The harness starts the local stack with docker-compose.local-e2e.yml, seeds deterministic smoke data, launches the frontend with:

LOCAL_E2E=true
VITE_LOCAL_SERVERS=ALL
VITE_ENABLE_BEARER_TOKEN_AUTH=true
LOCAL_JWT=<generated-or-exported-token>

Then it runs Playwright from js/app.

Open Playwright UI mode with:

just local-e2e-ui

Run the frontend Playwright command directly only after the local stack and seed data are ready:

cd js/app
LOCAL_E2E=true bunx playwright test

Start Tauri targets

Run native commands from the Tauri workspace under js/app/tauri.

cd js/app/tauri
cargo tauri dev

cd js/app/tauri
cargo tauri ios dev

cd js/app/tauri
cargo tauri android dev

tauri.conf.json defines:

just dev-tauri starts the same Vite app. just build-tauri builds the production frontend bundle before Tauri packages native artifacts.

For devices or emulators that cannot resolve localhost HMR, set:

export TAURI_DEV_HOST=<host-reachable-from-device>
cargo tauri ios dev

Vite uses TAURI_DEV_HOST as the HMR host while still serving on port 3000.

Build native bundles

cd js/app/tauri
cargo tauri build

cd js/app/tauri
cargo tauri ios build

cd js/app/tauri
cargo tauri android build

Tauri packages the static files emitted into js/app/packages/app/dist. The configured bundle target is all, with iOS minimum system version 14.0.

For updater-server testing, create the frontend archive from js/app:

just dist-archive

That recipe builds production assets, zips packages/app/dist, and writes:

rust/cloud-storage/tools/native_app_server/app-archive.zip

Platform-specific runtime behavior

The app detects platform at runtime instead of producing separate web/mobile bundles.

The fetch override intentionally skips localhost URLs so Vite HMR and dev-server requests continue to use normal browser fetch during Tauri development.

iOS worker deadlock rule

The problematic pattern is eager construction:

import WorkerImpl from './worker?worker';

const worker = new WorkerImpl(); // avoid at module load on iOS

Use lazy construction instead. Importing a worker constructor is safe; calling new WorkerImpl() must happen on first use.

import HeicWorker from './heic-worker?worker';

class Service {
  private static instance: Service | null = null;

  private constructor() {
    this.workerPool = WorkerPool.getInstance(HeicWorker);
  }

  static getInstance(): Service {
    if (!Service.instance) Service.instance = new Service();
    return Service.instance;
  }
}

export const service = new Proxy({} as Service, {
  get: (_target, prop, receiver) =>
    Reflect.get(Service.getInstance(), prop, receiver),
});

Safe examples in this codebase instantiate workers inside runtime paths such as a block load() function or a service singleton that is reached only when a conversion/upload operation starts.

Symptom signature

An iOS worker deadlock usually looks like:

• HTML loads.
• Initial JavaScript starts.
• JavaScript silently stops.
• Safari Web Inspector attaches but shows no useful app progress.
• The WebContent process stays alive with 0.0% CPU.

Diagnose an iOS worker freeze

cd js/app/tauri
cargo tauri ios dev "iPhone 15"

/usr/bin/log stream --predicate 'process == "macro"' --info --debug --style compact

tail -200 <logfile> | grep -vE 'tauri:// request|tauri_protocol.rs|^\s*\\134'

ps -o pid,pcpu,comm -p <webcontent_pid>

sample <webcontent_pid> 3 -file /tmp/sample.txt

WorkerOrWorkletScriptController::loadModuleSynchronously
WorkerDedicatedRunLoop::runInMode
Condition::waitUntilUnchecked

Quality checks

Run these from js/app before handing off frontend changes:

bun run check
bun run lint
bun run test

Use the AGENTS guidance for SolidJS changes:

• Avoid createEffect unless syncing with an external imperative system.
• Use derived signals for derived state.
• Use createMemo only for referential stability or expensive derivations.
• Check solid-primitives before adding custom reactive utilities.
• Keep reusable components small and decoupled from query/mutation state.
• Use semantic color tokens instead of raw Tailwind color classes.

Related pages

14. Change a Rust service API

Guide: Update Axum handlers, OpenAPI emitters, generated client specs, Orval output, and CI checks for service API changes.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/14-change-a-rust-service-api.md
• Generated: 2026-06-01T00:55:38.799Z

Source Files

• rust/cloud-storage/document_storage_service/src/openapi.rs
• rust/cloud-storage/document_cognition_service/src/openapi.rs
• js/app/scripts/generate-api-schema.ts
• js/app/scripts/services.ts
• js/app/packages/service-clients/orval.config.ts
• .github/workflows/web-app-check-main.yml
• rust/cloud-storage/AGENTS.md

Rust service API changes flow from Axum handlers and utoipa annotations into local *_openapi Rust binaries, then through js/app/scripts/generate-api-schema.ts into committed openapi.json files and Orval-generated TypeScript under js/app/packages/service-clients.

API generation path

Rust handler + request/response models
  └─ #[utoipa::path(...)] + ToSchema
      └─ api/swagger.rs ApiDoc paths(...) and components.schemas(...)
          └─ src/openapi.rs binary prints ApiDoc::openapi().to_pretty_json()
              └─ js/app/scripts/generate-api-schema.ts writes openapi.json
                  └─ orval.config.ts regenerates generated client/schema files
                      └─ biome formats service-clients
                          └─ --check mode fails if git diff or untracked generated files remain

The live services also mount Swagger UI at /docs and serve the same OpenAPI document at /api-doc/openapi.json.

Service registry

The generator accepts service names from js/app/scripts/services.ts and maps them to Rust crate names in js/app/scripts/generate-api-schema.ts.

Change an existing endpoint

Example surfaces:
- Storage handlers live under `rust/cloud-storage/document_storage_service/src/api/**`.
- Cognition handlers live under `rust/cloud-storage/document_cognition_service/src/api/**`.
- Shared request/response models may live under service-local `model/**` modules or shared crates such as `model`, `models_*`, `chat`, or `memory`.

If a route is mounted under both `/{version}` and the unversioned router, document the unversioned path in `#[utoipa::path]`, matching the existing service pattern.

Storage uses `rust/cloud-storage/document_storage_service/src/api/swagger.rs`.

Cognition uses `rust/cloud-storage/document_cognition_service/src/api/swagger.rs`.

Missing `components.schemas` entries commonly produce incomplete generated TypeScript even when the Rust code compiles.

```bash
cd js/app
bun run gen-api -- cloud-storage
bun run gen-api -- document-cognition
```

To regenerate every registered service:

```bash
cd js/app
bun run gen-api
```

The script builds OpenAPI binaries with `SQLX_OFFLINE=true cargo build`, runs each binary, writes `openapi.json`, removes the previous `generated/` directory, runs Orval, and then runs Biome on `packages/service-clients/`.

Document Cognition special generation

document-cognition has additional generated artifacts beyond Orval output.

When the target service includes document-cognition, generate-api-schema.ts also builds or runs:

• document_cognition_service_models
• gen_tool_schemas

It then runs scripts/generate-dcs-types.ts, which generates:

• service-cognition/generated/schemas/model.ts
• service-cognition/generated/tools/schemas.ts
• service-cognition/generated/tools/types.ts
• service-cognition/generated/tools/tool.ts

Add a new service to generation

For a new Rust service client, add all registry points together:

1. Add a Rust OpenAPI binary, usually named <crate>_openapi, that prints ApiDoc::openapi().to_pretty_json().
2. Add a [[bin]] entry in the service Cargo.toml.
3. Add the service entry in js/app/scripts/services.ts with name, dev, prod, local, output, and orvalKey.
4. Add the service-to-crate mapping in js/app/scripts/generate-api-schema.ts.
5. Add a matching Orval project in js/app/packages/service-clients/orval.config.ts.
6. Run targeted generation and commit openapi.json plus generated/.

Orval output modes

js/app/packages/service-clients/orval.config.ts controls the generated TypeScript shape.

Storage, properties, and notification generate Zod validators. Most other services generate fetch clients and TypeScript schemas. Search writes schemas under service-search/generated/models instead of generated/schemas.

Verification

Run the checks that match the change size.

cd js/app
bun run gen-api -- <service-name>
bun run gen-api -- --check
bun run --bun --silent tsc --project ./packages/app/tsconfig.json

For Rust changes:

cd rust/cloud-storage
cargo test -p <crate-name>
just check
just clippy
just format

If the API change includes SQLx query or schema changes, update the SQLx offline cache from the workspace root:

cd rust/cloud-storage
just prepare_db

CI behavior

The web app PR workflow has two path filters:

• should_run for frontend package, app, lockfile, Biome, and workflow changes.
• api_changed for rust/cloud-storage/**/*.rs, Rust lock/config files, flake files, API generation scripts, setup actions, and the workflow.

When api_changed is true, the Typecheck job:

1. sets up web prerequisites,
2. restores Rust cache for rust/cloud-storage,
3. runs bun run gen-api -- --check in js/app,
4. runs TypeScript checking.

--check mode fails when generated service-client files differ from the committed tree or when untracked generated files exist.

Troubleshooting

Generated types are out of sync with Rust API definitions

Run generation locally and commit the resulting changes:

cd js/app
bun run gen-api

For a smaller diff, target the changed service:

bun run gen-api -- cloud-storage

No matching services found

Use one of the name values from js/app/scripts/services.ts, such as cloud-storage, document-cognition, or search-service.

No crate mapping found, skipping

The service exists in services.ts but is missing from serviceToCrate in generate-api-schema.ts. Add the crate mapping before regenerating.

TypeScript type is missing or too broad

Check the Rust OpenAPI source:

• the model derives ToSchema,
• the model is included in components(schemas(...)),
• the handler is included in paths(...),
• the #[utoipa::path] response body names the expected type.

Orval writes to the wrong package

Keep these values aligned:

• services.ts output
• services.ts orvalKey
• orval.config.ts project key
• orval.config.ts input.target
• orval.config.ts output.target and output.schemas

OpenAPI binary fails in generation

The generator runs binaries from rust/cloud-storage/target/debug unless OPENAPI_BINS_DIR is set. Each binary has a 120 second timeout and reports captured stderr on non-zero exit or timeout.

To skip the cargo build phase with prebuilt binaries:

cd js/app
OPENAPI_BINS_DIR=/path/to/bins bun run gen-api -- <service-name>

Related pages

15. Generate service clients and tool schemas

Guide: Regenerate OpenAPI JSON, TypeScript clients, DCS model types, AI tool schemas, and MCP documentation pages from Rust sources.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/15-generate-service-clients-and-tool-schemas.md
• Generated: 2026-06-01T00:55:20.399Z

Source Files

• js/app/scripts/generate-api-schema.ts
• js/app/scripts/generate-dcs-tools.ts
• js/app/scripts/services.ts
• rust/cloud-storage/ai_tools/src/lib.rs
• rust/cloud-storage/ai_tools/src/bin/gen_tool_schemas.rs
• docs/scripts/generate-mcp-tool-pages.ts
• docs/package.json

Macro regenerates frontend service clients and AI tool artifacts from local Rust binaries under rust/cloud-storage; js/app/scripts/generate-api-schema.ts builds OpenAPI emitters, writes openapi.json, runs Orval, and applies DCS-specific model and tool generation when document-cognition is included.

Generated artifact map

Rust sources
  ├─ */src/openapi.rs                         -> service-*/openapi.json
  ├─ document_cognition_service/src/models_bin.rs -> service-cognition/generated/schemas/model.ts
  └─ ai_tools/src/bin/gen_tool_schemas.rs     -> ai_tools/schemas/tools.json
                                                   -> service-cognition/generated/tools/*
                                                   -> docs/AI/mcp/tools/*

Prerequisites

• Bun installed for js/app and docs scripts.
• Rust toolchain available for cargo build in rust/cloud-storage.
• Run generation from the package directory shown in each command.
• SQLx is built in offline mode by the scripts with SQLX_OFFLINE=true.

Regenerate service clients

cd js/app
bun run gen-api

cd js/app
bun run gen-api -- document-cognition

cd js/app
bun run gen-api -- --check

Service and crate mapping

generate-api-schema.ts maps public service names to Rust crate binaries. Each mapped crate must expose an OpenAPI binary named <crate>_openapi.

Build behavior

The script builds all requested OpenAPI binaries in one Cargo invocation:

cd rust/cloud-storage
SQLX_OFFLINE=true cargo build --bin <crate>_openapi ...

Then it runs each binary from target/debug, captures stdout as OpenAPI JSON, writes the service package openapi.json, removes the old generated directory, and runs:

cd js/app/packages/service-clients
bun run orval --config orval.config.ts --project <orvalKey>

Set OPENAPI_BINS_DIR to reuse prebuilt binaries and skip the Cargo build phase:

cd js/app
OPENAPI_BINS_DIR=/absolute/path/to/bins bun run gen-api

DCS model and tool generation

When document-cognition is part of the service set, generate-api-schema.ts performs extra work after Orval:

1. Runs document_cognition_service_models.
2. Writes temporary .models.json.
3. Runs MODELS_JSON=.models.json bun scripts/generate-dcs-types.ts.
4. Deletes .models.json.

generate-dcs-types.ts runs both DCS generators:

cd js/app
bun scripts/generate-dcs-types.ts

Model enum output

generate-dcs-models.ts writes:

js/app/packages/service-clients/service-cognition/generated/schemas/model.ts

When MODELS_JSON is set, it reads model metadata from the local file produced by the Rust model binary. Without MODELS_JSON, it fetches ${documentCognitionBase}/models, where MODE and LOCAL_BACKEND=true control the selected base URL.

The generated file exports:

• Model
• Model constant map
• AllModels
• ModelEnum

Tool schema output

Run the tool generator directly when only AI tool types changed:

cd js/app
bun run gen-tools

The generator builds and runs:

cd rust/cloud-storage
SQLX_OFFLINE=true cargo build --bin gen_tool_schemas

cd rust/cloud-storage/ai_tools
../target/debug/gen_tool_schemas

The Rust binary writes rust/cloud-storage/ai_tools/schemas/tools.json as a combined schema:

{
  "$defs": {},
  "tools": [
    {
      "name": "ContentSearch",
      "input": "ContentSearch",
      "output": "SearchToolResponse"
    }
  ]
}

generate-dcs-tools.ts validates that combined shape, dereferences shared definitions, and writes:

deserializeToolCall and deserializeToolResponse return neverthrow Result values. Unknown tool names return not_found; schema parse failures return parse_error.

Regenerate MCP tool documentation pages

The docs package exposes a separate generator:

cd docs
bun install
bun run generate:tools

This script rebuilds gen_tool_schemas, removes the generated MCP tools directory, then writes:

docs/AI/mcp/tools/index.mdx
docs/AI/mcp/tools/<slug>.mdx
docs/config/tool-pages.json

docs/package.json also runs this generator during prepare.

Runtime and schema ownership

rust/cloud-storage/ai_tools/src/lib.rs owns toolset composition:

• all_tools() builds the DCS chat toolset and prompt.
• all_tool_combined_schema() merges all_tools() with the ReadThread phantom tool for schema generation.
• mcp_tools() builds the MCP runtime toolset separately.

Do not assume generated MCP documentation is automatically scoped to the MCP runtime toolset unless the schema generator is changed to use mcp_tools().

Verification

After regeneration, check the generated files that match the source change:

git diff -- js/app/packages/service-clients
git diff -- rust/cloud-storage/ai_tools/schemas/tools.json
git diff -- docs/AI/mcp/tools docs/config/tool-pages.json

For frontend type safety:

cd js/app
bun run type-check

For docs links:

cd docs
bun run lint

CI runs bun run gen-api -- --check for Rust/API changes in the web app workflow and fails when generated service clients drift from Rust sources.

Troubleshooting

Related pages

16. Work with local seed data

Guide: Use seed_cli, local_e2e fixtures, manifest aliases, reset SQL, and shared Playwright and Rust fixture loaders.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/16-work-with-local-seed-data.md
• Generated: 2026-06-01T00:55:38.119Z

Source Files

• rust/cloud-storage/seed_cli/README.md
• rust/cloud-storage/seed_cli/src/main.rs
• rust/cloud-storage/seed_cli/seed/local_e2e/manifest.json
• rust/cloud-storage/seed_cli/seed/local_e2e/users.json
• rust/cloud-storage/seed_cli/seed/local_e2e/reset.sql
• js/app/tests/e2e/fixtures/local-e2e-seed.ts
• rust/cloud-storage/local_e2e_test_support/README.md

seed_cli owns the deterministic local fixture set in rust/cloud-storage/seed_cli/seed, and the repo-level local E2E harness applies it with just local-e2e-seed before Playwright or ignored Rust integration tests run against the local Docker stack.

Fixture layout

:::files

rust/cloud-storage/seed_cli/
  README.md
  justfile
  seed/
    channels.json
    channel_messages.json
    documents/
      documents.json
      files/
    local_e2e/
      manifest.json
      reset.sql
      users.json
  src/
    main.rs
    entity/
      scenario/mod.rs
      document/mod.rs
      channel/mod.rs
      channel_message/mod.rs

:::

Run the local seed

This starts local databases, drops and initializes `macrodb`, then runs the seed CLI local E2E scenario.

The harness starts the local E2E service subset, applies the seed, launches the frontend with local bearer-token auth, and runs `bunx playwright test` with `LOCAL_E2E=true`.

Rust local E2E tests are ignored by default and run with `SQLX_OFFLINE=true`.

Seed CLI behavior

The CLI entrypoint initializes the local Macro runtime, loads required environment variables, connects to Postgres, initializes FusionAuth and S3 clients, then dispatches an entity command.

The local smoke scenario is:

cd rust/cloud-storage/seed_cli
just local-e2e-smoke

Internally it runs:

cargo r -- scenario local-e2e-smoke

The local-e2e-smoke recipe supplies local-only defaults, including:

Local smoke scenario order

1. Load local_e2e/manifest.json and local_e2e/users.json.
2. Resolve the primary smoke user from manifest.user.email.
3. Delete local E2E contact backfill rows if public.contacts_backfill_outbox exists.
4. Execute local_e2e/reset.sql.
5. Delete and reinsert local user rows derived from users.json.
6. Seed documents from seed/documents/documents.json.
7. Seed channels from seed/channels.json.
8. Seed channel messages from seed/channel_messages.json.
9. Print Local e2e smoke seed data ready for <user_id>.

Reset scope

reset.sql removes fixture data by deterministic UUID prefixes:

User cleanup is generated from local_e2e/users.json and deletes matching rows from "User" and macro_user by auth user ID, macro user ID, or email before reinsert.

Manifest aliases

local_e2e/manifest.json is the stable contract for smoke tests. It does not duplicate full fixture rows; it names canonical fixture records that must exist in the seed files.

{
  "user": {
    "email": "[email protected]"
  },
  "documents": {
    "projectRoadmap": {
      "id": "00000000-0000-0000-0002-000000000001",
      "name": "Project Roadmap"
    }
  },
  "channels": {
    "general": {
      "id": "00000000-0000-0000-0000-000000000001",
      "name": "general",
      "message": "Welcome to the general channel everyone!"
    }
  }
}

When adding a new smoke alias, update the manifest and the underlying seed file in the same change. The shared Playwright and Rust loaders fail fast when an alias points at a missing user, document, channel, or message.

Playwright fixture loader

js/app/tests/e2e/fixtures/local-e2e-seed.ts reads the seed JSON directly from the repository root. It locates the root by walking upward until rust/cloud-storage/seed_cli/seed exists.

The exported localE2ESeed object includes:

Local Playwright specs skip unless LOCAL_E2E=true. The Playwright config generates LOCAL_JWT automatically for local E2E mode unless LOCAL_JWT is already exported.

cd js/app
LOCAL_E2E=true bunx playwright test

Prefer the repo-level harness:

just local-e2e

It seeds first and starts the frontend with:

VITE_LOCAL_SERVERS=ALL
VITE_ENABLE_BEARER_TOKEN_AUTH=true
LOCAL_JWT=<generated token>
bun run dev

Rust fixture loader

local_e2e_test_support is the Rust counterpart for ignored local integration tests. It reads the same seed files and exposes typed fixture accessors, service URL helpers, and local Macro API JWT generation.

use local_e2e_test_support::{
    LocalE2eConfig, LocalE2eSeed, LocalE2eServices, LocalJwtOptions,
    encode_local_jwt_with,
};

let config = LocalE2eConfig::load()?;
let seed = LocalE2eSeed::from_config(&config)?;
let services = LocalE2eServices::from_config(&config)?;

let user = seed.smoke_user()?;
let channel = seed.general_channel()?;
let token = encode_local_jwt_with(&config, LocalJwtOptions::new(user))?;
let ws_url = services.connection_gateway_ws_url_with_token(&token)?;

Rust helper defaults

Service URL overrides must remain local. The helper rejects non-local hosts and mismatched schemes for mutating local E2E tests.

Generate a local E2E token

Playwright calls js/app/scripts/generate-local-e2e-token.ts, which shells out to the Rust binary in local_e2e_test_support.

cd js/app
bun scripts/generate-local-e2e-token.ts

Optional arguments are forwarded to the Rust generator:

bun scripts/generate-local-e2e-token.ts \
  --email [email protected] \
  --expiry-seconds 3600 \
  --organization-id 1

MACRO_API_TOKEN_PRIVATE_SECRET_KEY is required in .env or the process environment.

Update fixture data safely

Add or change a seeded user

1. Edit rust/cloud-storage/seed_cli/seed/local_e2e/users.json.
2. Keep macro_user_id, fusion_user_id, and user_id stable once tests depend on them.
3. If this is the primary smoke user, update manifest.user.email.
4. Run just local-e2e-seed.
5. Run at least one consumer:

   just local-e2e
   # or
   just local-e2e-rust
   

Add or change a seeded document

1. Add a row to seed/documents/documents.json.
2. Put the source file under seed/documents/files/.
3. Use a deterministic document ID in the 00000000-0000-0000-0002-* range if it should be cleaned by reset.sql.
4. Update manifest.documents only for canonical smoke-test aliases.
5. Run just local-e2e-seed.

Document seeding creates database metadata and uploads the fixture file to the configured document storage bucket.

Add or change a seeded channel

1. Edit seed/channels.json.
2. Use a deterministic channel ID in the 00000000-0000-0000-0000-00000000000* range if it should be cleaned by reset.sql.
3. Set channel_type to a value accepted by the seed CLI model, such as public, private, or direct_message.
4. List participants excluding or including the owner; the seed command appends the scenario owner when absent.
5. Update manifest.channels only for canonical smoke-test aliases.

Add or change a seeded message

1. Edit seed/channel_messages.json.
2. Use a deterministic message ID in the 00000000-0000-0000-0001-* range if it should be cleaned by reset.sql.
3. Set channel_id to an existing seeded channel.
4. Set sender_id to a seeded auth user ID such as macro|[email protected].
5. Use thread_id only when the message is a reply.
6. Add entity_mentions when the message content references a document or user.

For non-user shareable mentions, the seed command attempts to create message mentions and update channel share permissions for the mentioned entity.

Troubleshooting

Related pages

17. Storage and workspace APIs

Reference: Document, channel, project, call, CRM, soup, pin, history, permissions, properties, and search endpoints.

• Page Markdown: https://grok-wiki.com/public/docs/macro-inc-macro-bb988e1a448e/pages/17-storage-and-workspace-apis.md
• Generated: 2026-06-01T01:00:28.495Z

Source Files

• js/app/packages/service-clients/service-storage/openapi.json
• js/app/packages/service-clients/service-storage/client.ts
• rust/cloud-storage/document_storage_service/src/openapi.rs
• js/app/packages/service-clients/service-properties/openapi.json
• js/app/packages/service-clients/service-properties/client.ts
• js/app/packages/service-clients/service-search/openapi.json
• js/app/packages/service-clients/service-search/client.ts

Macro’s workspace API surface is served primarily by Document Storage Service (DSS), with /properties and /search mounted on the same document-storage-service host and consumed from TypeScript clients that wrap fetchWithToken and return neverthrow Result values.

Runtime and client boundaries

The Rust OpenAPI JSON for DSS is produced from rust/cloud-storage/document_storage_service/src/openapi.rs, which prints the utoipa ApiDoc. The checked-in frontend OpenAPI snapshots under js/app/packages/service-clients/*/openapi.json and generated schema directories provide the wire types used by the clients.

Documents

Core document operations

Creation and copy paths show the file-limit paywall when the returned error message includes 403.

Locations and document bytes

Presigned URL caching uses a 14-minute client lifetime because the server expiration is treated as 15 minutes.

Processing and task helpers

Processing helpers currently validate PREPROCESS results against CoParseSchema; missing or invalid JSON returns INVALID_RESPONSE.

Annotations

The storageServiceClient.annotations namespace wraps document comments and anchors.

Channels

Channel lifecycle

getChannels still targets /comms/channels on the DSS host; the source comment says it should move to /channels when the comms teardown finishes.

Messages, replies, reactions, attachments