cfw-logging/README.md

# Logger System

Independent TypeScript logger collection system with:

- `@logger/sdk`: application logger API and transports.
- `@logger/collector`: JSONL store, HTTP ingest, and read-only HTTP query.
- `@logger/cli`: `log tail` / `log config` for reading records, driven by a `.logging/config.json` profile.
- `logging-agent` skill: a usage contract for Codex / Claude Code to read logs safely.

## Install

```bash
pnpm install
pnpm test
pnpm build
```

## SDK

```ts
import {createLogger, HttpTransport} from '@logger/sdk';

const log = createLogger({
  app: 'shop',
  service: 'orders',
  scope: 'orders.refund',
  runtime: 'server',
  transport: new HttpTransport('http://127.0.0.1:4319/ingest'),
});

log.info('hello');
log.error('error msg', new Error('payment timeout'), {orderId: 'o-2099'});
await log.flush();
```

Records are structured JSON with `timestamp`, `level`, `message`, `app`, `service`, `scope`, `runtime`, optional `attributes`, optional serialized `error`, and `version: 1`.

Sensitive fields are redacted by default when passed in attributes:

- `password`
- `token`
- `secret`
- `apiKey`
- `authorization`
- `cookie`
- `set-cookie`

## Collector

Use `JsonlLogStore` directly:

```ts
import {JsonlLogStore} from '@logger/collector';

const store = new JsonlLogStore('.logging/logs.jsonl');
await store.append(record);
const latest = await store.tail({limit: 10, level: 'error'});
```

Start an HTTP collector server from application code:

```ts
import {createHttpCollectorServer, JsonlLogStore} from '@logger/collector';

const store = new JsonlLogStore('.logging/logs.jsonl');
createHttpCollectorServer(store).listen(4319);
```

Endpoints:

- `POST /ingest` — append a single `LogRecord` or an array of `LogRecord` objects.
- `GET /query` — read-only tail. Supports query params `level`, `app`, `service`, `scope`, `since`, `limit`; returns a JSON array of matching records (last `limit`, file order).

## Configuration (`.logging/config.json`)

The CLI is driven by a project-local profile at `.logging/config.json`. It declares the system name, where logs live, optional remote access, and safe defaults. Secrets are never stored in the file — only the name of the environment variable that holds a token.

```json
{
  "version": 1,
  "systemName": "shop",
  "local": {"file": ".logging/logs.jsonl"},
  "server": {"baseUrl": "https://logs.example.com", "queryPath": "/query", "ingestPath": "/ingest"},
  "auth": {"type": "bearer", "tokenEnv": "LOGGING_TOKEN"},
  "defaults": {"app": "shop", "service": "orders", "limit": 50},
  "output": {"format": "compact", "redact": true, "maxLines": 200}
}
```

Required: `version` (must be `1`) and `systemName`. Everything else is optional. See `skills/logging-agent/references/config-contract.md` for the full field reference and security rules.

### Bootstrapping a profile

```bash
log config init --system=shop
log config init --system=shop --server=https://logs.example.com --token-env=LOGGING_TOKEN
```

`config init` writes `.logging/config.json` plus a `.logging/.gitignore` that ignores `*.local.json`, `*.secret.json`, `tokens*`, and `logs.jsonl` while keeping `config.json` committable. Pass `--force` to overwrite an existing profile.

### Checking a profile

```bash
log config doctor
```

Validates the config shape, confirms `systemName`, checks that any required token environment variable is set (without printing its value), probes remote reachability when a server is configured, and checks the local file. Exits non-zero if any hard check fails.

## CLI

After build:

```bash
node packages/cli/dist/index.js tail
node packages/cli/dist/index.js tail --level=error --app=shop --json
node packages/cli/dist/index.js tail --remote --json
```

The package binary name is `log`:

```bash
log tail
log tail --level=error --app=shop --service=orders --scope=orders.refund --json
log config init --system=shop
log config doctor
```

### Where `tail` reads from

`tail` resolves its source predictably and never silently calls a remote service:

1. `--remote` → query the configured `server` (requires `server.baseUrl`; auth resolved from `auth`).
2. `--file=<path>` or `LOGGER_FILE` → read that local JSONL file.
3. `config.local.file` → read the configured local file.
4. `config.server` configured with no local source → query the server.
5. fallback → read `.logging/logs.jsonl` and print a hint to run `log config init`.

### Filter and output resolution

Command-line flags override `config.defaults`, which override the built-in defaults. Supported filters: `--level`, `--app`, `--service`, `--scope`, `--since`, `-n`/`--limit`. Output is capped to `config.output.maxLines` (default `200`); `--json` emits one record per line; `output.redact` (default `true`) masks known sensitive fields as a safety net even for records that were not redacted at ingest.

Missing config, missing token, or a missing query endpoint each produce a clear error message instead of a silent guess.

## Publishing (private Verdaccio)

Build first, then publish with the helper script:

```bash
pnpm build
./scripts/publish.sh            # version defaults to packages/shared-schema's version
./scripts/publish.sh 0.2.2      # or pass an explicit version
```

Use the script instead of `pnpm publish -r`. Verdaccio v6 does **not** extract README from the tarball — it only reads the manifest `readme` field, which modern `npm`/`pnpm publish` no longer populate, so a plain publish makes the registry UI show "No README data found!". The script injects `readme` into each published manifest (rewriting `workspace:*` to the real version on the way, then restoring `package.json`).

The registry at `http://192.168.0.15:4873` is the same Verdaccio as `http://localhost:4873`; anonymous reads are allowed, only publishing needs an `_authToken`. Verify a publish with `curl -s http://192.168.0.15:4873/@logger/cli | node -e "let s='';process.stdin.on('data',d=>s+=d).on('end',()=>console.log('readme len:',(JSON.parse(s).readme||'').length))"`.

## Docker (collector server)

The `Dockerfile` builds an image that installs the published `@logger/collector` from Verdaccio and runs its `logging-collector` server entry:

```bash
docker build -t logger-collector:0.1.0 .
docker run -d --name logger-collector -p 4319:4319 -v logger-collector-data:/data logger-collector:0.1.0
```

Configure with env vars `PORT` (default `4319`), `HOST` (default `0.0.0.0`), `LOG_FILE` (default `/data/logs.jsonl`). Endpoints: `POST /ingest`, `GET /query`. Logs persist to the mounted volume.

The build reads from `http://host.docker.internal:4873` by default (Docker Desktop's alias for the host, where the Verdaccio port is published). Override with `--build-arg REGISTRY=http://192.168.0.15:4873` if your Docker network reaches the registry directly.

End-to-end (SDK writes, CLI reads), from a consumer project:

```bash
npm install @logger/sdk @logger/cli --registry=http://192.168.0.15:4873
# .logging/config.json -> { "server": { "baseUrl": "http://127.0.0.1:4319", "queryPath": "/query" }, "auth": {"type":"none"} }
npx log tail --remote --json --level=error
```

## Cloudflare Worker + D1 (remote collector)

`packages/worker` is the Cloudflare-native collector. It exposes the same HTTP surface as the Node collector, but stores records in D1:

- `GET /healthz` — unauthenticated liveness check.
- `POST /ingest` — authenticated ingest for one `LogRecord` or an array of records.
- `GET /query` — authenticated read-only tail query with `level`, `app`, `service`, `scope`, `since`, and `limit`.

The Worker uses D1 binding `DB` and Wrangler secret `LOGGING_TOKEN`. Do not put the token in `wrangler.jsonc`.

```bash
cd packages/worker
pnpm wrangler d1 create logger-collector
# copy the returned database_id into packages/worker/wrangler.jsonc
pnpm wrangler d1 migrations apply logger-collector --remote
pnpm wrangler secret put LOGGING_TOKEN
pnpm wrangler deploy
```

Consumer `.logging/config.json`:

```json
{
  "version": 1,
  "systemName": "shop",
  "server": {
    "baseUrl": "https://logger-collector.<account>.workers.dev",
    "queryPath": "/query",
    "ingestPath": "/ingest"
  },
  "auth": {"type": "bearer", "tokenEnv": "LOGGING_TOKEN"},
  "defaults": {"app": "shop", "limit": 50},
  "output": {"format": "compact", "redact": true, "maxLines": 200}
}
```

SDK usage:

```ts
import {createLogger, HttpTransport} from '@logger/sdk';

const log = createLogger({
  app: 'shop',
  scope: 'orders',
  runtime: 'worker',
  transport: new HttpTransport('https://logger-collector.<account>.workers.dev/ingest', {
    headers: {authorization: `Bearer ${process.env.LOGGING_TOKEN}`},
  }),
});
```

## Current Boundaries

Included:

- TypeScript SDK.
- Console, memory, batch, and HTTP transports.
- JSONL append-only store.
- HTTP ingest and read-only HTTP query endpoints.
- Cloudflare Worker remote collector backed by D1.
- CLI driven by `.logging/config.json`: `tail`, `config init`, `config doctor`, local and remote reads, redaction, and `maxLines` safety cap.
- `logging-agent` skill for Codex / Claude Code.

Not included yet:

- UI console.
- Multi-tenant access control (the Worker first version uses one shared Bearer token).
- Full-text search index.
- Alerting.
- Metrics and trace collection.
- OpenTelemetry exporter.