built filter, operations, etc. for migrations
This commit is contained in:
John Yeo
145
.context/migration-interface/CATALOG.md
Normal file
145
.context/migration-interface/CATALOG.md
Normal file
@@ -0,0 +1,145 @@
|
||||
# Catalog: Migration & Import Patterns
|
||||
|
||||
Source: 4-agent parallel discovery on 2026-04-29 covering `scripts/src/common/migrations/`, `scripts-v2/runs/`, `scripts/mintlify-import.sh` chain, `scripts/firecrawl-import-from-csv.sh` chain, plus a deep read of billing v2 action machinery.
|
||||
|
||||
---
|
||||
|
||||
## A. Migration archetypes (what scripts actually do)
|
||||
|
||||
### A1. Version bump via `billingActions.migrate()`
|
||||
- `scripts/src/common/migrations/perform-migration.ts`, `lingo.ts`, `lingo-prepaid-to-payperuse.ts`
|
||||
- `scripts-v2/runs/browser-use/migrate-free/` (uses `billingActions.migrate` with `noBillingChanges: true`)
|
||||
- Trivially fits `{ customerId, plans: [{ expire: cusProductId, insert: { productId, version } }] }`.
|
||||
- Note: scripts loop multiple matching cusProducts per customer and reload state mid-loop because `migrate()` mutates state.
|
||||
|
||||
### A2. Multi-cusProduct subscription update — **the canonical complex case**
|
||||
- `scripts-v2/runs/mintlify/migrate-credits/multi-update/` — replaces legacy AI-credits prepaid with V2 prepaid+consumable across N cusProducts sharing one Stripe sub.
|
||||
- Operates on `CusProductGroup = { subscriptionId, cusProducts[] }`.
|
||||
- Per group: `setupStripeBillingContext` once → per-product `computeCustomPlanNewCustomerProduct` → aggregate into `AutumnBillingPlan` → `evaluateStripeBillingPlan` → `executeBillingPlan`.
|
||||
- This is structurally ~95% of the proposed interface, coupled to AI-credits-specific helpers (`resolveAiCreditsQuantity`, `setupUpdateFullProductContext`).
|
||||
|
||||
### A3. Pricing-shape swap, cusProduct identity preserved
|
||||
- `scripts/src/common/migrations/firecrawl-credit-pack-to-usage-price.ts`
|
||||
- Keeps the same `customer_product_id`, replaces its child cus_ents/cus_prices with custom rows. Preserves balance/reset/expiry.
|
||||
- **Does NOT fit `{expire, insert}`** — cusProduct stays.
|
||||
|
||||
### A4. 1→N cusProduct split
|
||||
- `scripts/src/common/migrations/split-one-cus-product-into-two.ts` (Email → Domain + Inbox).
|
||||
- Expires one cusProduct, inserts N new ones referencing the SAME `subscription_ids`.
|
||||
- Fits `{ expire, insert: [...] }` only if interface explicitly supports shared subscription_ids and balance/option carryover per inserted plan.
|
||||
|
||||
### A5. Flag-flip / balance-clamp on existing rows
|
||||
- `cont-use-to-single-use.ts`, `fix-creator-connections-interval.ts`, `browser-use-reset-spam-users.ts`, `additive-entitlement-migration.ts` (gifting branch), `retroactively-add-plan-item-to-customers.ts` "update" mode.
|
||||
- Pure UPDATE on `entitlements` / `customer_entitlements`. No cusProduct churn, no Stripe.
|
||||
- **Does not fit `{expire, insert}` at all** — separate primitive.
|
||||
|
||||
### A6. Raw-DB bulk replace (no Stripe, no billing v2)
|
||||
- `scripts-v2/runs/sebipaps/sandbox-product-sync/migrate-plan.ts` — chunked `delete cus_ent + delete cus_price + update cusProduct.internal_product_id`.
|
||||
- Free-tier or 1:1 product remap. Bypasses billing v2 entirely.
|
||||
- Should be subsumed by a "DB-only bulk swap" mode of the unified interface.
|
||||
|
||||
---
|
||||
|
||||
## B. Catalog-prep archetypes (what runs BEFORE migrations)
|
||||
|
||||
These are *preparation* patterns — extending the product catalog so a migration can land.
|
||||
|
||||
### B1. Add entitlement (+optional price) to all versions of a product
|
||||
- `retroactively-add-plan-item-to-versions.ts` — uses latest version that already has the ent as a TEMPLATE, clones onto older versions.
|
||||
- `FINAL-add-entitlement-to-plans.ts` — adds feature to all versions matching id-prefix.
|
||||
- `add-creator-connections-entitlement.ts`, `add-boolean-customer-entitlement.ts`, `add-feature-to-customers.ts`, `add-new-boolean-feature.ts`.
|
||||
- Pure catalog (`entitlements`, `prices`) mutations. No customer-side touch.
|
||||
|
||||
### B2. Backfill cus_ents for existing customers on extended products
|
||||
- `retroactively-add-plan-item-to-customers.ts` — sibling of B1; per active cusProduct on listed products, insert missing `customer_entitlements` (and raw `customer_prices` if `requirePrice`).
|
||||
- `add-creator-connections-cus-entitlement.ts`, `add-boolean-customer-entitlement.ts` (also does B1).
|
||||
- Uses `initCusEntitlement` + direct `CusEntService.insert`. **Zero scripts use** `createFullCusProduct`.
|
||||
|
||||
### B3. Catalog clone (sandbox → live, or version bump)
|
||||
- `scripts-v2/runs/sebipaps/sandbox-product-sync/copy-products.ts` — `handleCopyFeatures`, `handleVersionProductV2`, `createProduct`.
|
||||
- `scripts-v2/runs/mintlify/migrate-credits/steps/init-plan-resources/` — deterministic-id init for V2 prepaid/consumable price+entitlement; sibling entrypoint `setup-credits-catalog.ts`.
|
||||
- Pattern: prep entrypoint shares folder + `data/` with the migration entrypoint.
|
||||
|
||||
### B4. Stripe-side prep (clone archived Stripe prices)
|
||||
- `scripts-v2/runs/mintlify/fix-archived-price/` — clones archived Stripe prices, updates `Price.config.stripe_price_id`, then patches subscription items.
|
||||
- Doesn't fit `{expire, insert}` — it's a price-id swap on the same cusProduct.
|
||||
|
||||
---
|
||||
|
||||
## C. Import patterns (Stripe → Autumn)
|
||||
|
||||
### C1. Mintlify import (`scripts/src/mintlify/import/`)
|
||||
- Source: Mongo orgs/deployments/period_usage. Stripe customers/subs.
|
||||
- Steps: create customers → create entities (per deployment) → attach base plans via `createFullCusProduct` → derive addons from Stripe subs → "loose" custom entitlements via direct Drizzle insert → sync period usage into balances → cleanup.
|
||||
- Plan resolver: Mongo `plan` string field.
|
||||
- Custom allowances: signature-deduped per-customer entitlement rows.
|
||||
|
||||
### C2. Firecrawl import (`scripts/src/firecrawl/import/`)
|
||||
- Source: BigQuery `autumn_shared` (teams, orgs, balances, coupons). Stripe subs/prices/customers.
|
||||
- Steps: select Stripe subs → resolve plan from `lookup_key` → resolve/create customer → ensure team entity → build target product state with per-customer overrides (custom allowance, custom topup price, signature dedup) → flush via `customerProductService.insertIfNotExists` → repair Stripe sub link → schedule end-of-cycle plan switches via `billingActions.attach` with `no_billing_changes: true` → sync team credit state → import coupon balances directly to `customer_entitlements.balance` with deterministic ids.
|
||||
- Plan resolver: Stripe `price.lookup_key`.
|
||||
- Custom allowances: per-customer freshly-created `entitlements` + `prices` rows with signature dedup cache.
|
||||
|
||||
### Cross-cutting import friction (top 5)
|
||||
1. Stripe price → Autumn plan resolver is org-specific (Mongo field vs lookup_key vs price.metadata).
|
||||
2. Per-customer catalog overrides require signature-deduped per-customer `entitlements`+`prices` rows.
|
||||
3. Linking to existing Stripe sub (idempotent), not creating it. `initOptions.subscriptionId` + repair.
|
||||
4. Balance carryover is a separate concern: deterministic `balanceId`, direct write to `customer_entitlements.balance`, bypass `createFullCusProduct`.
|
||||
5. Entity scoping + duplicate cleanup (same Stripe sub re-attached to wrong entity, stale entities, duplicate plans across entities).
|
||||
|
||||
---
|
||||
|
||||
## D. Billing v2 architecture (the substrate)
|
||||
|
||||
### D1. The 5-stage pipeline (every action follows this)
|
||||
```
|
||||
setup<Action>BillingContext({ ctx, params }) -> <Action>BillingContext
|
||||
handle<Action>Errors({ ctx, billingContext }) -> throws RecaseError
|
||||
compute<Action>Plan({ ctx, billingContext }) -> AutumnBillingPlan
|
||||
evaluateStripeBillingPlan({ ctx, billingContext, autumnBillingPlan }) -> StripeBillingPlan
|
||||
executeBillingPlan({ ctx, billingContext, billingPlan }) -> BillingResult
|
||||
```
|
||||
|
||||
### D2. AutumnBillingPlan already expresses every migration shape
|
||||
- `insertCustomerProducts: FullCusProduct[]`
|
||||
- `updateCustomerProduct(s)?: CustomerProductUpdate[]` — patch options/status/canceled/anchor/sub_ids
|
||||
- `deleteCustomerProduct(s)?: FullCusProduct[]` — for scheduled product replacement
|
||||
- `customPrices?: Price[]`, `customEntitlements?: Entitlement[]`, `customFreeTrial?`
|
||||
- `updateByStripeScheduleId?` — for schedule swaps
|
||||
- `insertCustomerEntitlements?`, `updateCustomerEntitlements?`
|
||||
- `lineItems?`, `customLineItems?`
|
||||
- `autoTopupRebalance`, `upsertSubscription`, `upsertInvoice`, `refundPlan`
|
||||
|
||||
### D3. evaluateStripeBillingPlan capabilities
|
||||
- ✅ Multi-product, mixed prepaid/metered, schedules, cancel+replace, multi-entity, refunds, manual invoices.
|
||||
- ✅ DB-only mode: `billingContext.skipBillingChanges = true` returns `{}` and execute no-ops Stripe.
|
||||
- ⚠️ Single-customer per evaluation. No batch mode.
|
||||
|
||||
### D4. Existing `migrate` action
|
||||
- `actions/migrate/migrate.ts` is already in the registry. Need to read what it actually does — likely the simple version-bump path; we may extend or replace.
|
||||
|
||||
### D5. Critical gap: shared custom rows
|
||||
- `setupCustomFullProduct` always mints fresh per-customer `Price`/`Entitlement` rows when `hasCustomItems` is true.
|
||||
- For bulk migration where 10k customers should share the same prepared price/ent rows, we need either:
|
||||
- A `prepared` mode that reuses existing catalog rows, OR
|
||||
- A separate one-shot prep phase that creates the shared rows once, with the migration action only inserting `customer_products` + `customer_prices`/`customer_entitlements` referencing those existing IDs.
|
||||
|
||||
### D6. customizePlanV1 is PUT-style
|
||||
- `items?: ApiPlanItemV1[]` — when provided, replaces full list.
|
||||
- For migrations we want PATCH-style (incremental add/remove/update of plan items).
|
||||
|
||||
---
|
||||
|
||||
## E. Verdict on `{ customerId, plans: [{ expire, insert }] }`
|
||||
|
||||
Fits cleanly: **A1, A2 (with subscription-group dimension), A4 (with shared sub_ids), A6 (with DB-only mode).**
|
||||
|
||||
Does NOT fit, needs separate primitive(s):
|
||||
- **A3** — pricing-shape swap on same cusProduct identity.
|
||||
- **A5** — flag-flip / balance-clamp on existing rows.
|
||||
- **B1, B2** — catalog prep & cus_ent backfill (different layer entirely).
|
||||
- **B3** — catalog clone (different layer).
|
||||
- **B4** — Stripe price-id swap (different layer).
|
||||
- **C1, C2 imports** — need 5 additional dimensions on top of migration shape.
|
||||
|
||||
**Implication: the unified "migration interface" is really a layered system, not a single action.**
|
||||
29
.context/migration-interface/DECISIONS.md
Normal file
29
.context/migration-interface/DECISIONS.md
Normal file
@@ -0,0 +1,29 @@
|
||||
# Decisions
|
||||
|
||||
Append-only log of architectural and design decisions.
|
||||
|
||||
## 2026-04-29 — Project lives in `autumn/.context/`, not the outer worktree
|
||||
The work targets first-class billing actions in `autumn/server/src/internal/billing/v2/actions/` and ships with tests in `autumn/`. Context is colocated with the code it describes. The outer `.context/migration-interface/` holds only a redirect README.
|
||||
|
||||
## 2026-04-29 — Phase 1 is cataloging, not designing
|
||||
The user explicitly asked to survey existing migration & import scripts before committing to interface shape. Risk of premature lock-in is high — there are at least three classes of patterns (DB-only mutations, DB+Stripe mutations, imports with balance carryover) and each could break the proposed shape differently. Output of Phase 1 is a verdict-per-script catalog.
|
||||
|
||||
## 2026-04-29 — Build as core product feature, not internal tooling
|
||||
Will eventually surface to users on the dashboard. Implications: lives in `autumn/server/src/internal/operations/` (peer to `billing/v2/`), has integration tests, no `as any`, follows full billing-action conventions for the heavyweight executors (`setupBillingContext` → compute Autumn plan → `evaluateStripeBillingPlan` → execute).
|
||||
|
||||
## 2026-04-29 — Planner + Executor model, not single billing action
|
||||
Migrations and imports differ in HOW they produce a plan but converge in HOW the plan executes. A single `migrate` action would conflate heterogeneous mutation surfaces (catalog, customer product graph, balance, Stripe repair) into one sprawling compute function. Splitting produces:
|
||||
- Durable, inspectable, persistable `OperationPlan` artifact
|
||||
- Dry-run identical to live (only `apply()` differs)
|
||||
- Pluggable planners (MigrationDefinition, ImportDefinition) emit same operation graph
|
||||
- Specialized executors per mutation surface, each with own idempotency/guards
|
||||
- Reuses billing v2 pipeline INSIDE the CustomerProductGraphOperation executor — no replacement, just orchestration above
|
||||
|
||||
## 2026-04-29 — Operations typed by mutation surface, not lifecycle intent
|
||||
Initial design used `PlanOp = expire | insert | swap | patch` (lifecycle-flavored). Better: `Operation = CatalogOperation | CustomerProductGraphOperation | CustomerEntitlementOperation | BalanceOperation | StripeRepairOperation | CacheInvalidationOperation`. Each surface has different idempotency, transaction boundaries, and concurrency rules — surface-typed ops let executors specialize cleanly.
|
||||
|
||||
## 2026-04-29 — `is_custom` decoupled from "uses shared catalog rows"
|
||||
Today `setupCustomFullProduct` always mints fresh per-customer Price/Entitlement rows when customize is provided. For 10k-customer migrations this creates 10k Stripe prices unnecessarily. New model: `ProductItemPatch.add_existing { entitlement_id, price_id }` references shared catalog rows by id; `is_custom` becomes purely an ownership flag (one-off vs catalog), not "this row is per-customer." Existing custom rows untouched; new code uses new model.
|
||||
|
||||
## 2026-04-29 — Ownership is first-class
|
||||
Today repair logic is bespoke per-script because we can't distinguish imported from user-created customer_products. New model: `Ownership { origin: "imported"|"user_created"|"migrated", source?, imported_at? }` on every TargetAssignment. Likely stored as new column on `customer_products`. Unblocks safe re-runnable imports and convergence/repair semantics.
|
||||
71
.context/migration-interface/PLAN.md
Normal file
71
.context/migration-interface/PLAN.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# migration-interface Plan
|
||||
|
||||
## Phase 1: Catalog & Stress-Test (IN PROGRESS)
|
||||
Goal: enumerate every migration/import pattern we've shipped, then check the proposed interface against each one. Output: a catalog doc with per-script summaries and "interface covers? / what breaks?" verdicts.
|
||||
|
||||
- Survey `scripts/src/common/migrations/` (retroactively-add-plan-item-to-customers, -to-versions, -plan-items, plus siblings)
|
||||
- Survey `scripts-v2/runs/` migration scripts (mintlify migrate-credits is the canonical example)
|
||||
- Survey `scripts/mintlify-import.sh` and `scripts/firecrawl-import.sh` import flows
|
||||
- Categorize each script along axes:
|
||||
- DB-only vs DB + Stripe
|
||||
- Single-customer vs bulk
|
||||
- Plan transformation shape (add item / swap plan / change pricing / cancel-and-replace / schedule)
|
||||
- Custom per-customer overrides (rollover, lifetime balance, balance carryover)
|
||||
- Preparation step required (new prices/products created before migration)
|
||||
- Verdict per script: covered by proposed interface? What's missing?
|
||||
|
||||
## Phase 2: Interface Design
|
||||
- Lock the migration action signature(s) — likely one new `migrate` action alongside `multiAttach` and `createSchedule`
|
||||
- Solve the "insert + customize" friction:
|
||||
- Move customize from PUT-style (full item list) to PATCH-style (incremental add/remove/update)
|
||||
- Decide: do migrations create new custom price/entitlement rows, or reuse a single per-plan row? (Stripe price/product creation is the binding constraint)
|
||||
- Decide whether "preparation" (creating shared prices/entitlements ahead of migration) is a separate action or a step in the same one
|
||||
- Validate design against catalog from Phase 1
|
||||
|
||||
## Phase 3: Implementation
|
||||
- Build the action(s) under `autumn/server/src/internal/billing/v2/actions/`
|
||||
- Tests: integration tests covering catalog scenarios from Phase 1
|
||||
- Reuse `evaluateStripeBillingPlan`, `setupBillingContext` patterns from existing actions
|
||||
|
||||
## Phase 4: Migrate Existing Scripts
|
||||
- Port mintlify migrate-credits to use the new action
|
||||
- Port retroactively-add-plan-item-to-customers
|
||||
- Each port validates the interface; refine as needed
|
||||
|
||||
## Phase 5 (stretch): Imports
|
||||
- Extend or sibling action for Stripe→Autumn linkage
|
||||
- Handle the hard cases: balance carryover for metered features, custom configurations (rollovers, lifetime balances), many-plans Stripe accounts
|
||||
- Goal: collapse the bespoke mintlify-import.sh / firecrawl-import.sh logic
|
||||
|
||||
---
|
||||
|
||||
## Initial Design Sketch (from kickoff conversation)
|
||||
|
||||
**Scope of one call:** one customer, one Stripe subscription's worth of customer_products.
|
||||
|
||||
**Proposed shape:**
|
||||
```ts
|
||||
{
|
||||
customerId: string,
|
||||
subscriptionId?: string, // inferable from plans
|
||||
plans: [
|
||||
{
|
||||
expire?: { customer_product_id?: string; plan_id?: string; entity_id?: string },
|
||||
insert?: { plan_id: string; customize?: ... },
|
||||
}
|
||||
],
|
||||
schedules?: [...] // future, optional
|
||||
}
|
||||
```
|
||||
|
||||
**Open questions:**
|
||||
1. Customize: PATCH vs PUT — strongly leaning PATCH (incremental add/remove/update of plan items)
|
||||
2. Custom prices: per-customer rows vs shared per-plan row reused across migrations (Stripe-side cost is the constraint)
|
||||
3. Preparation step (creating shared prices/entitlements + backfilling onto prior plan versions and customer_products): part of this interface or a separate "prepare migration" action?
|
||||
4. Mapping public `apiPlanItemV1` ↔ DB `price + entitlement` pair — how does the interface accept input?
|
||||
|
||||
## Architecture Notes
|
||||
- Build on `evaluateStripeBillingPlan` — the Autumn-plan → Stripe-state mapper already handles schedules, multi-entity, mixed prepaid/metered. Don't reinvent.
|
||||
- `customer.processor?.id` is Stripe customer ID (per autumn-operations-constraints rule). Watch the ID confusion.
|
||||
- Stripe writes must respect: amount conversion via `atmnToStripeAmount`, idempotent price creation via `createStripePriceIFNotExist`, `proration_behavior: "none"` for surgical item changes.
|
||||
- This is product code, not script code — full TypeScript discipline, no `as any`, full test coverage.
|
||||
306
.context/migration-interface/PROPOSAL.md
Normal file
306
.context/migration-interface/PROPOSAL.md
Normal file
@@ -0,0 +1,306 @@
|
||||
# Proposal: Operation Plan + Pluggable Planners
|
||||
|
||||
## Mental model
|
||||
|
||||
> **Planning produces a typed `OperationPlan`. Execution consumes it.**
|
||||
> Migrations and imports differ in how they *produce* the plan; everything downstream — guards, dry-run, persistence, execution, reconciliation — is shared.
|
||||
|
||||
This is a strict departure from "build one big billing action." The migration interface is not a single action — it is a **planner / executor pair**, where the plan is a durable, inspectable artifact and execution is a typed reducer over operations.
|
||||
|
||||
The reason: scripts today informally do exactly this lifecycle (scope → load → resolve → prepare → plan → guard → execute → verify), but the contracts between stages are bespoke. The win is canonizing the stage contracts, not unifying the verbs.
|
||||
|
||||
---
|
||||
|
||||
## Why planner + executors (vs. single billing action)
|
||||
|
||||
1. **Operations are heterogeneous.** Catalog mutations, customer-product graph changes, balance syncs, Stripe repair, cache invalidation — each has different idempotency, different transaction boundaries, different blast radius. One "migrate" action that handles all of them gets a sprawling `compute*Plan` function. Separate executors keep each surface clean.
|
||||
2. **Plans are first-class artifacts.** Persistable. Diffable. Re-runnable. Approval-able. Eventually user-facing on the dashboard ("preview migration → approve → apply").
|
||||
3. **Dry-run = real plan.** Same code path produces the plan; only `apply()` differs. No drift.
|
||||
4. **Imports vs migrations diverge cleanly.** They're different *planners* emitting the same operation graph. Shared execution; specialized planning.
|
||||
5. **Concrete reuse path.** `CustomerProductGraphOperation.replace` executor wraps the existing 5-stage billing v2 pipeline. We're not replacing billing v2 — we're putting an orchestration layer above it.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Module placement
|
||||
|
||||
```
|
||||
autumn/server/src/internal/operations/
|
||||
types/
|
||||
operation.ts -- Operation discriminated union
|
||||
operationPlan.ts -- OperationPlan, CustomerOperationPlan, RunContext
|
||||
operationResult.ts -- OperationResult, GuardResult, MigrationDiff
|
||||
productItemPatch.ts -- ProductItemPatch union (PATCH-style customize)
|
||||
executors/
|
||||
catalog/ -- CatalogOperation executor
|
||||
customerProductGraph/ -- CustomerProductGraphOperation executor (wraps billing v2)
|
||||
customerEntitlement/ -- CustomerEntitlementOperation executor
|
||||
balance/ -- BalanceOperation executor
|
||||
stripeRepair/ -- StripeRepairOperation executor
|
||||
cache/ -- CacheInvalidationOperation executor
|
||||
planners/
|
||||
migration/ -- MigrationDefinition runner
|
||||
import/ -- ImportDefinition runner
|
||||
guards/ -- composable, named guard sets
|
||||
reporting/ -- diff, audit, CSV export
|
||||
persistence/ -- migration_runs / migration_plans schema + service
|
||||
execute.ts -- top-level executePlan(plan, { dryRun })
|
||||
```
|
||||
|
||||
This sits **as a peer** to `billing/v2/`, not under it. `billing/v2/` is the substrate; `operations/` is the orchestration layer.
|
||||
|
||||
### Type model
|
||||
|
||||
```ts
|
||||
// The durable artifact
|
||||
type OperationPlan = {
|
||||
run: RunContext; // org_id, env, dry_run, source description, created_at
|
||||
catalog: Operation[]; // run-level (executes once, before per-customer)
|
||||
customers: CustomerOperationPlan[];// per-customer
|
||||
warnings: GuardResult[]; // non-blocking
|
||||
blockers: GuardResult[]; // execution refuses if any blockers
|
||||
};
|
||||
|
||||
type CustomerOperationPlan = {
|
||||
customer_id: string;
|
||||
// Operations declared in execution order. Within a customer, ops execute serially.
|
||||
// Operations targeting the same Stripe subscription are batched by the executor
|
||||
// into a single subscriptions.update — this is the existing multi-update atomicity
|
||||
// boundary.
|
||||
operations: Operation[];
|
||||
guards: GuardResult[];
|
||||
preview: CustomerPreview; // human-readable diff of intended changes
|
||||
};
|
||||
|
||||
type Operation =
|
||||
| CatalogOperation
|
||||
| CustomerProductGraphOperation
|
||||
| CustomerEntitlementOperation
|
||||
| BalanceOperation
|
||||
| StripeRepairOperation
|
||||
| CacheInvalidationOperation;
|
||||
|
||||
// Mutation surface 1: catalog
|
||||
type CatalogOperation =
|
||||
| { type: "ensure_feature"; spec: FeatureSpec }
|
||||
| { type: "ensure_plan_item"; product_selector: ProductSelector; item: ProductItemSpec }
|
||||
| { type: "ensure_shared_price"; signature: string; spec: PriceSpec }
|
||||
| { type: "ensure_shared_entitlement"; signature: string; spec: EntitlementSpec }
|
||||
| { type: "link_stripe_resource"; price_id: string; stripe_price_id?: string; stripe_product_id?: string }
|
||||
| { type: "swap_archived_stripe_price"; price_id: string };
|
||||
|
||||
// Mutation surface 2: customer product graph (the big one)
|
||||
type CustomerProductGraphOperation =
|
||||
| { type: "attach"; assignment: TargetAssignment; ownership: Ownership }
|
||||
| { type: "replace"; subscription_id?: string;
|
||||
from: CusProductSelector[]; // expire these
|
||||
to: TargetAssignment[]; // insert these
|
||||
carry?: { balance: boolean; subscription_ids: boolean; anchors: boolean } }
|
||||
| { type: "expire"; targets: CusProductSelector[]; reason: string }
|
||||
| { type: "schedule"; phases: ScheduledPhase[] };
|
||||
|
||||
// Mutation surface 3: customer entitlement (B-layer in old proposal)
|
||||
type CustomerEntitlementOperation =
|
||||
| { type: "backfill_for_customers"; product_selector; feature_id; require_price?: boolean }
|
||||
| { type: "patch_flag"; selector; flags: Partial<Pick<Entitlement, "usage_allowed" | "carry_from_previous">> }
|
||||
| { type: "clamp_balance"; selector; ceiling: number };
|
||||
|
||||
// Mutation surface 4: balance (imports especially)
|
||||
type BalanceOperation =
|
||||
| { type: "seed_balance"; balance_id: string; cus_ent_selector; amount: number; entity_id?: string }
|
||||
| { type: "set_balance"; cus_ent_id: string; balance: number }
|
||||
| { type: "delete_stale_balance"; cus_ent_id: string; reason: string };
|
||||
|
||||
// Mutation surface 5: stripe repair (imports especially)
|
||||
type StripeRepairOperation =
|
||||
| { type: "link_subscription"; cus_product_id: string; stripe_subscription_id: string }
|
||||
| { type: "expire_orphaned_imported_cus_product"; cus_product_id: string };
|
||||
|
||||
type CacheInvalidationOperation =
|
||||
| { type: "invalidate_customer_cache"; customer_ids: string[] }
|
||||
| { type: "invalidate_products_cache" };
|
||||
```
|
||||
|
||||
### TargetAssignment + ProductItemPatch
|
||||
|
||||
The two key composite types.
|
||||
|
||||
```ts
|
||||
type TargetAssignment = {
|
||||
product: { id: string; version?: number };
|
||||
entity_id?: string | null;
|
||||
subscription_id?: string | null; // for imports linking to existing Stripe sub
|
||||
feature_quantities?: FeatureQuantity[];
|
||||
item_patches?: ProductItemPatch[]; // PATCH-style customize
|
||||
billing_dates?: { trial_ends_at?: number; reset_cycle_anchor?: number; billing_cycle_anchor?: number };
|
||||
ownership?: Ownership; // imported / user-created / migrated
|
||||
};
|
||||
|
||||
// PATCH-style. Compiles into FullProduct + customPrices + customEnts
|
||||
// that existing billing v2 compute already understands.
|
||||
type ProductItemPatch =
|
||||
| { op: "add_existing"; entitlement_id: string; price_id?: string } // reference shared catalog rows
|
||||
| { op: "replace_feature"; feature_id: string; entitlement_id: string; price_id?: string }
|
||||
| { op: "remove_feature"; feature_id: string }
|
||||
| { op: "override_allowance"; feature_id: string; allowance: number; unlimited?: boolean }
|
||||
| { op: "override_price"; feature_id?: string; price: PricePatch } // mints a custom Price row
|
||||
| { op: "set_options"; feature_id: string; options: { quantity?: number } };
|
||||
|
||||
type Ownership = {
|
||||
origin: "imported" | "user_created" | "migrated";
|
||||
source?: string; // e.g. "stripe_subscription:sub_abc"
|
||||
imported_at?: number;
|
||||
};
|
||||
```
|
||||
|
||||
`add_existing` is the load-bearing fix: it lets a 10k-customer migration reference one shared catalog row instead of minting 10k custom rows. **`is_custom` becomes an ownership flag (one-off vs catalog) decoupled from "uses custom prepared rows."**
|
||||
|
||||
### Execution model
|
||||
|
||||
```ts
|
||||
// Top-level entry point
|
||||
async function executePlan(plan: OperationPlan, opts: { dryRun: boolean }): Promise<PlanResult>
|
||||
|
||||
// Strict execution rules:
|
||||
// 1. If plan.blockers is non-empty AND !dryRun → throw, refuse.
|
||||
// 2. Catalog ops execute first, in declared order, before any customer ops.
|
||||
// 3. Customer plans execute in parallel (configurable concurrency); same-customer
|
||||
// operations execute serially in declared order.
|
||||
// 4. Within one customer, operations targeting the same Stripe subscription are
|
||||
// GROUPED by the CustomerProductGraphOperation executor and emitted as ONE
|
||||
// subscriptions.update via the existing evaluateStripeBillingPlan + executeBillingPlan
|
||||
// path. This preserves multi-update atomicity.
|
||||
// 5. Each operation returns OperationResult { status, diff?, error?, artifacts? }.
|
||||
// 6. Dry-run produces the same operations list; executors emit `would_apply` results
|
||||
// with diffs but mutate nothing.
|
||||
```
|
||||
|
||||
**The CustomerProductGraphOperation executor for `replace` is essentially today's `multi-update/`** generalized: build the per-product context list, call `computeCustomPlanNewCustomerProduct` per item, aggregate into `AutumnBillingPlan`, run through `evaluateStripeBillingPlan` + `executeBillingPlan`. Everything downstream of the Operation type is reused.
|
||||
|
||||
### Persistence
|
||||
|
||||
Plans persist to DB for non-trivial runs. Schema:
|
||||
|
||||
```
|
||||
migration_runs -- one per orchestrated run (org, env, source, status)
|
||||
migration_plans -- the OperationPlan JSON, snapshot at planning time
|
||||
migration_ops -- one row per operation, with status + diff + result
|
||||
migration_artifacts -- CSV exports, snapshots, audit rows
|
||||
```
|
||||
|
||||
Trivial cases (single-customer one-off) can skip persistence and go straight through `executePlan`. Bulk runs persist.
|
||||
|
||||
States: `draft → planned → applying → completed | failed | partial`. A `partial` failed run is resumable: the executor skips ops with `status: applied` and re-runs `pending` / `failed` ops.
|
||||
|
||||
### Planners
|
||||
|
||||
```ts
|
||||
type MigrationDefinition = {
|
||||
id: string;
|
||||
selectCandidates(ctx): Promise<CustomerScope[]>;
|
||||
buildTargetState(ctx, customer): Promise<TargetState>;
|
||||
plan(ctx, target): OperationPlan["customers"][number];
|
||||
};
|
||||
|
||||
type ImportDefinition = {
|
||||
id: string;
|
||||
loadExternalSource(ctx): Promise<SourceSnapshot>;
|
||||
resolveIdentities(ctx, source): Promise<IdentityMap>;
|
||||
mapTargets(ctx, source, identities): Promise<TargetState[]>;
|
||||
plan(ctx, target): OperationPlan["customers"][number];
|
||||
};
|
||||
```
|
||||
|
||||
Mintlify import becomes `ImportDefinition`. Mintlify migrate-credits becomes `MigrationDefinition`. Both emit the same `OperationPlan` shape.
|
||||
|
||||
---
|
||||
|
||||
## Phased build order
|
||||
|
||||
### v0 — Migration Core (3-4 weeks)
|
||||
**Concrete deliverable: port mintlify migrate-credits' multi-update path to use this system.** That single port validates 80% of the abstraction.
|
||||
|
||||
1. `Operation` type + `ProductItemPatch` compiler (`patches → FullProduct + customPrices + customEnts`).
|
||||
2. `CustomerProductGraphOperation.replace` executor wrapping the existing multi-update pipeline.
|
||||
3. `OperationResult` + `CustomerPreview` shapes (Autumn diff + Stripe sub-update preview).
|
||||
4. Guard set: `subscription-replace` (mismatched-sub-in-group, missing payment, anchor drift, collection_method preservation).
|
||||
5. `executePlan` for in-memory plans (no DB persistence yet).
|
||||
6. Integration tests: port the mintlify migrate-credits scenarios.
|
||||
7. Migration runner harness for scripts-v2 (one-line: `runPlan(planner, opts)`).
|
||||
|
||||
### v0.5 — Catalog Prep (1-2 weeks)
|
||||
8. `CatalogOperation` executor: `ensure_feature`, `ensure_plan_item`, `ensure_shared_price`, `ensure_shared_entitlement`, `link_stripe_resource`.
|
||||
9. Run-level catalog ops execute before per-customer ops.
|
||||
10. Port `retroactively-add-plan-item-to-versions` + `retroactively-add-plan-item-to-customers` to the new system (validates catalog → customer-entitlement composition).
|
||||
|
||||
### v0.75 — Persistence + Resume (1 week)
|
||||
11. `migration_runs` / `migration_plans` / `migration_ops` schema.
|
||||
12. `executePlan` writes operation results; partial-failure resume.
|
||||
13. Read-only view of past runs (no UI yet, just service methods).
|
||||
|
||||
### v1 — Auxiliary surfaces (2 weeks)
|
||||
14. `BalanceOperation` executor (Mintlify period_usage, Firecrawl coupon balances).
|
||||
15. `CustomerEntitlementOperation` executor (flag-flips, balance clamps, backfills).
|
||||
16. `StripeRepairOperation` executor.
|
||||
17. Port one full Firecrawl import slice as a smoke test.
|
||||
|
||||
### v1.5 — Imports as planner (2-3 weeks)
|
||||
18. `ImportDefinition` runner with identity resolution + source normalization stages.
|
||||
19. Port mintlify-import to use this. Firecrawl-import follows.
|
||||
|
||||
### v2 (deferred) — Dashboard + API
|
||||
20. Dashboard view: list runs, inspect plan, dry-run preview, approve & apply.
|
||||
21. Public API for self-serve migrations.
|
||||
|
||||
---
|
||||
|
||||
## Synthesis: what changed from v1 of this proposal
|
||||
|
||||
| v1 (PROPOSAL.md before) | v2 (now) | Why |
|
||||
|---|---|---|
|
||||
| Single `migrate` action with `PlanOp` discriminated union | Planner + executors with typed operations by mutation surface | Heterogeneous mutation surfaces don't belong in one action |
|
||||
| 3 layers (catalog / backfill / transition) | Flat operation graph with execution ordering rules | Layers were artificial; ops at the same level with declarative dependencies is cleaner |
|
||||
| `use_shared_rows` flag bolt-on | `ProductItemPatch.add_existing { entitlement_id, price_id }` first-class | Semantics are clearer when they're a discrete patch op |
|
||||
| `is_custom` overloaded for "one-off" + "uses custom rows" | Decouple: `is_custom` = ownership flag; shared rows = first-class catalog rows | Resolves a long-standing concept conflation |
|
||||
| Imports = migration + 5 extensions | Imports = different planner emitting same operation graph | Cleaner separation of concerns; identity/source/mapping is genuinely different from migration |
|
||||
| Action-level `preview` flag | First-class plan artifact; dry-run = identical plan path | Plan becomes inspectable, persistable, approvable |
|
||||
| No persistence story | `migration_runs/_plans/_ops` schema; resumable partial runs | User asked for "core feature, not internal tool" — first-class needs persistence |
|
||||
| No ownership concept | `Ownership` first-class on every TargetAssignment | Repair logic is currently bespoke per-script because we can't tell imported from user-created |
|
||||
|
||||
---
|
||||
|
||||
## Open design questions (the load-bearing ones)
|
||||
|
||||
1. **Plan persistence: opt-in or default-on?**
|
||||
Lean: **default-on for `bulk` planners (>1 customer); opt-out for one-shot operations.** Persistence enables resume, audit, dashboard surfacing — worth the schema cost.
|
||||
|
||||
2. **Operation ordering within a customer plan: declared or topologically derived?**
|
||||
Lean: **declared.** Planners produce ops in execution order. Simpler, and the planner already knows what depends on what (e.g. `attach` before `seed_balance`). No DAG solver needed.
|
||||
|
||||
3. **Subscription-group atomicity: explicit or implicit?**
|
||||
Lean: **implicit, in the executor.** The executor groups operations by Stripe sub before executing. This keeps the operation list flat and forgiving — caller doesn't have to think about Stripe's batching boundary, the executor handles it.
|
||||
|
||||
4. **Ownership flag: stored where?**
|
||||
Options: (a) new column on `customer_products`, (b) jsonb metadata, (c) separate `customer_product_ownership` table.
|
||||
Lean: **new column** — it's load-bearing for repair queries; jsonb is too soft.
|
||||
|
||||
5. **Existing `billingActions.migrate`: keep, deprecate, or absorb?**
|
||||
Need to read it. If it's a thin version-bump helper, keep as a shorthand that produces a one-op plan. If it's heavier, deprecate in favor of `MigrationDefinition`.
|
||||
|
||||
6. **Where does identity resolution live for imports?**
|
||||
In the `ImportDefinition.resolveIdentities` stage. The output is an `IdentityMap` that subsequent stages consume. Don't bake identity logic into Operations — keep operations identity-agnostic (they take `customer_id`, not "find or create customer with this email").
|
||||
|
||||
7. **Public-facing types: when?**
|
||||
Defer until v2 (dashboard). Internal types can move freely; public types lock when we expose this to users. Avoid premature stability commitments.
|
||||
|
||||
8. **`is_custom` decoupling: backwards compat?**
|
||||
Existing custom rows have `is_custom: true`. New "shared catalog rows referenced by patch" don't set this flag. We need a cleanup or co-existence story for current per-customer custom rows. Lean: leave existing rows alone; new code uses the new model; eventually a `null is_custom` migration cleans up.
|
||||
|
||||
---
|
||||
|
||||
## Open question deferred from v1
|
||||
|
||||
- **PATCH vs PUT customize:** resolved by `ProductItemPatch` discriminated union (PATCH-style, with `add_existing` for shared rows).
|
||||
- **Custom prices: per-customer rows vs reused per-plan rows:** resolved by `add_existing` / `ensure_shared_*` catalog ops.
|
||||
- **Preparation step placement:** resolved — it's `CatalogOperation` ops inside the same plan.
|
||||
35
.context/migration-interface/STATUS.md
Normal file
35
.context/migration-interface/STATUS.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# migration-interface
|
||||
|
||||
## Goal
|
||||
Design a shared billing-action interface for customer migrations and imports so scripts stop hand-rolling Stripe + Autumn DB orchestration. Eventually exposed to users on the dashboard, so it must be a first-class product feature with tests — not an internal tool.
|
||||
|
||||
## Current Phase
|
||||
Phase 1: Discovery complete (catalog written). PROPOSAL.md v2 drafted — planner + executor model with typed operations. Awaiting user review of architecture and 8 open design questions.
|
||||
|
||||
## Branch
|
||||
john/migration-interface
|
||||
|
||||
## Active Tasks
|
||||
- (none yet — will create parallel cataloging tasks if Phase 1 surface gets large)
|
||||
|
||||
## What's Done
|
||||
- Phase 1 discovery: cataloged `scripts/src/common/migrations/` (23 files), `scripts-v2/runs/` migration scripts, mintlify+firecrawl import chains, and billing v2 action machinery (`multiAttach`, `createSchedule`, `updateSubscription`, `evaluateStripeBillingPlan`, `AutumnBillingPlan`, `customizePlanV1`, `setupCustomFullProduct`).
|
||||
- See `CATALOG.md` for full pattern inventory and `PROPOSAL.md` for the layered-interface design.
|
||||
|
||||
## What's Next
|
||||
- Walk through `PROPOSAL.md` with user — confirm 3-layer model and discuss the 7 open design questions
|
||||
- Read existing `actions/migrate/migrate.ts` to decide extend-vs-replace
|
||||
- Lock Layer 3 scope (include `patch` op? or split into `migrate` + extended `updateSubscription`?)
|
||||
- Begin Phase 2: design `MigrateParams`, `MigrateBillingContext`, `computeMigratePlan` signatures concretely
|
||||
|
||||
## Active Blockers
|
||||
- None
|
||||
|
||||
## Key Files
|
||||
- `server/src/internal/billing/v2/actions/index.ts` — where new action(s) will register
|
||||
- `server/src/internal/billing/v2/actions/multiAttach/multiAttach.ts` — closest existing analog
|
||||
- `server/src/internal/billing/v2/actions/createSchedule/createSchedule.ts` — schedule pattern reference
|
||||
- `server/src/internal/billing/v2/actions/updateSubscription/updateSubscription.ts` — single-subscription mutation reference
|
||||
- `server/src/internal/billing/v2/providers/stripe/actionBuilders/evaluateStripeBillingPlan.ts` — Autumn plan → Stripe state mapper
|
||||
- `shared/api/billing/common/customizePlan/customizePlanV1.ts` — public customize schema
|
||||
- `shared/api/products/items/apiPlanItemV1.ts` — public plan item shape
|
||||
Reference in New Issue
Block a user