Opening any person page on a large tree took 4-5s on an idle server. Root cause:
list_persons looped over every person calling privacy.person_visibility (which
issues TWO get_membership_role queries per call) AND _attach_primary_name (one
name query per person). On the reporter's 2,324-person tree that's ~7,000
serialized DB round-trips per page load — the person page fetches the full
person list to build its name-lookup map.
Fix:
- Resolve the viewer's membership role ONCE. Members see the whole tree (full),
so skip the per-person privacy engine entirely.
- Add _attach_primary_names: one batched names query (person_id IN (...),
ordered the same as the single-person query so it picks the same name) instead
of one per person.
- Apply the same batching to the non-member path, search_persons, the deleted-
persons list, and public_view_service.list_public_persons.
Member-path list_persons goes from ~3·N queries to ~3 total. Other tree-wide
list endpoints (events/relationships/media/citations) were already flat selects.
Adds a regression test that asserts list_persons issues a constant number of
queries (not proportional to person count). Suite: 103 passing.
Signed-off-by: Justin Paul <justin@jpaul.me>
Two changes.
1. Privacy fix (NN#2/NN#3) — the citation and source list endpoints gated only
on can_view_tree, so a non-member on a public/unlisted/site_members tree could
enumerate citations and sources tied to a redacted living person, leaking that
the person exists and has sourced facts (and possibly their name via a source
title). #46 closed this for events/media/names/relationships but not
citations/sources. Now citation_service.list_citations and
source_service.{list_sources,get_source} delegate non-member reads to
public_view_service, mirroring the #46 pattern:
- citations: shown only when the cited fact resolves to FULL-visibility
person(s) — covers the person_id, name_id, event_id (person or both-partner),
and relationship_id (both-partner) target paths.
- sources: shown only when they back at least one visible citation; a withheld
source 404s (don't reveal it exists).
Tests cover all four citation target types + source withholding + member-sees-all.
2. On-demand tree purge — owners can permanently delete a soft-deleted tree now
instead of waiting out the 30-day auto-purge window. POST /trees/{id}/purge
(owner-only): the tree must already be in the trash, and the caller retypes its
name to confirm. Media objects are deleted from storage, then a single
DELETE on trees cascades all tree-owned rows via the tree_id ON DELETE CASCADE;
the audit entry survives (tree_id SET NULL). Frontend adds a "Delete forever"
button to the Recently-deleted list. No migration.
Suite: 102 passing.
Signed-off-by: Justin Paul <justin@jpaul.me>
The operator decides which model providers exist (env / registry — Anthropic,
OpenAI, x.AI, Ollama, several at once). The *tree owner* decides who uses which:
- Members' assistant -> one configured provider (or none)
- Recommender (association/connection finder) -> one configured provider (or none)
- Owner -> may use any configured provider
Backend: two nullable columns on `trees` (ai_member_provider,
ai_recommender_provider) + migration; `configured_llm_providers()` exposes the
registry as {name, model} with no secrets; owner-gated GET/PATCH
/trees/{id}/ai validate names against the configured set. Frontend: owner-only
"AI models" page with a dropdown per role, graceful 403 for non-owners, and a
sidebar link.
Per-model-within-a-provider selection is a follow-up; today each provider maps
to its single configured model.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Implements non-negotiable #1: the AI assistant never writes autonomously. Every
assistant/contributor "write" emits a ChangeProposal — a structured diff a human
approves, edits, or rejects. Design: docs/design/change-proposal.md.
Structural guarantee: a proposal's operations reach the DB ONLY via
change_proposal_service.apply(), which requires the actor be an editor and
dispatches each op through the normal editing services (person/name/event/
relationship/source/citation create/update/delete) — so every change passes the
privacy engine and is audited as the approving human. propose() only inserts a
pending row; it performs no domain mutation. Model providers stay read-only, so
no model response can mutate tree data.
- ChangeProposal model + migration (status pending|applied|rejected, origin
assistant|contributor, JSONB operations, reviewer + apply_error).
- Service: propose / list / get / apply (with optional edited ops) / reject /
delete; a dispatcher mapping ops → editing services. v1 applies ops in order,
not cross-op transactional (single-op is atomic; documented).
- API /trees/{id}/proposals + a frontend review page (approve/reject; editor-
gated) and sidebar entry.
Tests: proposal doesn't apply until approved; reject doesn't apply; non-editor
member can see but not apply; multi-op; approve-with-edits; apply-error keeps it
pending. Full suite 87 passed; single alembic head.
Closes#214
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
TreeMembership was enforced on every read/write but had no API or UI to manage
members — trees were effectively single-user, breaking full-CRUD (NN#8).
Backend (/trees/{id}/members): list (members only — the list exposes emails, so
non-members never see it, even on public trees); add an existing user by email
(owner only, 404 if no such account, 409 if already a member); PATCH role;
DELETE. A tree must always keep ≥1 owner (demote/remove of the sole owner → 409).
All changes audited.
Frontend: a Members page (owner gets add-by-email + per-member role select +
remove; others see a read-only list) and a sidebar entry.
Test covers the full lifecycle + every guard. Suite 77 passed.
Closes#145
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Export emitted SOUR records but never the per-fact SOUR links, so a
Provenance→Provenance round-trip destroyed the sources graph (citations were
dropped). Emit citation links on the facts they sit on:
- person-level → 1 SOUR @Sx@ (2 PAGE)
- name-level → 2 SOUR under 1 NAME
- event-level → 2 SOUR under the event (incl. partnership events in FAM)
- relationship → 1 SOUR under FAM
Citations whose source didn't export are skipped.
Test: a person + event citation round-trips through export→import into a fresh
tree with their pages intact. GEDCOM suite 6 passed.
Closes#169
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Backlog §2.10: registration issued a live session and email_verified_at was
written but never read, so an unverified user had full access and there was no
switch to require verification.
Add REQUIRE_EMAIL_VERIFICATION (default false). When true:
- resolve_session_user returns None for a user whose email_verified_at is null —
the single read-side gate covering every authenticated request, incl. the
session minted at registration.
- login raises 403 ("email not verified") instead of issuing a useless token.
Default false on purpose: self-hosts without SMTP, and accounts created before
this gate existed (email_verified_at null), must not be locked out. Operators
enable it once mail works and accounts are verified. Documented in .env.example.
Tests: default-off keeps unverified accounts working; on → register's session
won't resolve (401), login is 403, and after verify-email both work. 75 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Unset sex renders blue (male-colored), which is misleading next to a confirmed
male partner. Add a Cleanup action that proposes the opposite sex for an unset
partner of someone whose sex is set (couples are opposite-sex in practice — a
confirmed-male husband ⇒ a female wife). People whose known partners disagree
are skipped as ambiguous.
It's a preview the user reviews and approves in the Cleanup tool (reusing the
existing gender apply path + audit) — not an autonomous write. Backend:
guess_gender_by_spouse + GET /cleanup/gender/from-spouse. Frontend: an "Infer
from spouse" button feeding the existing proposal list. Test covers
propose-opposite, skip-no-partner, skip-already-set, apply, and re-preview.
Full suite 73 passed; frontend build clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
A logged-in NON-member of a public/unlisted tree could read living people's
dates, real alternate names, and media (incl. downloading photos) through the
family-view endpoints — only the person LIST was redacted; list_events,
list_relationships, list_names, list_media gated on can_view_tree alone.
For non-members, these now delegate to the same visibility-filtered reads the
public surface uses (person_visibility-driven): living-person events/names
dropped, relationships touching a hidden person dropped, media limited to
full-visibility persons, and media download (get_media → media_content) 404s
for a redacted/unlinked person's media. Members are unchanged.
Adds list_public_relationships_for_person / list_public_media / can_view_media
to public_view_service. Test: an authed non-member sees no living-person PII
across events/names/relationships/media and can't download a living person's
file, while the owner still sees everything. Full suite: 72 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Adds the anonymous read surface (/api/v1/public) — the privacy-critical core.
- CurrentUserOrNone dependency: optional auth that never 401s (anonymous OK).
- public_view_service: every projection passes through privacy.person_visibility.
persons redacted (living → "Living person", hidden dropped); relationships
only when both endpoints non-hidden; events only for FULL-visibility persons
(partnership events only when both partners full); names only for FULL
persons. Not-viewable trees raise 404 (not 403) so the surface can't probe
for private trees. Media deferred (higher-sensitivity; own pass later).
- public router: read-only directory + tree + persons/relationships/events +
person detail/names/events. Directory lists `public` to all and adds
`site_members` for authenticated callers; never lists unlisted/private.
- PublicTreeRead omits owner_id.
Tests (ran locally — CI does not run pytest): an anonymous end-to-end leak test
asserting a living person's real name, alias, and birth year appear in NO public
response while the deceased person's data does; plus private=404, unlisted
viewable-by-link-but-unlisted, site_members requires login, and directory
visibility. Full suite: 70 passed. Regenerated openapi.json + TS client.
Note: the AUTHED list endpoints still leak per-person for non-members
(pre-existing) — fixed next, separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
can_view_tree() now distinguishes anonymous vs authenticated non-members so the
four-level model is enforceable:
- public / unlisted → anyone, including anonymous (unlisted gated only by the
link, so the API must never *list* it)
- site_members → any authenticated account (denies anonymous)
- private → members only
Members (any role) always view; soft-deleted trees stay hidden from everyone.
person_visibility (living-person redaction) is unchanged.
Tests: a full can_view_tree matrix across {anonymous, logged-in non-member,
member} × {public, unlisted, site_members, private}, plus deleted-tree-hidden
and the site_members anon-vs-logged-in case. Adds `engine`/`db_session` fixtures
(refactored out of `client`) so the engine can be unit-tested directly,
including the anonymous path that has no HTTP endpoint yet.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Root cause of the blank Jung tree: a child double-linked to the same parent
(and, generally, any cycle) made family-chart recurse forever.
Backend (the real fix):
- create_relationship now rejects an equivalent existing edge → 409.
parent_child is directional (parent→child); partnership/sibling match the
pair in either order. So you can't link the same two people the same way
twice. (GEDCOM import already deduped; manual creates didn't.)
Frontend (defense in depth so data can never blank the view):
- Tree view sanitizes the graph before rendering: dedupes parents/spouses,
drops self-links, and greedily breaks ancestor cycles (a person can't be
their own ancestor); children are derived from the kept edges. The render is
wrapped in try/catch and shows a note instead of a blank canvas, telling you
which conflicting links were skipped.
- Person page surfaces the 409 ("They're already linked that way.").
59 backend tests pass (incl. dup-rejection + reverse-parent-child allowed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A "Guess from first name" option in the Cleanup gender section: a bundled,
curated given-name -> sex dictionary (weighted English + German for the first
real tree) proposes sex for people who don't have it set. Deterministic, offline,
no model. Genuinely ambiguous names (Marion, Frances, Jordan, …) are excluded
from both sets so they're left for a human. Reuses the existing preview/apply
gender flow, so every guess is reviewed before saving.
No migration. 56 backend tests pass; frontend builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A new per-tree Cleanup page (and cleanup_service + endpoints), each fix
preview-first per the propose-then-approve rule:
- Mark deceased by birth year: lists people born ≤ a cutoff (default 1930) not
already deceased; apply sets is_living=false for the ones you keep checked.
- Set sex from a source GEDCOM: upload the source .ged (it carries SEX); matches
by name and proposes sex only where it's missing — far more accurate than
guessing from first names. Review, then apply.
- Names that look broken: flags date-in-surname / date-in-given / no-surname /
packed given names, with inline editable given+surname; fix the checked ones.
No migration (uses existing columns). 55 backend tests pass (preview+apply for
all three); frontend builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New account_service + endpoints under /users/me:
- GET /me/export — zip of every owned tree (account.json + media blobs).
- POST /me/import — restore a backup into NEW trees (ids remapped, media
re-uploaded); non-destructive, never touches existing data.
- DELETE /me — soft-delete the user, their owned trees, and revoke sessions;
guarded by retyping the account email.
Settings page wires all three (export download, restore upload, delete with
typed-email confirmation). No migration — uses existing tables + soft-delete.
52 backend tests pass (export→restore round-trip + delete guards); frontend builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Sidebar bottom-left now shows the signed-in user; clicking opens a menu with
Settings and Sign out. New /settings page: account info + change password
(POST /auth/change-password, re-verifies current password). Export/restore/
delete are stubbed there for the next pass.
- Per-tree default/home person: tree.home_person_id (migration) + TreeUpdate/
Read; the tree and family views open focused on it; the person page gets a
"Set as default" control and "Default person" badge. Cleared if that person
is deleted. Complements the account-level "this is me" link.
- Tree visualization now fills the content area (AppShell drops the max-width
column on the /tree route); other pages stay centered.
- Audit records are coerced JSON-safe (UUIDs/enums), so PATCHing UUID fields
like home_person_id audits cleanly.
50 backend tests pass; migration up/down verified; frontend builds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Duplicate detection (the "merge / skip / overwrite" the user asked for):
- New POST /gedcom/preview dry-runs the file and flags incoming people that
resemble existing ones (name similarity via difflib + birth-year guard;
high/medium score). No writes.
- /gedcom/import takes default_action (new|skip|merge|overwrite) + per-xref
resolutions {xref: {action, target_id}}:
new create as a new person (current behavior)
skip link families to the existing person, copy nothing
merge attach the incoming names (as alternates), events, citations,
and notes onto the existing person
overwrite soft-delete the existing person, import the incoming one fresh
Relationship creation is deduped so a merge can't double an edge.
Richer record mapping (covers the user's repo's GEDCOM):
- Multiple NAME records honor their TYPE; _MARNM (and NICK) import as typed
alternate names — maiden stays primary, married becomes a "married" Name.
- RELI -> a "religion" event with the value in detail; OCCU/EDUC values too.
- NOTE -> person notes (and event notes); NOTE/RELI are no longer "unmapped".
- Export round-trips name TYPE.
Verified against the user's 2185-person export: 0 unmapped tags. 48 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Names (the genealogy standard: maiden name primary, married/alias as typed
alternates):
- Name model already supported multiple typed names; expose full CRUD —
NameCreate/Read/Update schemas, name_service (one-primary invariant,
promote-on-delete), nested /persons/{id}/names routes.
- Person page gains a Names card: add/edit/delete + "make primary", with a
curated name_type dropdown (birth/maiden, married, alias, nickname, …).
Self-person ("who am I"):
- users.self_person_id FK (use_alter for the users<->persons<->trees cycle)
+ migration; PATCH /users/me/self-person; "This is me" / "This is you"
on the person page. Soft-deleting the linked person clears it.
Deletion integrity (fixes the broken tree view):
- delete_person now soft-deletes the relationships touching the person, so no
dangling edges remain; family-chart also filters links to missing people.
- Optional cascade=true recursively deletes descendants (GEDCOM cleanup);
the person page asks "only this person" vs "with all descendants".
- DELETE returns {deleted: n}.
Family view surfaces "Not connected to anyone" so dangling people aren't lost.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes the rule #8 gap at the API layer: PATCH endpoints + service updates for Tree (name/description/visibility), Source, Citation (page/detail/confidence), Relationship (qualifier/notes), and Media (title/attachment) — editor-gated and audited. Every core entity now has create/read/update/delete. Edit UIs for these land in the frontend batch. 37 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Events and people are now editable, not write-once: PATCH /events/{id} (type, structured date, place, notes) and PATCH /persons/{id} (vitals, privacy, and the primary name's given/surname). CLAUDE.md gains rule #8: every stored object must support full CRUD in API and UI — historical research is constant correction. Tests cover both updates.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Fuzzy search: pg_trgm extension + trigram GIN indexes on name parts and a GET /trees/{id}/persons?q= search ranked by trigram similarity (finds Mueller for 'muller'), privacy-filtered. Living-person protection: the privacy engine now derives possibly-living status (explicit flag, else no death fact + birth within ~100y or unknown) and returns 'redacted' for non-members of public/unlisted trees; the service minimises those records ('Living person', no vitals). Members are unaffected. 31 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
A pragmatic GEDCOM parser + mapper: import reads INDI/FAM/SOUR and creates people, names, life events, partnership + qualified parent-child relationships, marriage events, places (deduped), sources, and citations from SOUR refs — returning a mapping report (counts + unmapped tags). Export serializes the tree back to GEDCOM (families derived from the edge model). Import is additive (no merge) and runs inline for now. Round-trip test passes; 29 tests total.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Tree and person soft-delete + restore (owner-only for trees, editor for people) with recovery listings (?deleted=true); the worker already purges past the 30-day window. Adds tree-wide GET /relationships and /events so the family/pedigree view loads the whole graph in a few calls. 27 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Presigned URLs point at the internal minio:9000 host a browser can't reach. Add ObjectStore.get_object and a GET /media/{id}/content endpoint that resolves visibility and streams the bytes; MediaRead.url now points there. Keeps the object store private and downloads behind the privacy engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Media model + migration; an ObjectStore interface with an S3/MinIO (boto3) implementation behind the service layer. Upload (multipart) stores bytes in object storage + a metadata row (checksum, size, content-type, optional attach to person/event/source); list returns presigned URLs; delete is soft. Editor-gated, privacy-filtered, audited. 24 tests pass (object store faked).
Introduces the worker container (same image, 'python -m app.worker'): its first job is the scheduled 30-day soft-delete purge across tables + media object cleanup. Compose gains worker + S3 env on backend/worker; dev override builds the worker too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Source CRUD (reusable, tree-scoped) and Citation create/list/soft-delete linking one source to exactly one fact (person/event/name/relationship). Editor-gated writes, privacy-filtered reads, audit throughout; tenant + existence validation on source and target. list_citations returns all tree citations so the UI can render 'sourced' indicators in one round-trip. 22 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Events (create/list-per-person/soft-delete) and relationships (create/list-per-person/soft-delete) through the layered stack: editor-gated writes, privacy-engine reads, audit on every change. Events carry exactly one subject (person XOR partnership); relationships are typed qualified edges (parent_child gets a biological/adoptive/step/foster/donor/guardian qualifier). Adds a single-person GET. 18 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Pluggable AuthProvider interface with a local (email+password) implementation, and a Mailer interface (ConsoleMailer for dev, SMTPMailer for operators). The auth service owns registration, login, opaque session issuance, email verification, and password reset (which revokes prior sessions). Endpoints under /api/v1/auth; sessions are returned as a Bearer token and set as an HttpOnly cookie.
Replaces the temporary X-User-Id shim: get_current_user now resolves a real session (Bearer or cookie). The open user-bootstrap endpoint is gone (registration replaces it). App logging is configured so the ConsoleMailer's verification/reset links are visible to self-hosters. Verified end-to-end on the deploy target, including the email-verification flow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
Wires the data model through repository -> service -> API/v1. The privacy engine (app/services/privacy.py) is the single enforcement point: every read resolves visibility there (tree role, tree visibility, per-person override; living-person redaction is a marked Phase 2 TODO). All writes record an attributable AuditEntry.
Endpoints: POST /users (open dev bootstrap until auth), GET /users/me, POST/GET /trees, GET /trees/{id}, and POST/GET /trees/{id}/persons. Authn is a temporary X-User-Id header shim; authz is membership-based (owner/editor/viewer). Domain errors map to 401/403/404/409. Verified on the deploy target: private tree -> 403 for non-members, missing actor -> 401, audit log populated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Justin Paul <justin@jpaul.me>
justin