Merge pull request 'refactor(catalog): make embedded template validation generic' (#42) from codex/block-15-generic-catalog into main

Reviewed-on: #42
Reviewed-by: tony <1+tony@noreply.localhost>

catalog/palworld/README.md

@@ -5,12 +5,12 @@ This template demonstrates every important V1 contract: official image, public U
 ## Verified upstream facts
 
 - Pocketpair publishes the official image and example Compose deployment at [pocketpairjp/palworld-dedicated-server-docker](https://github.com/pocketpairjp/palworld-dedicated-server-docker).
-- The current official example at specification time uses `ghcr.io/pocketpairjp/palserver:v1.0.2.101103`, UDP 8211, `/pal/Package/Pal/Saved` and the packaged `helper.sh` pattern.
+- Pocketpair publishes versioned image tags and a `latest` tag for the current official `ghcr.io/pocketpairjp/palserver` release. The official template uses `latest` for normal pulls and new deployments, with UDP 8211, `/pal/Package/Pal/Saved` and the packaged `helper.sh` pattern.
 - The [official requirements](https://docs.palworldgame.com/0.7.3/getting-started/requirements/) specify four or more CPU cores, 16 GB memory with over 32 GB recommended for stability, and UDP 8211. The template's 8 GB minimum is an explicit lower bootable boundary mentioned upstream, not a stability recommendation; the UI must warn below 16 GB.
 - The [configuration reference](https://docs.palworldgame.com/settings-and-operation/configuration/) documents `AdminPassword`, `RESTAPIEnabled`, `RESTAPIPort`, server name/password and maximum players.
 - The [REST API documentation](https://docs.palworldgame.com/category/rest-api/) documents information, players, metrics, announce, save, shutdown and moderation operations. REST API credentials and port must remain private.
 
-The image tag and upstream API may change. Catalog maintainers must verify and release a new immutable template version; existing instances remain pinned.
+The upstream API may change. A template snapshot version remains independent from the game-server image tag; existing instances remain pinned to their chosen configuration and DoGaMa does not automatically replace running containers.
 
 ## Reference limitations
 
@@ -18,4 +18,3 @@ The image tag and upstream API may change. Catalog maintainers must verify and r
 - The schema's storage sizes are conservative product defaults because upstream specifies SSD performance but not a fixed disk-size requirement.
 - Local hosted-world migration may require player identity conversion. The generic V1 importer detects structure and warns; it does not silently convert identities.
 - The checked-in module manifest is a source example. It becomes installable only after `module.wasm` is built and its real SHA-256 replaces the all-zero placeholder.
-

catalog/palworld/template.yaml

@@ -1,6 +1,6 @@
 schema_version: 1
 id: palworld-official
-version: 1.1.1
+version: 1.1.2
 
 source:
   type: official
@@ -29,7 +29,7 @@ requirements:
 
 container:
   image: ghcr.io/pocketpairjp/palserver
-  tag: v1.0.2.101103
+  tag: latest
   entrypoint:
     - /pal/helper.sh
   user_mode: image
@@ -41,7 +41,6 @@ container:
   assets:
     - source: assets/helper.sh
       destination: /pal/helper.sh
-      sha256: 52e58fe4e22654d0d312fe2bb6195db61dad5ffce78e0440a951d33d20f2c36f
       read_only: true
   stop_timeout_seconds: 120
   ports:

docs/PROJECT-STATE.md

@@ -58,6 +58,7 @@ Read this compact operational baseline before starting a milestone. Open detaile
 - Label values support only the explicit allowlist in `internal/instance/container_config.go`; unknown variables are errors, not arbitrary templates.
 - `{{game.icon_url}}` is the public icon for the game. `{{instance.slug}}` remains supported.
 - Template game artwork contains separate required local `logo` and horizontal `image` assets; template validation rejects missing files. Deployment previews expose distinct logo and artwork URLs while retaining `icon_url` as a compatible logo alias. Palworld template `1.1.0` is the first snapshot with this contract.
+- Embedded catalog validation is collection-based: every discovered template is schema- and cross-field-validated, including referenced assets, declared integration modules and ports/configuration. Template asset contents are not SHA-256-pinned, allowing administrator-maintained local assets while preserving required-path validation. A template `version` identifies a DoGaMa snapshot, whereas `container.tag` selects the game-server image; the official Palworld template follows Pocketpair's `latest` tag for new deployments and ordinary pulls.
 - Instance slugs are derived from the display name, not canonical IDs. Accents are normalized to ASCII; whitespace, `/`, punctuation and special characters become safe hyphen separators; repeated and edge hyphens are removed.
 - Docker user mode is fixed at creation to DoGaMa UID/GID, custom numeric UID/GID, or image-defined user. Never perform automatic recursive ownership changes.
 - A pinned image tag is an explicit mutable tag, not an immutable digest. Tracked mode follows the template's declared default tag.

docs/contributing/development.md

@@ -133,8 +133,8 @@ before the restricted agent creates the first container. Repeated installation
 submission recognizes an already attached import instead of copying it twice.
 
 At main-application startup, every embedded `catalog/*/template.yaml` is
-validated against `specs/template.schema.json`, checked for cross-reference and
-asset integrity, canonicalized deterministically and synchronized into SQLite.
+validated against `specs/template.schema.json`, checked for cross-references and
+referenced asset presence, canonicalized deterministically and synchronized into SQLite.
 An existing template ID/version is immutable: changing its digest fails startup
 instead of silently replacing the snapshot. Deployment previews pin that digest
 and redact secret defaults before a draft instance can enter the registry.

docs/operations/local-templates.md

@@ -19,7 +19,11 @@ Add or edit files on the Docker host, then sign in as an administrator and press
 
 ## Template format
 
-Templates use schema version `1` and are strict YAML documents. Existing deployment fields remain required: container image/tag, ports, storage mounts, configuration fields, capabilities, backup, healthcheck, imports, updates and compatibility. DoGaMa rejects unknown fields and unsafe paths, asset checksums, invalid configuration fields and invalid Docker-related declarations; a template cannot grant arbitrary Docker access.
+Templates use schema version `1` and are strict YAML documents. Existing deployment fields remain required: container image/tag, ports, storage mounts, configuration fields, capabilities, backup, healthcheck, imports, updates and compatibility. DoGaMa rejects unknown fields, unsafe paths, missing referenced assets, invalid configuration fields and invalid Docker-related declarations; a template cannot grant arbitrary Docker access. Asset contents are deliberately not checksum-pinned, so an administrator can maintain a local helper or configuration asset without invalidating an otherwise valid template.
+
+For local-template upgrade compatibility, an existing asset `sha256` key is accepted and ignored. New templates should omit it.
+
+`version` identifies the immutable DoGaMa template snapshot. It is not the game-server release. The server image release is selected separately by `container.tag`; a floating tag follows that image publisher's tag policy for newly created or normally pulled instances.
 
 The catalog information is under `game` and `requirements`:
 
@@ -32,17 +36,17 @@ game:
   id: example-game
   name: Example Game
   description: Concise dedicated-server description.
-  image: https://example.invalid/cover.jpg # vertical public cover URL
   artwork:
-    logo: assets/icon.png                  # retained deployment artwork asset
+    logo: assets/icon.png
     image: assets/banner.jpg
+    poster: assets/poster.jpg
     attribution: Your attribution text
 requirements:
   minimum: { cpu_cores: 2, memory_mb: 4096, storage_gb: 20, other: ["Network connection"] }
   recommended: { cpu_cores: 4, memory_mb: 8192, storage_gb: 40 }
 ```
 
-`game.image` is the vertical cover displayed by Catalog and the game page. Its URL must be HTTPS, but validation never fetches it: an unavailable remote cover does not block a scan and the interface has a graphical fallback. `requirements.minimum` and `requirements.recommended` contain generic CPU, memory and storage values plus optional `other` lines. Recommended resources cannot be below minimum resources.
+`game.artwork` contains required local logo, horizontal image and poster assets. Validation never fetches remote artwork, so a scan remains local and deterministic. `requirements.minimum` and `requirements.recommended` contain generic CPU, memory and storage values plus optional `other` lines. Recommended resources cannot be below minimum resources.
 
 `configuration.fields` drives the deployment form and the effective server configuration. Supported types are `string`, `integer`, `number`, `boolean`, `enum` and `secret`; defaults, required flags, numeric bounds, regular expressions and enum values are checked again by the server. Secret inputs are write-only, encrypted in the instance secret store when retained for runtime use, and never included in a persisted preview, API response, audit event or error.
 

internal/agent/agent_assets.go

@@ -1,8 +1,6 @@
 package agent
 
 import (
-	"crypto/sha256"
-	"encoding/hex"
 	"errors"
 	"fmt"
 	"os"
@@ -22,12 +20,10 @@ func (s *service) prepareAssets(plan agentwire.DeploymentPlan) ([]AssetMount, er
 		return nil, errors.New("asset root is not allowed")
 	}
 	result := make([]AssetMount, 0, len(approved))
-	for _, asset := range approved {
-		digest := sha256.Sum256(asset.Content)
-		if hex.EncodeToString(digest[:]) != asset.SHA256 {
-			return nil, errors.New("approved asset integrity check failed")
-		}
-		target := filepath.Join(assetRoot, asset.SHA256)
+	for index, asset := range approved {
+		// The template snapshot digest keeps agent-owned asset paths isolated
+		// without making an asset's declared content immutable.
+		target := filepath.Join(assetRoot, plan.TemplateDigest, fmt.Sprintf("asset-%d", index))
 		if err := writeImmutableAsset(target, asset.Content); err != nil {
 			return nil, err
 		}
@@ -41,18 +37,13 @@ func writeImmutableAsset(path string, content []byte) error {
 		if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 {
 			return errors.New("approved asset path is not a regular file")
 		}
-		existing, err := os.ReadFile(path)
-		if err != nil {
-			return errors.New("read approved asset")
-		}
-		existingDigest, wantedDigest := sha256.Sum256(existing), sha256.Sum256(content)
-		if existingDigest != wantedDigest {
-			return errors.New("approved asset content conflict")
-		}
 		return os.Chmod(path, 0o555)
 	} else if !errors.Is(err, os.ErrNotExist) {
 		return errors.New("inspect approved asset path")
 	}
+	if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+		return errors.New("create approved asset directory")
+	}
 	temporary, err := os.CreateTemp(filepath.Dir(path), ".asset-*")
 	if err != nil {
 		return fmt.Errorf("create approved asset: %w", err)

internal/agent/agent_plan.go

@@ -40,7 +40,6 @@ func NewPlanPolicy(snapshots []catalog.Snapshot, assets fs.FS) (*PlanPolicy, err
 
 type ApprovedAsset struct {
 	Destination string
-	SHA256      string
 	Content     []byte
 }
 
@@ -61,7 +60,7 @@ func (p *PlanPolicy) Assets(plan agentwire.DeploymentPlan) ([]ApprovedAsset, err
 		if err != nil {
 			return nil, fmt.Errorf("read approved template asset: %w", err)
 		}
-		result = append(result, ApprovedAsset{Destination: asset.Destination, SHA256: asset.SHA256, Content: content})
+		result = append(result, ApprovedAsset{Destination: asset.Destination, Content: content})
 	}
 	return result, nil
 }

internal/agent/agent_plan_test.go

@@ -2,6 +2,7 @@ package agent
 
 import (
 	"path/filepath"
+	"strings"
 	"testing"
 
 	catalogdata "git.zaynet.fr/DoGaMa/DoGaMa-serv/catalog"
@@ -9,73 +10,52 @@ import (
 	"git.zaynet.fr/DoGaMa/DoGaMa-serv/internal/instance"
 )
 
-func TestEmbeddedPalworldSnapshotMatchesApplicationDeploymentPlan(t *testing.T) {
+func TestEmbeddedSnapshotsMatchApplicationDeploymentPlans(t *testing.T) {
 	snapshots, err := catalog.LoadFS(catalogdata.Files, ".")
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	var snapshot *catalog.Snapshot
-	for i := range snapshots {
-		if snapshots[i].Template.ID == "palworld-official" {
-			snapshot = &snapshots[i]
-			break
-		}
-	}
-
-	if snapshot == nil {
-		t.Fatal(`embedded template "palworld-official" not found`)
-	}
-
-	if snapshot.Template.Version != "1.1.1" {
-		t.Fatalf(
-			"embedded Palworld snapshot = %s@%s",
-			snapshot.Template.ID,
-			snapshot.Template.Version,
-		)
-	}
-
-	preview, err := instance.BuildPreview(*snapshot, instance.PreviewRequest{
-		DisplayName: "Snapshot consistency",
-		Slug:        "snapshot-consistency",
-		HostPorts: map[string]int{
-			"game": 38211,
-		},
-		MountPaths: map[string]string{
-			"saved": filepath.Join(t.TempDir(), "saved"),
-		},
-		DataOrigin:      "new",
-		BackupRetention: 7,
-	})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	plan, err := preview.DeploymentPlan("abcdefghijklmnopqrstuvwx")
-	if err != nil {
-		t.Fatal(err)
-	}
-
 	policy, err := NewPlanPolicy(snapshots, catalogdata.Files)
 	if err != nil {
 		t.Fatal(err)
 	}
-
-	known, ok := policy.snapshots[plan.TemplateID+"@"+plan.TemplateVersion]
-	if !ok || known.Digest != plan.TemplateDigest {
-		t.Fatalf(
-			"agent snapshot=%#v plan=%s@%s digest=%s",
-			known,
-			plan.TemplateID,
-			plan.TemplateVersion,
-			plan.TemplateDigest,
-		)
-	}
-
-	if err := policy.Validate(plan); err != nil {
-		t.Fatalf(
-			"application plan rejected by matching embedded snapshot: %v",
-			err,
-		)
+	for index, snapshot := range snapshots {
+		t.Run(snapshot.Template.ID, func(t *testing.T) {
+			hostPorts := map[string]int{}
+			for portIndex, port := range snapshot.Template.Container.Ports {
+				if port.Publish {
+					hostPorts[port.ID] = 38000 + index*100 + portIndex
+				}
+			}
+			mountPaths := map[string]string{}
+			for _, mount := range snapshot.Template.Storage.Mounts {
+				mountPaths[mount.ID] = filepath.Join(t.TempDir(), mount.ID)
+			}
+			preview, previewErr := instance.BuildPreview(snapshot, instance.PreviewRequest{
+				DisplayName:     "Snapshot consistency",
+				HostPorts:       hostPorts,
+				MountPaths:      mountPaths,
+				DataOrigin:      "new",
+				BackupRetention: 7,
+			})
+			if previewErr != nil {
+				t.Fatalf("build preview for %s@%s: %v", snapshot.Template.ID, snapshot.Template.Version, previewErr)
+			}
+			if !strings.HasSuffix(preview.Image, ":"+snapshot.Template.Container.Tag) {
+				t.Fatalf("preview image %q does not use template tag %q", preview.Image, snapshot.Template.Container.Tag)
+			}
+			plan, planErr := preview.DeploymentPlan("abcdefghijklmnopqrstuvwx")
+			if planErr != nil {
+				t.Fatal(planErr)
+			}
+			known, ok := policy.snapshots[plan.TemplateID+"@"+plan.TemplateVersion]
+			if !ok || known.Digest != plan.TemplateDigest {
+				t.Fatalf("agent snapshot=%#v plan=%s@%s digest=%s", known, plan.TemplateID, plan.TemplateVersion, plan.TemplateDigest)
+			}
+			if validateErr := policy.Validate(plan); validateErr != nil {
+				t.Fatalf("application plan rejected by matching embedded snapshot: %v", validateErr)
+			}
+		})
 	}
 }

internal/catalog/local_test.go

@@ -12,49 +12,24 @@ import (
 
 func TestInitializeOfficialAndScanDirPreservesLocalFiles(t *testing.T) {
 	root := t.TempDir()
-
 	if err := catalog.InitializeOfficial(root, catalogdata.Files); err != nil {
 		t.Fatal(err)
 	}
-
 	first, err := catalog.ScanDir(root)
 	if err != nil {
 		t.Fatal(err)
 	}
-
-	if len(first.Valid) < 2 {
-		t.Fatalf("expected at least 2 valid official templates, got %#v", first)
+	if len(first.Valid) == 0 || first.Found != len(first.Valid) || len(first.Errors) != 0 {
+		t.Fatalf("initial catalog scan = %#v", first)
 	}
 
-	var palworld *catalog.Snapshot
-	for i := range first.Valid {
-		if first.Valid[i].Template.ID == "palworld-official" {
-			palworld = &first.Valid[i]
-			break
-		}
-	}
-
-	if palworld == nil {
-		t.Fatal(`official template "palworld-official" is missing`)
-	}
-
-	if palworld.Template.Game.Artwork.Image == "" {
-		t.Fatal("palworld artwork image was lost")
-	}
-
-	if len(palworld.Template.Configuration.Fields) == 0 {
-		t.Fatal("palworld configuration fields were lost")
-	}
-
-	path := filepath.Join(root, "palworld", "template.yaml")
+	path := filepath.Join(root, first.Valid[0].AssetRoot, "template.yaml")
 	if err := os.WriteFile(path, []byte("local customization"), 0o644); err != nil {
 		t.Fatal(err)
 	}
-
 	if err := catalog.InitializeOfficial(root, catalogdata.Files); err != nil {
 		t.Fatal(err)
 	}
-
 	body, err := os.ReadFile(path)
 	if err != nil || string(body) != "local customization" {
 		t.Fatalf("local template was overwritten: %q, %v", body, err)
@@ -63,54 +38,65 @@ func TestInitializeOfficialAndScanDirPreservesLocalFiles(t *testing.T) {
 
 func TestScanDirKeepsValidTemplatesWhenOneIsInvalid(t *testing.T) {
 	root := t.TempDir()
-
 	if err := catalog.InitializeOfficial(root, catalogdata.Files); err != nil {
 		t.Fatal(err)
 	}
-
-	body, err := catalogdata.Files.ReadFile("palworld/template.yaml")
-	if err != nil {
-		t.Fatal(err)
+	initial, err := catalog.ScanDir(root)
+	if err != nil || len(initial.Valid) == 0 {
+		t.Fatalf("initial scan = %#v, %v", initial, err)
 	}
 
-	broken := strings.Replace(
-		string(body),
-		"image: assets/banner.jpg",
-		"image: ../invalid.jpg",
-		1,
-	)
-
+	broken := strings.Replace(initial.Valid[0].CanonicalYAML, `"image": "`+initial.Valid[0].Template.Game.Artwork.Image+`"`, `"image": "../invalid.jpg"`, 1)
 	if err := os.Mkdir(filepath.Join(root, "broken"), 0o755); err != nil {
 		t.Fatal(err)
 	}
-
-	if err := os.WriteFile(
-		filepath.Join(root, "broken", "template.yaml"),
-		[]byte(broken),
-		0o644,
-	); err != nil {
+	if err := os.WriteFile(filepath.Join(root, "broken", "template.yaml"), []byte(broken), 0o644); err != nil {
 		t.Fatal(err)
 	}
-
 	result, err := catalog.ScanDir(root)
 	if err != nil {
 		t.Fatal(err)
 	}
-
-	if result.Found != 3 {
-		t.Fatalf("found = %d, want 3; scan = %#v", result.Found, result)
+	if result.Found != initial.Found+1 || len(result.Valid) != len(initial.Valid) || len(result.Errors) != 1 {
+		t.Fatalf("scan with invalid template = %#v", result)
 	}
-
-	if len(result.Valid) != 2 {
-		t.Fatalf("valid = %d, want 2; scan = %#v", len(result.Valid), result)
-	}
-
-	if len(result.Errors) != 1 {
-		t.Fatalf("errors = %d, want 1; scan = %#v", len(result.Errors), result)
-	}
-
-	if result.Errors[0].Template != "broken" ||
-		!strings.Contains(result.Errors[0].Message, "/game/artwork/image") {
+	if result.Errors[0].Template != "broken" || !strings.Contains(result.Errors[0].Message, "/game/artwork/image") {
 		t.Fatalf("error = %#v", result.Errors)
 	}
 }
+
+func TestScanDirAcceptsLocallyModifiedReferencedAsset(t *testing.T) {
+	root := t.TempDir()
+	if err := catalog.InitializeOfficial(root, catalogdata.Files); err != nil {
+		t.Fatal(err)
+	}
+	before, err := catalog.ScanDir(root)
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, snapshot := range before.Valid {
+		if len(snapshot.Template.Container.Assets) == 0 {
+			continue
+		}
+		asset := snapshot.Template.Container.Assets[0]
+		assetPath := filepath.Join(root, filepath.FromSlash(snapshot.AssetRoot), filepath.FromSlash(asset.Source))
+		if err := os.WriteFile(assetPath, []byte("locally maintained asset\n"), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		templatePath := filepath.Join(root, filepath.FromSlash(snapshot.AssetRoot), "template.yaml")
+		body, err := os.ReadFile(templatePath)
+		if err != nil {
+			t.Fatal(err)
+		}
+		legacy := strings.Replace(string(body), "      destination: "+asset.Destination+"\n", "      destination: "+asset.Destination+"\n      sha256: obsolete-checksum\n", 1)
+		if err := os.WriteFile(templatePath, []byte(legacy), 0o644); err != nil {
+			t.Fatal(err)
+		}
+		after, scanErr := catalog.ScanDir(root)
+		if scanErr != nil || len(after.Errors) != 0 || len(after.Valid) != len(before.Valid) {
+			t.Fatalf("scan after modifying asset %q = %#v, %v", assetPath, after, scanErr)
+		}
+		return
+	}
+	t.Skip("embedded catalog has no container assets")
+}

internal/catalog/template.go

@@ -72,7 +72,6 @@ type Template struct {
 		Assets             []struct {
 			Source      string `json:"source"`
 			Destination string `json:"destination"`
-			SHA256      string `json:"sha256"`
 			ReadOnly    bool   `json:"read_only"`
 		} `json:"assets"`
 	} `json:"container"`
@@ -472,10 +471,8 @@ func crossValidate(template Template, assetRoot string, source fs.FS) []Validati
 		issues = append(issues, ValidationIssue{Path: "/requirements/recommended", Message: "recommended resources must not be below minimum resources"})
 	}
 	for _, asset := range template.Container.Assets {
-		body, err := fs.ReadFile(source, path.Join(assetRoot, asset.Source))
-		digest := sha256.Sum256(body)
-		if err != nil || hex.EncodeToString(digest[:]) != asset.SHA256 {
-			issues = append(issues, ValidationIssue{Path: "/container/assets/" + asset.Source, Message: "asset is missing or its checksum does not match"})
+		if _, err := fs.Stat(source, path.Join(assetRoot, asset.Source)); err != nil {
+			issues = append(issues, ValidationIssue{Path: "/container/assets/" + asset.Source, Message: "asset is missing"})
 		}
 	}
 	return issues

internal/catalog/template_test.go

@@ -2,6 +2,10 @@ package catalog_test
 
 import (
 	"errors"
+	"io/fs"
+	"os"
+	"path"
+	"path/filepath"
 	"strings"
 	"testing"
 
@@ -9,7 +13,62 @@ import (
 	"git.zaynet.fr/DoGaMa/DoGaMa-serv/internal/catalog"
 )
 
-func TestBuiltInCatalogValidatesDeterministically(t *testing.T) {
+// TestBuiltInCatalogValidatesEveryDiscoveredTemplate deliberately enumerates
+// the embedded filesystem instead of knowing any game IDs, versions or order.
+func TestBuiltInCatalogValidatesEveryDiscoveredTemplate(t *testing.T) {
+	names, err := fs.Glob(catalogdata.Files, "*/template.yaml")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(names) == 0 {
+		t.Fatal("embedded catalog contains no templates")
+	}
+
+	snapshots, err := catalog.LoadFS(catalogdata.Files, ".")
+	if err != nil {
+		t.Fatalf("load embedded catalog: %v", err)
+	}
+	if len(snapshots) != len(names) {
+		t.Fatalf("loaded %d templates after discovering %d paths", len(snapshots), len(names))
+	}
+
+	loaded := make(map[string]catalog.Snapshot, len(snapshots))
+	for _, snapshot := range snapshots {
+		key := snapshot.Template.ID + "@" + snapshot.Template.Version
+		if _, duplicate := loaded[key]; duplicate {
+			t.Fatalf("duplicate loaded template identity %q", key)
+		}
+		loaded[key] = snapshot
+	}
+	for _, name := range names {
+		body, readErr := catalogdata.Files.ReadFile(name)
+		if readErr != nil {
+			t.Fatalf("read embedded template %q: %v", name, readErr)
+		}
+		snapshot, validateErr := catalog.Validate(body, path.Dir(name), catalogdata.Files)
+		if validateErr != nil {
+			t.Fatalf("validate embedded template path=%q: %v", name, validateErr)
+		}
+		key := snapshot.Template.ID + "@" + snapshot.Template.Version
+		if _, ok := loaded[key]; !ok {
+			t.Fatalf("validated embedded template path=%q id=%q was not loaded", name, snapshot.Template.ID)
+		}
+		for _, asset := range snapshot.Template.Container.Assets {
+			assetPath := path.Join(path.Dir(name), asset.Source)
+			if _, statErr := fs.Stat(catalogdata.Files, assetPath); statErr != nil {
+				t.Fatalf("template path=%q id=%q asset=%q is missing: %v", name, snapshot.Template.ID, assetPath, statErr)
+			}
+		}
+		if snapshot.Template.Integration != nil {
+			manifest := filepath.Join("..", "..", "modules", snapshot.Template.Integration.ModuleID, "manifest.yaml")
+			if _, statErr := os.Stat(manifest); statErr != nil {
+				t.Fatalf("template path=%q id=%q declared module=%q is missing at %q: %v", name, snapshot.Template.ID, snapshot.Template.Integration.ModuleID, manifest, statErr)
+			}
+		}
+	}
+}
+
+func TestBuiltInCatalogIsDeterministic(t *testing.T) {
 	first, err := catalog.LoadFS(catalogdata.Files, ".")
 	if err != nil {
 		t.Fatal(err)
@@ -18,137 +77,79 @@ func TestBuiltInCatalogValidatesDeterministically(t *testing.T) {
 	if err != nil {
 		t.Fatal(err)
 	}
-
 	if len(first) != len(second) {
 		t.Fatalf("catalog snapshot count differs: first=%d second=%d", len(first), len(second))
 	}
-
-	if len(first) < 2 {
-		t.Fatalf("expected at least 2 built-in templates, got %d", len(first))
-	}
-
-	secondByID := make(map[string]catalog.Snapshot, len(second))
-	for _, snapshot := range second {
-		secondByID[snapshot.Template.ID] = snapshot
-	}
-
-	firstByID := make(map[string]catalog.Snapshot, len(first))
-	for _, snapshot := range first {
-		firstByID[snapshot.Template.ID] = snapshot
-
-		other, ok := secondByID[snapshot.Template.ID]
-		if !ok {
-			t.Fatalf("template %q missing from second catalog load", snapshot.Template.ID)
+	for index, snapshot := range first {
+		other := second[index]
+		if snapshot.Template.ID != other.Template.ID || snapshot.Template.Version != other.Template.Version || snapshot.Digest != other.Digest || snapshot.CanonicalYAML != other.CanonicalYAML {
+			t.Fatalf("catalog snapshot %d is not deterministic: first=%#v second=%#v", index, snapshot, other)
 		}
-
-		if snapshot.Digest != other.Digest {
-			t.Fatalf(
-				"template %q digest is not deterministic: first=%q second=%q",
-				snapshot.Template.ID,
-				snapshot.Digest,
-				other.Digest,
-			)
-		}
-
-		if snapshot.CanonicalYAML != other.CanonicalYAML {
-			t.Fatalf("template %q canonical YAML is not deterministic", snapshot.Template.ID)
-		}
-	}
-
-	palworld, ok := firstByID["palworld-official"]
-	if !ok {
-		t.Fatal(`built-in template "palworld-official" is missing`)
-	}
-
-	if palworld.Template.Game.Artwork.Logo != "assets/icon.png" ||
-		palworld.Template.Game.Artwork.Image != "assets/banner.jpg" ||
-		palworld.Template.Game.Artwork.Poster != "assets/poster.jpg" {
-		t.Fatalf("palworld artwork = %#v", palworld.Template.Game.Artwork)
-	}
-
-	vrising, ok := firstByID["vrising-didstopia"]
-	if !ok {
-		t.Fatal(`built-in template "vrising-didstopia" is missing`)
-	}
-
-	if vrising.Template.Game.ID != "vrising" {
-		t.Fatalf("vrising game id = %q", vrising.Template.Game.ID)
-	}
-
-	if vrising.Template.Game.Artwork.Logo != "assets/icon.png" ||
-		vrising.Template.Game.Artwork.Image != "assets/banner.jpg" ||
-		vrising.Template.Game.Artwork.Poster != "assets/poster.jpg" {
-		t.Fatalf("vrising artwork = %#v", vrising.Template.Game.Artwork)
 	}
 }
 
 func TestCrossValidationRejectsMissingArtworkAsset(t *testing.T) {
-	body, err := catalogdata.Files.ReadFile("palworld/template.yaml")
-	if err != nil {
-		t.Fatal(err)
-	}
-	body = []byte(strings.Replace(string(body), "image: assets/banner.jpg", "image: assets/missing.jpg", 1))
-	_, err = catalog.Validate(body, "palworld", catalogdata.Files)
+	name, body, snapshot := embeddedTemplate(t)
+	body = []byte(strings.Replace(string(body), "image: "+snapshot.Template.Game.Artwork.Image, "image: assets/missing", 1))
+	_, err := catalog.Validate(body, path.Dir(name), catalogdata.Files)
 
 	var validation *catalog.ValidationErrors
 	if !errors.As(err, &validation) {
 		t.Fatalf("validation error = %#v", err)
 	}
-
 	for _, issue := range validation.Issues {
 		if issue.Path == "/game/artwork/image" {
 			return
 		}
 	}
-
 	t.Fatalf("issues = %#v", validation.Issues)
 }
 
 func TestSchemaErrorsContainFieldPathAndLine(t *testing.T) {
-	body, err := catalogdata.Files.ReadFile("palworld/template.yaml")
-	if err != nil {
-		t.Fatal(err)
-	}
+	name, body, _ := embeddedTemplate(t)
 	body = []byte(strings.Replace(string(body), "schema_version: 1", "schema_version: 2", 1))
-	_, err = catalog.Validate(body, "palworld", catalogdata.Files)
+	_, err := catalog.Validate(body, path.Dir(name), catalogdata.Files)
 
 	var validation *catalog.ValidationErrors
 	if !errors.As(err, &validation) || len(validation.Issues) == 0 {
 		t.Fatalf("validation error = %#v", err)
 	}
-
 	if validation.Issues[0].Path == "" || validation.Issues[0].Line == 0 {
 		t.Fatalf("validation issue = %#v", validation.Issues[0])
 	}
 }
 
 func TestCrossValidationRejectsUnknownBackupMount(t *testing.T) {
-	body, err := catalogdata.Files.ReadFile("palworld/template.yaml")
-	if err != nil {
-		t.Fatal(err)
-	}
-
+	name, body, snapshot := embeddedTemplate(t)
 	normalized := strings.ReplaceAll(string(body), "\r\n", "\n")
-	body = []byte(strings.Replace(
-		normalized,
-		"    - saved\n  restart_after_backup",
-		"    - missing\n  restart_after_backup",
-		1,
-	))
-
-	_, err = catalog.Validate(body, "palworld", catalogdata.Files)
+	body = []byte(strings.Replace(normalized, "    - "+snapshot.Template.Backup.SourceMounts[0]+"\n", "    - missing\n", 1))
+	_, err := catalog.Validate(body, path.Dir(name), catalogdata.Files)
 
 	var validation *catalog.ValidationErrors
 	if !errors.As(err, &validation) {
 		t.Fatalf("validation error = %#v", err)
 	}
-
-	found := false
 	for _, issue := range validation.Issues {
-		found = found || issue.Path == "/backup/source_mounts"
-	}
-
-	if !found {
-		t.Fatalf("issues = %#v", validation.Issues)
+		if issue.Path == "/backup/source_mounts" {
+			return
+		}
 	}
+	t.Fatalf("issues = %#v", validation.Issues)
+}
+
+func embeddedTemplate(t *testing.T) (string, []byte, catalog.Snapshot) {
+	t.Helper()
+	names, err := fs.Glob(catalogdata.Files, "*/template.yaml")
+	if err != nil || len(names) == 0 {
+		t.Fatalf("discover embedded template: names=%v err=%v", names, err)
+	}
+	body, err := catalogdata.Files.ReadFile(names[0])
+	if err != nil {
+		t.Fatal(err)
+	}
+	snapshot, err := catalog.Validate(body, path.Dir(names[0]), catalogdata.Files)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return names[0], body, snapshot
 }

specs/template.schema.json

@@ -66,11 +66,11 @@
           "items": {
             "type": "object",
             "additionalProperties": false,
-            "required": ["source", "destination", "sha256", "read_only"],
+            "required": ["source", "destination", "read_only"],
             "properties": {
               "source": { "type": "string", "pattern": "^(?!/)(?!.*\\.\\./).+$", "maxLength": 200 },
               "destination": { "type": "string", "pattern": "^/[^\\u0000]*$", "maxLength": 500 },
-              "sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
+              "sha256": { "type": "string", "maxLength": 128 },
               "read_only": { "const": true }
             }
           }

tools/validate_spec.py

@@ -53,7 +53,7 @@ def validate_links() -> None:
         raise ValueError("Broken internal links:\n  " + "\n  ".join(failures))
 
 
-def validate_cross_references(template: dict, manifest: dict) -> list[str]:
+def validate_cross_references(template: dict, manifest: dict | None, template_path: Path) -> list[str]:
     warnings = []
     port_ids = {item["id"] for item in template["container"]["ports"]}
     mount_ids = {item["id"] for item in template["storage"]["mounts"]}
@@ -61,6 +61,7 @@ def validate_cross_references(template: dict, manifest: dict) -> list[str]:
 
     integration = template.get("integration")
     if integration:
+        assert manifest is not None, f"Declared module is missing: {integration['module_id']}"
         assert integration["port_id"] in port_ids, "Template integration references an unknown port"
         assert integration["module_id"] == manifest["id"], "Template and manifest module IDs disagree"
         assert template["game"]["id"] in manifest["game_ids"], "Manifest does not support template game ID"
@@ -73,11 +74,15 @@ def validate_cross_references(template: dict, manifest: dict) -> list[str]:
     if mods.get("supported"):
         assert mods.get("destination_mount") in mount_ids, "Mods reference an unknown mount"
 
+    for field in ("logo", "image", "poster"):
+        path = (template_path.parent / template["game"]["artwork"][field]).resolve()
+        assert path.is_file(), f"Missing artwork asset {field}: {path}"
     for asset in template["container"].get("assets", []):
-        path = (ROOT / "catalog" / "palworld" / asset["source"]).resolve()
+        path = (template_path.parent / asset["source"]).resolve()
         assert path.is_file(), f"Missing packaged asset: {path}"
-        digest = hashlib.sha256(path.read_bytes()).hexdigest()
-        assert digest == asset["sha256"], f"Asset checksum mismatch: {path}"
+
+    if manifest is None:
+        return warnings
 
     wasm_digest = manifest["artifacts"]["sha256"]
     wasm_path = ROOT / "modules" / manifest["id"] / manifest["artifacts"]["wasm"]
@@ -91,6 +96,28 @@ def validate_cross_references(template: dict, manifest: dict) -> list[str]:
     return warnings
 
 
+def validate_catalog() -> list[str]:
+    template_schema = ROOT / "specs/template.schema.json"
+    manifest_schema = ROOT / "specs/module-manifest.schema.json"
+    template_paths = sorted((ROOT / "catalog").glob("*/template.yaml"))
+    assert template_paths, "Catalog contains no templates"
+    warnings = []
+    identities = set()
+    for template_path in template_paths:
+        template = validate(template_schema, template_path)
+        identity = (template["id"], template["version"])
+        assert identity not in identities, f"Duplicate template ID and version: {identity[0]}@{identity[1]}"
+        identities.add(identity)
+        integration = template.get("integration")
+        manifest = None
+        if integration:
+            manifest_path = ROOT / "modules" / integration["module_id"] / "manifest.yaml"
+            assert manifest_path.is_file(), f"Declared module manifest is missing: {manifest_path}"
+            manifest = validate(manifest_schema, manifest_path)
+        warnings.extend(validate_cross_references(template, manifest, template_path))
+    return warnings
+
+
 def validate_coverage() -> None:
     required = {
         "Docker agent": "docs/architecture/docker-agent.md",
@@ -156,14 +183,12 @@ def validate_workflows() -> None:
 
 
 def main() -> int:
-    template = validate(ROOT / "specs/template.schema.json", ROOT / "catalog/palworld/template.yaml")
-    manifest = validate(ROOT / "specs/module-manifest.schema.json", ROOT / "modules/palworld-rest/manifest.yaml")
     validate_compose()
     validate_workflows()
     for fixture in ROOT.rglob("*.json"):
         load_json(fixture)
     validate_links()
-    warnings = validate_cross_references(template, manifest)
+    warnings = validate_catalog()
     validate_coverage()
     print("DoGaMa specification validation passed.")
     for warning in warnings: