refactor(catalog): make embedded template validation generic #42
@@ -5,12 +5,12 @@ This template demonstrates every important V1 contract: official image, public U
|
||||
## Verified upstream facts
|
||||
|
||||
- Pocketpair publishes the official image and example Compose deployment at [pocketpairjp/palworld-dedicated-server-docker](https://github.com/pocketpairjp/palworld-dedicated-server-docker).
|
||||
- The current official example at specification time uses `ghcr.io/pocketpairjp/palserver:v1.0.2.101103`, UDP 8211, `/pal/Package/Pal/Saved` and the packaged `helper.sh` pattern.
|
||||
- Pocketpair publishes versioned image tags and a `latest` tag for the current official `ghcr.io/pocketpairjp/palserver` release. The official template uses `latest` for normal pulls and new deployments, with UDP 8211, `/pal/Package/Pal/Saved` and the packaged `helper.sh` pattern.
|
||||
- The [official requirements](https://docs.palworldgame.com/0.7.3/getting-started/requirements/) specify four or more CPU cores, 16 GB memory with over 32 GB recommended for stability, and UDP 8211. The template's 8 GB minimum is an explicit lower bootable boundary mentioned upstream, not a stability recommendation; the UI must warn below 16 GB.
|
||||
- The [configuration reference](https://docs.palworldgame.com/settings-and-operation/configuration/) documents `AdminPassword`, `RESTAPIEnabled`, `RESTAPIPort`, server name/password and maximum players.
|
||||
- The [REST API documentation](https://docs.palworldgame.com/category/rest-api/) documents information, players, metrics, announce, save, shutdown and moderation operations. REST API credentials and port must remain private.
|
||||
|
||||
The image tag and upstream API may change. Catalog maintainers must verify and release a new immutable template version; existing instances remain pinned.
|
||||
The upstream API may change. A template snapshot version remains independent from the game-server image tag; existing instances remain pinned to their chosen configuration and DoGaMa does not automatically replace running containers.
|
||||
|
||||
## Reference limitations
|
||||
|
||||
@@ -18,4 +18,3 @@ The image tag and upstream API may change. Catalog maintainers must verify and r
|
||||
- The schema's storage sizes are conservative product defaults because upstream specifies SSD performance but not a fixed disk-size requirement.
|
||||
- Local hosted-world migration may require player identity conversion. The generic V1 importer detects structure and warns; it does not silently convert identities.
|
||||
- The checked-in module manifest is a source example. It becomes installable only after `module.wasm` is built and its real SHA-256 replaces the all-zero placeholder.
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
schema_version: 1
|
||||
id: palworld-official
|
||||
version: 1.1.1
|
||||
version: 1.1.2
|
||||
|
||||
source:
|
||||
type: official
|
||||
@@ -29,7 +29,7 @@ requirements:
|
||||
|
||||
container:
|
||||
image: ghcr.io/pocketpairjp/palserver
|
||||
tag: v1.0.2.101103
|
||||
tag: latest
|
||||
entrypoint:
|
||||
- /pal/helper.sh
|
||||
user_mode: image
|
||||
@@ -41,7 +41,6 @@ container:
|
||||
assets:
|
||||
- source: assets/helper.sh
|
||||
destination: /pal/helper.sh
|
||||
sha256: 52e58fe4e22654d0d312fe2bb6195db61dad5ffce78e0440a951d33d20f2c36f
|
||||
read_only: true
|
||||
stop_timeout_seconds: 120
|
||||
ports:
|
||||
|
||||
@@ -58,6 +58,7 @@ Read this compact operational baseline before starting a milestone. Open detaile
|
||||
- Label values support only the explicit allowlist in `internal/instance/container_config.go`; unknown variables are errors, not arbitrary templates.
|
||||
- `{{game.icon_url}}` is the public icon for the game. `{{instance.slug}}` remains supported.
|
||||
- Template game artwork contains separate required local `logo` and horizontal `image` assets; template validation rejects missing files. Deployment previews expose distinct logo and artwork URLs while retaining `icon_url` as a compatible logo alias. Palworld template `1.1.0` is the first snapshot with this contract.
|
||||
- Embedded catalog validation is collection-based: every discovered template is schema- and cross-field-validated, including referenced assets, declared integration modules and ports/configuration. Template asset contents are not SHA-256-pinned, allowing administrator-maintained local assets while preserving required-path validation. A template `version` identifies a DoGaMa snapshot, whereas `container.tag` selects the game-server image; the official Palworld template follows Pocketpair's `latest` tag for new deployments and ordinary pulls.
|
||||
- Instance slugs are derived from the display name, not canonical IDs. Accents are normalized to ASCII; whitespace, `/`, punctuation and special characters become safe hyphen separators; repeated and edge hyphens are removed.
|
||||
- Docker user mode is fixed at creation to DoGaMa UID/GID, custom numeric UID/GID, or image-defined user. Never perform automatic recursive ownership changes.
|
||||
- A pinned image tag is an explicit mutable tag, not an immutable digest. Tracked mode follows the template's declared default tag.
|
||||
|
||||
@@ -133,8 +133,8 @@ before the restricted agent creates the first container. Repeated installation
|
||||
submission recognizes an already attached import instead of copying it twice.
|
||||
|
||||
At main-application startup, every embedded `catalog/*/template.yaml` is
|
||||
validated against `specs/template.schema.json`, checked for cross-reference and
|
||||
asset integrity, canonicalized deterministically and synchronized into SQLite.
|
||||
validated against `specs/template.schema.json`, checked for cross-references and
|
||||
referenced asset presence, canonicalized deterministically and synchronized into SQLite.
|
||||
An existing template ID/version is immutable: changing its digest fails startup
|
||||
instead of silently replacing the snapshot. Deployment previews pin that digest
|
||||
and redact secret defaults before a draft instance can enter the registry.
|
||||
|
||||
@@ -19,7 +19,11 @@ Add or edit files on the Docker host, then sign in as an administrator and press
|
||||
|
||||
## Template format
|
||||
|
||||
Templates use schema version `1` and are strict YAML documents. Existing deployment fields remain required: container image/tag, ports, storage mounts, configuration fields, capabilities, backup, healthcheck, imports, updates and compatibility. DoGaMa rejects unknown fields and unsafe paths, asset checksums, invalid configuration fields and invalid Docker-related declarations; a template cannot grant arbitrary Docker access.
|
||||
Templates use schema version `1` and are strict YAML documents. Existing deployment fields remain required: container image/tag, ports, storage mounts, configuration fields, capabilities, backup, healthcheck, imports, updates and compatibility. DoGaMa rejects unknown fields, unsafe paths, missing referenced assets, invalid configuration fields and invalid Docker-related declarations; a template cannot grant arbitrary Docker access. Asset contents are deliberately not checksum-pinned, so an administrator can maintain a local helper or configuration asset without invalidating an otherwise valid template.
|
||||
|
||||
For local-template upgrade compatibility, an existing asset `sha256` key is accepted and ignored. New templates should omit it.
|
||||
|
||||
`version` identifies the immutable DoGaMa template snapshot. It is not the game-server release. The server image release is selected separately by `container.tag`; a floating tag follows that image publisher's tag policy for newly created or normally pulled instances.
|
||||
|
||||
The catalog information is under `game` and `requirements`:
|
||||
|
||||
@@ -32,17 +36,17 @@ game:
|
||||
id: example-game
|
||||
name: Example Game
|
||||
description: Concise dedicated-server description.
|
||||
image: https://example.invalid/cover.jpg # vertical public cover URL
|
||||
artwork:
|
||||
logo: assets/icon.png # retained deployment artwork asset
|
||||
logo: assets/icon.png
|
||||
image: assets/banner.jpg
|
||||
poster: assets/poster.jpg
|
||||
attribution: Your attribution text
|
||||
requirements:
|
||||
minimum: { cpu_cores: 2, memory_mb: 4096, storage_gb: 20, other: ["Network connection"] }
|
||||
recommended: { cpu_cores: 4, memory_mb: 8192, storage_gb: 40 }
|
||||
```
|
||||
|
||||
`game.image` is the vertical cover displayed by Catalog and the game page. Its URL must be HTTPS, but validation never fetches it: an unavailable remote cover does not block a scan and the interface has a graphical fallback. `requirements.minimum` and `requirements.recommended` contain generic CPU, memory and storage values plus optional `other` lines. Recommended resources cannot be below minimum resources.
|
||||
`game.artwork` contains required local logo, horizontal image and poster assets. Validation never fetches remote artwork, so a scan remains local and deterministic. `requirements.minimum` and `requirements.recommended` contain generic CPU, memory and storage values plus optional `other` lines. Recommended resources cannot be below minimum resources.
|
||||
|
||||
`configuration.fields` drives the deployment form and the effective server configuration. Supported types are `string`, `integer`, `number`, `boolean`, `enum` and `secret`; defaults, required flags, numeric bounds, regular expressions and enum values are checked again by the server. Secret inputs are write-only, encrypted in the instance secret store when retained for runtime use, and never included in a persisted preview, API response, audit event or error.
|
||||
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
@@ -22,12 +20,10 @@ func (s *service) prepareAssets(plan agentwire.DeploymentPlan) ([]AssetMount, er
|
||||
return nil, errors.New("asset root is not allowed")
|
||||
}
|
||||
result := make([]AssetMount, 0, len(approved))
|
||||
for _, asset := range approved {
|
||||
digest := sha256.Sum256(asset.Content)
|
||||
if hex.EncodeToString(digest[:]) != asset.SHA256 {
|
||||
return nil, errors.New("approved asset integrity check failed")
|
||||
}
|
||||
target := filepath.Join(assetRoot, asset.SHA256)
|
||||
for index, asset := range approved {
|
||||
// The template snapshot digest keeps agent-owned asset paths isolated
|
||||
// without making an asset's declared content immutable.
|
||||
target := filepath.Join(assetRoot, plan.TemplateDigest, fmt.Sprintf("asset-%d", index))
|
||||
if err := writeImmutableAsset(target, asset.Content); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -41,18 +37,13 @@ func writeImmutableAsset(path string, content []byte) error {
|
||||
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 {
|
||||
return errors.New("approved asset path is not a regular file")
|
||||
}
|
||||
existing, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return errors.New("read approved asset")
|
||||
}
|
||||
existingDigest, wantedDigest := sha256.Sum256(existing), sha256.Sum256(content)
|
||||
if existingDigest != wantedDigest {
|
||||
return errors.New("approved asset content conflict")
|
||||
}
|
||||
return os.Chmod(path, 0o555)
|
||||
} else if !errors.Is(err, os.ErrNotExist) {
|
||||
return errors.New("inspect approved asset path")
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
|
||||
return errors.New("create approved asset directory")
|
||||
}
|
||||
temporary, err := os.CreateTemp(filepath.Dir(path), ".asset-*")
|
||||
if err != nil {
|
||||
return fmt.Errorf("create approved asset: %w", err)
|
||||
|
||||
@@ -40,7 +40,6 @@ func NewPlanPolicy(snapshots []catalog.Snapshot, assets fs.FS) (*PlanPolicy, err
|
||||
|
||||
type ApprovedAsset struct {
|
||||
Destination string
|
||||
SHA256 string
|
||||
Content []byte
|
||||
}
|
||||
|
||||
@@ -61,7 +60,7 @@ func (p *PlanPolicy) Assets(plan agentwire.DeploymentPlan) ([]ApprovedAsset, err
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("read approved template asset: %w", err)
|
||||
}
|
||||
result = append(result, ApprovedAsset{Destination: asset.Destination, SHA256: asset.SHA256, Content: content})
|
||||
result = append(result, ApprovedAsset{Destination: asset.Destination, Content: content})
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ package agent
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
catalogdata "git.zaynet.fr/DoGaMa/DoGaMa-serv/catalog"
|
||||
@@ -9,73 +10,52 @@ import (
|
||||
"git.zaynet.fr/DoGaMa/DoGaMa-serv/internal/instance"
|
||||
)
|
||||
|
||||
func TestEmbeddedPalworldSnapshotMatchesApplicationDeploymentPlan(t *testing.T) {
|
||||
func TestEmbeddedSnapshotsMatchApplicationDeploymentPlans(t *testing.T) {
|
||||
snapshots, err := catalog.LoadFS(catalogdata.Files, ".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var snapshot *catalog.Snapshot
|
||||
for i := range snapshots {
|
||||
if snapshots[i].Template.ID == "palworld-official" {
|
||||
snapshot = &snapshots[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if snapshot == nil {
|
||||
t.Fatal(`embedded template "palworld-official" not found`)
|
||||
}
|
||||
|
||||
if snapshot.Template.Version != "1.1.1" {
|
||||
t.Fatalf(
|
||||
"embedded Palworld snapshot = %s@%s",
|
||||
snapshot.Template.ID,
|
||||
snapshot.Template.Version,
|
||||
)
|
||||
}
|
||||
|
||||
preview, err := instance.BuildPreview(*snapshot, instance.PreviewRequest{
|
||||
DisplayName: "Snapshot consistency",
|
||||
Slug: "snapshot-consistency",
|
||||
HostPorts: map[string]int{
|
||||
"game": 38211,
|
||||
},
|
||||
MountPaths: map[string]string{
|
||||
"saved": filepath.Join(t.TempDir(), "saved"),
|
||||
},
|
||||
DataOrigin: "new",
|
||||
BackupRetention: 7,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
plan, err := preview.DeploymentPlan("abcdefghijklmnopqrstuvwx")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
policy, err := NewPlanPolicy(snapshots, catalogdata.Files)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
known, ok := policy.snapshots[plan.TemplateID+"@"+plan.TemplateVersion]
|
||||
if !ok || known.Digest != plan.TemplateDigest {
|
||||
t.Fatalf(
|
||||
"agent snapshot=%#v plan=%s@%s digest=%s",
|
||||
known,
|
||||
plan.TemplateID,
|
||||
plan.TemplateVersion,
|
||||
plan.TemplateDigest,
|
||||
)
|
||||
}
|
||||
|
||||
if err := policy.Validate(plan); err != nil {
|
||||
t.Fatalf(
|
||||
"application plan rejected by matching embedded snapshot: %v",
|
||||
err,
|
||||
)
|
||||
for index, snapshot := range snapshots {
|
||||
t.Run(snapshot.Template.ID, func(t *testing.T) {
|
||||
hostPorts := map[string]int{}
|
||||
for portIndex, port := range snapshot.Template.Container.Ports {
|
||||
if port.Publish {
|
||||
hostPorts[port.ID] = 38000 + index*100 + portIndex
|
||||
}
|
||||
}
|
||||
mountPaths := map[string]string{}
|
||||
for _, mount := range snapshot.Template.Storage.Mounts {
|
||||
mountPaths[mount.ID] = filepath.Join(t.TempDir(), mount.ID)
|
||||
}
|
||||
preview, previewErr := instance.BuildPreview(snapshot, instance.PreviewRequest{
|
||||
DisplayName: "Snapshot consistency",
|
||||
HostPorts: hostPorts,
|
||||
MountPaths: mountPaths,
|
||||
DataOrigin: "new",
|
||||
BackupRetention: 7,
|
||||
})
|
||||
if previewErr != nil {
|
||||
t.Fatalf("build preview for %s@%s: %v", snapshot.Template.ID, snapshot.Template.Version, previewErr)
|
||||
}
|
||||
if !strings.HasSuffix(preview.Image, ":"+snapshot.Template.Container.Tag) {
|
||||
t.Fatalf("preview image %q does not use template tag %q", preview.Image, snapshot.Template.Container.Tag)
|
||||
}
|
||||
plan, planErr := preview.DeploymentPlan("abcdefghijklmnopqrstuvwx")
|
||||
if planErr != nil {
|
||||
t.Fatal(planErr)
|
||||
}
|
||||
known, ok := policy.snapshots[plan.TemplateID+"@"+plan.TemplateVersion]
|
||||
if !ok || known.Digest != plan.TemplateDigest {
|
||||
t.Fatalf("agent snapshot=%#v plan=%s@%s digest=%s", known, plan.TemplateID, plan.TemplateVersion, plan.TemplateDigest)
|
||||
}
|
||||
if validateErr := policy.Validate(plan); validateErr != nil {
|
||||
t.Fatalf("application plan rejected by matching embedded snapshot: %v", validateErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,49 +12,24 @@ import (
|
||||
|
||||
func TestInitializeOfficialAndScanDirPreservesLocalFiles(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
|
||||
if err := catalog.InitializeOfficial(root, catalogdata.Files); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
first, err := catalog.ScanDir(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(first.Valid) < 2 {
|
||||
t.Fatalf("expected at least 2 valid official templates, got %#v", first)
|
||||
if len(first.Valid) == 0 || first.Found != len(first.Valid) || len(first.Errors) != 0 {
|
||||
t.Fatalf("initial catalog scan = %#v", first)
|
||||
}
|
||||
|
||||
var palworld *catalog.Snapshot
|
||||
for i := range first.Valid {
|
||||
if first.Valid[i].Template.ID == "palworld-official" {
|
||||
palworld = &first.Valid[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if palworld == nil {
|
||||
t.Fatal(`official template "palworld-official" is missing`)
|
||||
}
|
||||
|
||||
if palworld.Template.Game.Artwork.Image == "" {
|
||||
t.Fatal("palworld artwork image was lost")
|
||||
}
|
||||
|
||||
if len(palworld.Template.Configuration.Fields) == 0 {
|
||||
t.Fatal("palworld configuration fields were lost")
|
||||
}
|
||||
|
||||
path := filepath.Join(root, "palworld", "template.yaml")
|
||||
path := filepath.Join(root, first.Valid[0].AssetRoot, "template.yaml")
|
||||
if err := os.WriteFile(path, []byte("local customization"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := catalog.InitializeOfficial(root, catalogdata.Files); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
body, err := os.ReadFile(path)
|
||||
if err != nil || string(body) != "local customization" {
|
||||
t.Fatalf("local template was overwritten: %q, %v", body, err)
|
||||
@@ -63,54 +38,65 @@ func TestInitializeOfficialAndScanDirPreservesLocalFiles(t *testing.T) {
|
||||
|
||||
func TestScanDirKeepsValidTemplatesWhenOneIsInvalid(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
|
||||
if err := catalog.InitializeOfficial(root, catalogdata.Files); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
body, err := catalogdata.Files.ReadFile("palworld/template.yaml")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
initial, err := catalog.ScanDir(root)
|
||||
if err != nil || len(initial.Valid) == 0 {
|
||||
t.Fatalf("initial scan = %#v, %v", initial, err)
|
||||
}
|
||||
|
||||
broken := strings.Replace(
|
||||
string(body),
|
||||
"image: assets/banner.jpg",
|
||||
"image: ../invalid.jpg",
|
||||
1,
|
||||
)
|
||||
|
||||
broken := strings.Replace(initial.Valid[0].CanonicalYAML, `"image": "`+initial.Valid[0].Template.Game.Artwork.Image+`"`, `"image": "../invalid.jpg"`, 1)
|
||||
if err := os.Mkdir(filepath.Join(root, "broken"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if err := os.WriteFile(
|
||||
filepath.Join(root, "broken", "template.yaml"),
|
||||
[]byte(broken),
|
||||
0o644,
|
||||
); err != nil {
|
||||
if err := os.WriteFile(filepath.Join(root, "broken", "template.yaml"), []byte(broken), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
result, err := catalog.ScanDir(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if result.Found != 3 {
|
||||
t.Fatalf("found = %d, want 3; scan = %#v", result.Found, result)
|
||||
if result.Found != initial.Found+1 || len(result.Valid) != len(initial.Valid) || len(result.Errors) != 1 {
|
||||
t.Fatalf("scan with invalid template = %#v", result)
|
||||
}
|
||||
|
||||
if len(result.Valid) != 2 {
|
||||
t.Fatalf("valid = %d, want 2; scan = %#v", len(result.Valid), result)
|
||||
}
|
||||
|
||||
if len(result.Errors) != 1 {
|
||||
t.Fatalf("errors = %d, want 1; scan = %#v", len(result.Errors), result)
|
||||
}
|
||||
|
||||
if result.Errors[0].Template != "broken" ||
|
||||
!strings.Contains(result.Errors[0].Message, "/game/artwork/image") {
|
||||
if result.Errors[0].Template != "broken" || !strings.Contains(result.Errors[0].Message, "/game/artwork/image") {
|
||||
t.Fatalf("error = %#v", result.Errors)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScanDirAcceptsLocallyModifiedReferencedAsset(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
if err := catalog.InitializeOfficial(root, catalogdata.Files); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
before, err := catalog.ScanDir(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, snapshot := range before.Valid {
|
||||
if len(snapshot.Template.Container.Assets) == 0 {
|
||||
continue
|
||||
}
|
||||
asset := snapshot.Template.Container.Assets[0]
|
||||
assetPath := filepath.Join(root, filepath.FromSlash(snapshot.AssetRoot), filepath.FromSlash(asset.Source))
|
||||
if err := os.WriteFile(assetPath, []byte("locally maintained asset\n"), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
templatePath := filepath.Join(root, filepath.FromSlash(snapshot.AssetRoot), "template.yaml")
|
||||
body, err := os.ReadFile(templatePath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
legacy := strings.Replace(string(body), " destination: "+asset.Destination+"\n", " destination: "+asset.Destination+"\n sha256: obsolete-checksum\n", 1)
|
||||
if err := os.WriteFile(templatePath, []byte(legacy), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
after, scanErr := catalog.ScanDir(root)
|
||||
if scanErr != nil || len(after.Errors) != 0 || len(after.Valid) != len(before.Valid) {
|
||||
t.Fatalf("scan after modifying asset %q = %#v, %v", assetPath, after, scanErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
t.Skip("embedded catalog has no container assets")
|
||||
}
|
||||
|
||||
@@ -72,7 +72,6 @@ type Template struct {
|
||||
Assets []struct {
|
||||
Source string `json:"source"`
|
||||
Destination string `json:"destination"`
|
||||
SHA256 string `json:"sha256"`
|
||||
ReadOnly bool `json:"read_only"`
|
||||
} `json:"assets"`
|
||||
} `json:"container"`
|
||||
@@ -472,10 +471,8 @@ func crossValidate(template Template, assetRoot string, source fs.FS) []Validati
|
||||
issues = append(issues, ValidationIssue{Path: "/requirements/recommended", Message: "recommended resources must not be below minimum resources"})
|
||||
}
|
||||
for _, asset := range template.Container.Assets {
|
||||
body, err := fs.ReadFile(source, path.Join(assetRoot, asset.Source))
|
||||
digest := sha256.Sum256(body)
|
||||
if err != nil || hex.EncodeToString(digest[:]) != asset.SHA256 {
|
||||
issues = append(issues, ValidationIssue{Path: "/container/assets/" + asset.Source, Message: "asset is missing or its checksum does not match"})
|
||||
if _, err := fs.Stat(source, path.Join(assetRoot, asset.Source)); err != nil {
|
||||
issues = append(issues, ValidationIssue{Path: "/container/assets/" + asset.Source, Message: "asset is missing"})
|
||||
}
|
||||
}
|
||||
return issues
|
||||
|
||||
@@ -2,6 +2,10 @@ package catalog_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -9,7 +13,62 @@ import (
|
||||
"git.zaynet.fr/DoGaMa/DoGaMa-serv/internal/catalog"
|
||||
)
|
||||
|
||||
func TestBuiltInCatalogValidatesDeterministically(t *testing.T) {
|
||||
// TestBuiltInCatalogValidatesEveryDiscoveredTemplate deliberately enumerates
|
||||
// the embedded filesystem instead of knowing any game IDs, versions or order.
|
||||
func TestBuiltInCatalogValidatesEveryDiscoveredTemplate(t *testing.T) {
|
||||
names, err := fs.Glob(catalogdata.Files, "*/template.yaml")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(names) == 0 {
|
||||
t.Fatal("embedded catalog contains no templates")
|
||||
}
|
||||
|
||||
snapshots, err := catalog.LoadFS(catalogdata.Files, ".")
|
||||
if err != nil {
|
||||
t.Fatalf("load embedded catalog: %v", err)
|
||||
}
|
||||
if len(snapshots) != len(names) {
|
||||
t.Fatalf("loaded %d templates after discovering %d paths", len(snapshots), len(names))
|
||||
}
|
||||
|
||||
loaded := make(map[string]catalog.Snapshot, len(snapshots))
|
||||
for _, snapshot := range snapshots {
|
||||
key := snapshot.Template.ID + "@" + snapshot.Template.Version
|
||||
if _, duplicate := loaded[key]; duplicate {
|
||||
t.Fatalf("duplicate loaded template identity %q", key)
|
||||
}
|
||||
loaded[key] = snapshot
|
||||
}
|
||||
for _, name := range names {
|
||||
body, readErr := catalogdata.Files.ReadFile(name)
|
||||
if readErr != nil {
|
||||
t.Fatalf("read embedded template %q: %v", name, readErr)
|
||||
}
|
||||
snapshot, validateErr := catalog.Validate(body, path.Dir(name), catalogdata.Files)
|
||||
if validateErr != nil {
|
||||
t.Fatalf("validate embedded template path=%q: %v", name, validateErr)
|
||||
}
|
||||
key := snapshot.Template.ID + "@" + snapshot.Template.Version
|
||||
if _, ok := loaded[key]; !ok {
|
||||
t.Fatalf("validated embedded template path=%q id=%q was not loaded", name, snapshot.Template.ID)
|
||||
}
|
||||
for _, asset := range snapshot.Template.Container.Assets {
|
||||
assetPath := path.Join(path.Dir(name), asset.Source)
|
||||
if _, statErr := fs.Stat(catalogdata.Files, assetPath); statErr != nil {
|
||||
t.Fatalf("template path=%q id=%q asset=%q is missing: %v", name, snapshot.Template.ID, assetPath, statErr)
|
||||
}
|
||||
}
|
||||
if snapshot.Template.Integration != nil {
|
||||
manifest := filepath.Join("..", "..", "modules", snapshot.Template.Integration.ModuleID, "manifest.yaml")
|
||||
if _, statErr := os.Stat(manifest); statErr != nil {
|
||||
t.Fatalf("template path=%q id=%q declared module=%q is missing at %q: %v", name, snapshot.Template.ID, snapshot.Template.Integration.ModuleID, manifest, statErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuiltInCatalogIsDeterministic(t *testing.T) {
|
||||
first, err := catalog.LoadFS(catalogdata.Files, ".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -18,137 +77,79 @@ func TestBuiltInCatalogValidatesDeterministically(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(first) != len(second) {
|
||||
t.Fatalf("catalog snapshot count differs: first=%d second=%d", len(first), len(second))
|
||||
}
|
||||
|
||||
if len(first) < 2 {
|
||||
t.Fatalf("expected at least 2 built-in templates, got %d", len(first))
|
||||
}
|
||||
|
||||
secondByID := make(map[string]catalog.Snapshot, len(second))
|
||||
for _, snapshot := range second {
|
||||
secondByID[snapshot.Template.ID] = snapshot
|
||||
}
|
||||
|
||||
firstByID := make(map[string]catalog.Snapshot, len(first))
|
||||
for _, snapshot := range first {
|
||||
firstByID[snapshot.Template.ID] = snapshot
|
||||
|
||||
other, ok := secondByID[snapshot.Template.ID]
|
||||
if !ok {
|
||||
t.Fatalf("template %q missing from second catalog load", snapshot.Template.ID)
|
||||
for index, snapshot := range first {
|
||||
other := second[index]
|
||||
if snapshot.Template.ID != other.Template.ID || snapshot.Template.Version != other.Template.Version || snapshot.Digest != other.Digest || snapshot.CanonicalYAML != other.CanonicalYAML {
|
||||
t.Fatalf("catalog snapshot %d is not deterministic: first=%#v second=%#v", index, snapshot, other)
|
||||
}
|
||||
|
||||
if snapshot.Digest != other.Digest {
|
||||
t.Fatalf(
|
||||
"template %q digest is not deterministic: first=%q second=%q",
|
||||
snapshot.Template.ID,
|
||||
snapshot.Digest,
|
||||
other.Digest,
|
||||
)
|
||||
}
|
||||
|
||||
if snapshot.CanonicalYAML != other.CanonicalYAML {
|
||||
t.Fatalf("template %q canonical YAML is not deterministic", snapshot.Template.ID)
|
||||
}
|
||||
}
|
||||
|
||||
palworld, ok := firstByID["palworld-official"]
|
||||
if !ok {
|
||||
t.Fatal(`built-in template "palworld-official" is missing`)
|
||||
}
|
||||
|
||||
if palworld.Template.Game.Artwork.Logo != "assets/icon.png" ||
|
||||
palworld.Template.Game.Artwork.Image != "assets/banner.jpg" ||
|
||||
palworld.Template.Game.Artwork.Poster != "assets/poster.jpg" {
|
||||
t.Fatalf("palworld artwork = %#v", palworld.Template.Game.Artwork)
|
||||
}
|
||||
|
||||
vrising, ok := firstByID["vrising-didstopia"]
|
||||
if !ok {
|
||||
t.Fatal(`built-in template "vrising-didstopia" is missing`)
|
||||
}
|
||||
|
||||
if vrising.Template.Game.ID != "vrising" {
|
||||
t.Fatalf("vrising game id = %q", vrising.Template.Game.ID)
|
||||
}
|
||||
|
||||
if vrising.Template.Game.Artwork.Logo != "assets/icon.png" ||
|
||||
vrising.Template.Game.Artwork.Image != "assets/banner.jpg" ||
|
||||
vrising.Template.Game.Artwork.Poster != "assets/poster.jpg" {
|
||||
t.Fatalf("vrising artwork = %#v", vrising.Template.Game.Artwork)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCrossValidationRejectsMissingArtworkAsset(t *testing.T) {
|
||||
body, err := catalogdata.Files.ReadFile("palworld/template.yaml")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
body = []byte(strings.Replace(string(body), "image: assets/banner.jpg", "image: assets/missing.jpg", 1))
|
||||
_, err = catalog.Validate(body, "palworld", catalogdata.Files)
|
||||
name, body, snapshot := embeddedTemplate(t)
|
||||
body = []byte(strings.Replace(string(body), "image: "+snapshot.Template.Game.Artwork.Image, "image: assets/missing", 1))
|
||||
_, err := catalog.Validate(body, path.Dir(name), catalogdata.Files)
|
||||
|
||||
var validation *catalog.ValidationErrors
|
||||
if !errors.As(err, &validation) {
|
||||
t.Fatalf("validation error = %#v", err)
|
||||
}
|
||||
|
||||
for _, issue := range validation.Issues {
|
||||
if issue.Path == "/game/artwork/image" {
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
t.Fatalf("issues = %#v", validation.Issues)
|
||||
}
|
||||
|
||||
func TestSchemaErrorsContainFieldPathAndLine(t *testing.T) {
|
||||
body, err := catalogdata.Files.ReadFile("palworld/template.yaml")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
name, body, _ := embeddedTemplate(t)
|
||||
body = []byte(strings.Replace(string(body), "schema_version: 1", "schema_version: 2", 1))
|
||||
_, err = catalog.Validate(body, "palworld", catalogdata.Files)
|
||||
_, err := catalog.Validate(body, path.Dir(name), catalogdata.Files)
|
||||
|
||||
var validation *catalog.ValidationErrors
|
||||
if !errors.As(err, &validation) || len(validation.Issues) == 0 {
|
||||
t.Fatalf("validation error = %#v", err)
|
||||
}
|
||||
|
||||
if validation.Issues[0].Path == "" || validation.Issues[0].Line == 0 {
|
||||
t.Fatalf("validation issue = %#v", validation.Issues[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestCrossValidationRejectsUnknownBackupMount(t *testing.T) {
|
||||
body, err := catalogdata.Files.ReadFile("palworld/template.yaml")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
name, body, snapshot := embeddedTemplate(t)
|
||||
normalized := strings.ReplaceAll(string(body), "\r\n", "\n")
|
||||
body = []byte(strings.Replace(
|
||||
normalized,
|
||||
" - saved\n restart_after_backup",
|
||||
" - missing\n restart_after_backup",
|
||||
1,
|
||||
))
|
||||
|
||||
_, err = catalog.Validate(body, "palworld", catalogdata.Files)
|
||||
body = []byte(strings.Replace(normalized, " - "+snapshot.Template.Backup.SourceMounts[0]+"\n", " - missing\n", 1))
|
||||
_, err := catalog.Validate(body, path.Dir(name), catalogdata.Files)
|
||||
|
||||
var validation *catalog.ValidationErrors
|
||||
if !errors.As(err, &validation) {
|
||||
t.Fatalf("validation error = %#v", err)
|
||||
}
|
||||
|
||||
found := false
|
||||
for _, issue := range validation.Issues {
|
||||
found = found || issue.Path == "/backup/source_mounts"
|
||||
}
|
||||
|
||||
if !found {
|
||||
t.Fatalf("issues = %#v", validation.Issues)
|
||||
if issue.Path == "/backup/source_mounts" {
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatalf("issues = %#v", validation.Issues)
|
||||
}
|
||||
|
||||
func embeddedTemplate(t *testing.T) (string, []byte, catalog.Snapshot) {
|
||||
t.Helper()
|
||||
names, err := fs.Glob(catalogdata.Files, "*/template.yaml")
|
||||
if err != nil || len(names) == 0 {
|
||||
t.Fatalf("discover embedded template: names=%v err=%v", names, err)
|
||||
}
|
||||
body, err := catalogdata.Files.ReadFile(names[0])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
snapshot, err := catalog.Validate(body, path.Dir(names[0]), catalogdata.Files)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return names[0], body, snapshot
|
||||
}
|
||||
|
||||
@@ -66,11 +66,11 @@
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["source", "destination", "sha256", "read_only"],
|
||||
"required": ["source", "destination", "read_only"],
|
||||
"properties": {
|
||||
"source": { "type": "string", "pattern": "^(?!/)(?!.*\\.\\./).+$", "maxLength": 200 },
|
||||
"destination": { "type": "string", "pattern": "^/[^\\u0000]*$", "maxLength": 500 },
|
||||
"sha256": { "type": "string", "pattern": "^[a-f0-9]{64}$" },
|
||||
"sha256": { "type": "string", "maxLength": 128 },
|
||||
"read_only": { "const": true }
|
||||
}
|
||||
}
|
||||
|
||||
+32
-7
@@ -53,7 +53,7 @@ def validate_links() -> None:
|
||||
raise ValueError("Broken internal links:\n " + "\n ".join(failures))
|
||||
|
||||
|
||||
def validate_cross_references(template: dict, manifest: dict) -> list[str]:
|
||||
def validate_cross_references(template: dict, manifest: dict | None, template_path: Path) -> list[str]:
|
||||
warnings = []
|
||||
port_ids = {item["id"] for item in template["container"]["ports"]}
|
||||
mount_ids = {item["id"] for item in template["storage"]["mounts"]}
|
||||
@@ -61,6 +61,7 @@ def validate_cross_references(template: dict, manifest: dict) -> list[str]:
|
||||
|
||||
integration = template.get("integration")
|
||||
if integration:
|
||||
assert manifest is not None, f"Declared module is missing: {integration['module_id']}"
|
||||
assert integration["port_id"] in port_ids, "Template integration references an unknown port"
|
||||
assert integration["module_id"] == manifest["id"], "Template and manifest module IDs disagree"
|
||||
assert template["game"]["id"] in manifest["game_ids"], "Manifest does not support template game ID"
|
||||
@@ -73,11 +74,15 @@ def validate_cross_references(template: dict, manifest: dict) -> list[str]:
|
||||
if mods.get("supported"):
|
||||
assert mods.get("destination_mount") in mount_ids, "Mods reference an unknown mount"
|
||||
|
||||
for field in ("logo", "image", "poster"):
|
||||
path = (template_path.parent / template["game"]["artwork"][field]).resolve()
|
||||
assert path.is_file(), f"Missing artwork asset {field}: {path}"
|
||||
for asset in template["container"].get("assets", []):
|
||||
path = (ROOT / "catalog" / "palworld" / asset["source"]).resolve()
|
||||
path = (template_path.parent / asset["source"]).resolve()
|
||||
assert path.is_file(), f"Missing packaged asset: {path}"
|
||||
digest = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
assert digest == asset["sha256"], f"Asset checksum mismatch: {path}"
|
||||
|
||||
if manifest is None:
|
||||
return warnings
|
||||
|
||||
wasm_digest = manifest["artifacts"]["sha256"]
|
||||
wasm_path = ROOT / "modules" / manifest["id"] / manifest["artifacts"]["wasm"]
|
||||
@@ -91,6 +96,28 @@ def validate_cross_references(template: dict, manifest: dict) -> list[str]:
|
||||
return warnings
|
||||
|
||||
|
||||
def validate_catalog() -> list[str]:
|
||||
template_schema = ROOT / "specs/template.schema.json"
|
||||
manifest_schema = ROOT / "specs/module-manifest.schema.json"
|
||||
template_paths = sorted((ROOT / "catalog").glob("*/template.yaml"))
|
||||
assert template_paths, "Catalog contains no templates"
|
||||
warnings = []
|
||||
identities = set()
|
||||
for template_path in template_paths:
|
||||
template = validate(template_schema, template_path)
|
||||
identity = (template["id"], template["version"])
|
||||
assert identity not in identities, f"Duplicate template ID and version: {identity[0]}@{identity[1]}"
|
||||
identities.add(identity)
|
||||
integration = template.get("integration")
|
||||
manifest = None
|
||||
if integration:
|
||||
manifest_path = ROOT / "modules" / integration["module_id"] / "manifest.yaml"
|
||||
assert manifest_path.is_file(), f"Declared module manifest is missing: {manifest_path}"
|
||||
manifest = validate(manifest_schema, manifest_path)
|
||||
warnings.extend(validate_cross_references(template, manifest, template_path))
|
||||
return warnings
|
||||
|
||||
|
||||
def validate_coverage() -> None:
|
||||
required = {
|
||||
"Docker agent": "docs/architecture/docker-agent.md",
|
||||
@@ -156,14 +183,12 @@ def validate_workflows() -> None:
|
||||
|
||||
|
||||
def main() -> int:
|
||||
template = validate(ROOT / "specs/template.schema.json", ROOT / "catalog/palworld/template.yaml")
|
||||
manifest = validate(ROOT / "specs/module-manifest.schema.json", ROOT / "modules/palworld-rest/manifest.yaml")
|
||||
validate_compose()
|
||||
validate_workflows()
|
||||
for fixture in ROOT.rglob("*.json"):
|
||||
load_json(fixture)
|
||||
validate_links()
|
||||
warnings = validate_cross_references(template, manifest)
|
||||
warnings = validate_catalog()
|
||||
validate_coverage()
|
||||
print("DoGaMa specification validation passed.")
|
||||
for warning in warnings:
|
||||
|
||||
Reference in New Issue
Block a user