Allow remote templates, instead of needing to add local templates here

.github/BROKEN_APP.md

@@ -7,7 +7,7 @@ Sorry it's playing up! But please read this before opening an issue here - it'll
 ### Where to report it
 
 - **A bug in the app itself** - report it to that **app's own project**.
-- **A wrong template** (dead image, bad port, missing variable) - report it to the **source it came from**. Every app in the [README](https://github.com/lissy93/portainer-templates#supported-apps-and-stacks) has a **"Report issues"** link beside it, pointing straight at its source.
+- **A wrong template** (dead image, bad port, missing variable) - report it to the **source it came from**. Every app in the [README](https://github.com/lissy93/portainer-templates#supported-apps-and-stacks) has a **"Report issues"** link beside it, pointing straight at its source. For the growing number of apps whose own authors publish their template, that link goes directly to them, and their fix reaches us the next day.
 
 Once a fix lands upstream, it flows into our list automatically on the next build.
 

.github/CONTRIBUTING.md

@@ -4,11 +4,79 @@ Thanks for wanting to help out - it's genuinely appreciated! 🙌
 
 This repo compiles Portainer app templates from lots of [sources](../sources.csv) into a single `templates.json`. Most contributions are either adding apps to that list, or improving the tooling around it.
 
-## Adding apps or stacks
+## Adding your app
 
-- **Maintain your own template list?** Add its name and raw URL to the bottom of [`sources.csv`](../sources.csv), and the next build will pull it in. Sources are in priority order, so the first one listed wins when the same app appears in several.
-- **Just have a template or two?** Drop a JSON file into [`sources/local/`](../sources/local). It needs to match [Portainer's template format](https://docs.portainer.io/advanced/app-templates/format) - there's a [`Schema.json`](../Schema.json) you can check against.
-- **Adding a docker-compose stack?** Put the compose file in [`sources/stacks/`](../sources/stacks) and point your template at it.
+If you've built a self-hosted app and want it in the list, you have two options. The first is much less hassle for you, and it's the one we'd recommend.
+
+### Option 1: publish your own template (recommended)
+
+Keep the template **in your own repo**, and tell us where to find it. We re-fetch it every day, so after the one PR that registers it you never need to come back here again - edit your own file, and the change is live within 24 hours.
+
+**1.** Add a JSON file to your repo (`portainer-template.json` is a good name). A single app is just the template object on its own:
+
+```json
+{
+  "type": 1,
+  "title": "My App",
+  "description": "What it does, in a sentence or two.",
+  "categories": ["Productivity"],
+  "platform": "linux",
+  "logo": "https://raw.githubusercontent.com/you/my-app/main/logo.png",
+  "image": "ghcr.io/you/my-app:latest",
+  "restart_policy": "unless-stopped",
+  "ports": ["8080:8080/tcp"],
+  "volumes": [{ "container": "/app/data" }],
+  "env": [{ "name": "TZ", "label": "Timezone", "default": "UTC" }],
+  "note": "Anything people need to know to get it running."
+}
+```
+
+**2.** Check it, by pointing the validator at your raw URL:
+
+```bash
+python lib/validate_sources.py https://raw.githubusercontent.com/you/my-app/main/portainer-template.json
+```
+
+**3.** Add one row to the bottom of [`sources.csv`](../sources.csv), ending in `app`:
+
+```csv
+my_app, https://raw.githubusercontent.com/you/my-app/main/portainer-template.json, https://github.com/you/my-app/, app
+```
+
+That's `name, url, maintainer, kind`. The name has to be unique and lowercase (it becomes a file name), the URL is the raw JSON, and the maintainer link is where people should report problems with your app.
+
+A few things worth knowing:
+
+- **Point at a branch, not a tag or commit SHA**, otherwise your template freezes and the whole point is lost.
+- **Your copy wins.** When your app also appears in one of the big collections, yours is the one we publish - you know your own app best.
+- **Up to 5 templates per app source**, so you can ship a container and a stack, or a plain and a GPU build. More than that is a collection, see below.
+- **It has to be your own app.** Registering someone else's is what the collections are for.
+- **Compose stacks stay in your repo too.** Use `"type": 3` and point `repository` at your own compose file, and there's nothing for us to keep in sync:
+  ```json
+  { "type": 3, "title": "My App", "description": "...",
+    "repository": { "url": "https://github.com/you/my-app", "stackfile": "docker-compose.yml" } }
+  ```
+- **Already sent us a template?** If your app is currently in [`sources/local/`](../sources/local), delete it in the same PR that adds your row. Local copies beat everything, so leaving it there means your updates are silently ignored. The build warns when that happens, but it's easier to just remove it.
+
+### Option 2: let us host it
+
+If you'd rather not keep a file in your own repo, drop a JSON file into [`sources/local/`](../sources/local) instead, matching [Portainer's template format](https://docs.portainer.io/advanced/app-templates/format). A docker-compose stack goes in [`sources/stacks/`](../sources/stacks), with your template pointing at it.
+
+The catch is that it becomes ours to maintain, and every change to it needs another PR from you. Fine for a one-off, annoying if your app moves quickly.
+
+## Maintaining a whole list of templates
+
+If you publish a template list covering **lots of apps**, add it to [`sources.csv`](../sources.csv) with `collection` as the kind (or leave the kind blank, which means the same thing):
+
+```csv
+your_templates, https://raw.githubusercontent.com/you/templates/main/templates.json, https://github.com/you/templates/, collection
+```
+
+Collections are kept in the order they're listed, and the first one listed wins when the same app turns up in several. Overall, duplicates are settled in this order:
+
+1. `sources/local/` - the copies we maintain
+2. `app` sources - the author's own template for their own app
+3. `collection` sources, in the order they appear in `sources.csv`
 
 More detail on all of this is in the [Editing](README.md#editing) section of the README.
 
@@ -59,6 +127,12 @@ make validate_sources       # validate your templates, stacks + sources list
 make                        # or build the whole thing end-to-end
 ```
 
+If you're publishing your own template, you can check it straight from its URL, without cloning anything into this repo:
+
+```bash
+python lib/validate_sources.py https://raw.githubusercontent.com/you/my-app/main/portainer-template.json
+```
+
 <!--
 When submitting a PR please follow the PR template (PULL_REQUEST_TEMPLATE.md)
 

.github/PULL_REQUEST_TEMPLATE.md

@@ -4,7 +4,7 @@ So that your PR can be handled quickly, please ensure you've followed .github/CO
 -->
 
 ### Category
-<!-- New template / new source / tooling / docs / workflows / security -->
+<!-- New app source / new template / new collection source / tooling / docs / workflows / security -->
 
 ### Summary
 <!--
@@ -17,3 +17,6 @@ If it relates to an issue, mention the ticket num here (like "closes #420")
 - [ ] I've read the [contributing guide](.github/CONTRIBUTING.md)
 - [ ] I've self-reviewed my changes to ensure they are valid
 - [ ] I'm not modifying `templates.json` directly (it's auto-generated)
+<!-- Only if you're registering your own app in sources.csv with kind `app`: -->
+- [ ] My template URL points at a branch (not a tag or commit), so updates flow through
+- [ ] I've removed my app's old copy from `sources/local/`, if it had one

.github/README.md

@@ -89,16 +89,28 @@ Running the `make` command will download all listed sources, parse them, and com
 _So, you only need to edit **either** the [`sources.csv`](https://github.com/Lissy93/portainer-templates/blob/main/sources.csv) file, **or** add your templates directly into the [`sources/local/`](https://github.com/Lissy93/portainer-templates/tree/main/sources/local) directory._
 
 ### Adding a new Source
-If you're maintaining your own list of Portainer templates, and would like them to be included, just add the URL to your template.json file to the bottom of [`sources.csv`](https://github.com/Lissy93/portainer-templates/blob/main/sources.csv), along with a name.
-When the action runs, it will download the content, parse it and add it to the final template.
-Sources are listed in priority order: if the same app appears in several sources, the one listed first is kept (anything in `sources/local/` beats them all).
+Every row in [`sources.csv`](https://github.com/Lissy93/portainer-templates/blob/main/sources.csv) is `name, url, maintainer, kind`, where `kind` is either:
+
+- `collection` - somebody's maintained list covering many apps (the default, if you leave it blank)
+- `app` - a single self-hosted app, published by its own author
+
+Add a row, and the next build downloads it, parses it and merges it into the final template. Because that happens every night, an `app` author only needs one PR here: from then on they edit the file in their own repo and the change shows up within a day.
+
+If the same app appears more than once, duplicates are settled in this order:
+
+1. `sources/local/` - the copies maintained in this repo
+2. `app` sources - the author's own template for their own app
+3. `collection` sources, in the order they're listed in `sources.csv`
+
+An `app` source may carry up to 5 templates, so an author can ship a container and a stack, or a plain and a GPU build. If one stops resolving it's simply left out of that build, whereas a missing collection stops the build, since it would take hundreds of apps with it.
 
 ### Adding a Template / Template list
-Alternatively, place your template file within the [`sources/local/`](https://github.com/Lissy93/portainer-templates/tree/main/sources/local) directory, and it will be automatically combined into the main `template.json`. Be sure that your template corresponds to [Portainer's App Template JSON Format](https://docs.portainer.io/advanced/app-templates/format).
+Alternatively, place your template file within the [`sources/local/`](https://github.com/Lissy93/portainer-templates/tree/main/sources/local) directory, and it will be automatically combined into the main `template.json`. Be sure that your template corresponds to [Portainer's App Template JSON Format](https://docs.portainer.io/advanced/app-templates/format). Note that anything in there becomes ours to maintain, and needs a PR for every change - so if it's your own app, publishing it from your own repo as an `app` source is usually the better deal.
 
 ### Validating Templates
 There is a schema defined in [`Schema.json`](https://github.com/Lissy93/portainer-templates/blob/main/Schema.json), which can be used to validate any Portainer template.
 Run `make validate` to ensure your template conforms to Portainer's App Template [specification](https://docs.portainer.io/advanced/app-templates/format).
+To check a template you host yourself, point the validator straight at its URL with `python lib/validate_sources.py <url>`.
 
 ### Maintaining your own Templates
 If you'd like to use this repo as a base, but maintain your own list of template, just fork the repository, and update `lissy93` with your username in the import URL.
@@ -928,6 +940,10 @@ If you'd like to use this repo as a base, but maintain your own list of template
 Full credit to the authors of the following templates.
 The main `templates.json` file is composes of these sources, along with the content of the [`sources`](https://github.com/Lissy93/portainer-templates/tree/main/sources) directory.
 
+### Template Collections
+
+Community-maintained lists, each covering many apps.
+
 <!-- auto-insert-sources:start -->
 1. <img src="https://github.com/portainer.png?size=40" width="26" height="26" /> [template](https://raw.githubusercontent.com/portainer/templates/v3/templates.json) by [@portainer](https://github.com/portainer)
 2. <img src="https://github.com/technorabilia.png?size=40" width="26" height="26" /> [template](https://raw.githubusercontent.com/technorabilia/portainer-templates/main/lsio/templates/templates.json) by [@technorabilia](https://github.com/technorabilia)
@@ -939,10 +955,18 @@ The main `templates.json` file is composes of these sources, along with the cont
 8. <img src="https://github.com/mikestraney.png?size=40" width="26" height="26" /> [template](https://raw.githubusercontent.com/mikestraney/portainer-templates/master/templates.json) by [@mikestraney](https://github.com/mikestraney)
 9. <img src="https://github.com/SelfhostedPro.png?size=40" width="26" height="26" /> [template](https://raw.githubusercontent.com/SelfhostedPro/selfhosted_templates/portainer-2.0/Template/template.json) by [@SelfhostedPro](https://github.com/SelfhostedPro)
 10. <img src="https://github.com/dnburgess.png?size=40" width="26" height="26" /> [template](https://raw.githubusercontent.com/dnburgess/self-hosted-template/master/template.json) by [@dnburgess](https://github.com/dnburgess)
-11. [template](https://deployable.sh/portainer/templates.json)
+11. <img src="https://github.com/deployable-sh.png?size=40" width="26" height="26" /> [template](https://deployable.sh/portainer/templates.json) by [@deployable-sh](https://github.com/deployable-sh)
 
 <!-- auto-insert-sources:end -->
 
+### Individual Apps
+
+Self-hosted apps whose own authors publish and maintain their Portainer template, which we pull in fresh every day.
+
+<!-- auto-insert-app-sources:start -->
+_None yet - if you maintain a self-hosted app, [add yours](CONTRIBUTING.md#option-1-publish-your-own-template-recommended)._
+<!-- auto-insert-app-sources:end -->
+
 </details>
 
 ---

.github/workflows/pr-check.yml

@@ -118,6 +118,8 @@ jobs:
         uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
+          # Deep enough to diff sources.csv against the PR's base
+          fetch-depth: 0
 
       - name: Set up Python
         uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
@@ -132,6 +134,32 @@ jobs:
       - name: Validate sources.csv + overrides.json
         run: python lib/validate_sources.py sources.csv overrides.json
 
+      # Fetch any source URL this PR adds and check it the way the build will,
+      # so a broken template is caught here rather than in the nightly build
+      - name: Validate newly added sources
+        if: github.event_name == 'pull_request'
+        env:
+          BASE_SHA: ${{ github.event.pull_request.base.sha }}
+        run: |
+          set -euo pipefail
+          # -d '[:blank:]' strips spaces/tabs but keeps the newline between urls
+          added=$(git diff "$BASE_SHA" -- sources.csv \
+                  | grep '^+[^+]' \
+                  | cut -d, -f2 \
+                  | tr -d '[:blank:]' \
+                  | grep -E '^https?://' || true)
+          if [ -z "$added" ]; then
+            echo "No new sources in this PR."
+            exit 0
+          fi
+          status=0
+          for url in $added; do
+            echo "::group::$url"
+            python lib/validate_sources.py "$url" || status=1
+            echo "::endgroup::"
+          done
+          exit $status
+
   # When the tooling or schema changes: compile, self-check the schema, re-validate output
   scripts:
     name: 🐍 Scripts + Schema

lib/combine.py

@@ -1,4 +1,3 @@
-import csv
 import json
 import os
 import re
@@ -8,6 +7,7 @@ from collections import Counter
 
 import jsonschema
 
+import sources_list
 from log import get_logger, banner
 
 log = get_logger()
@@ -19,6 +19,8 @@ with open(os.path.join(BASE_DIR, 'Schema.json')) as f:
   SCHEMA = json.load(f)
 FORMAT_CHECKER = jsonschema.FormatChecker()
 TEMPLATE_KEYS = set(SCHEMA['properties']['templates']['items']['properties'])
+ITEM_VALIDATOR = jsonschema.Draft7Validator(SCHEMA['properties']['templates']['items'],
+                                            format_checker=FORMAT_CHECKER)
 
 TYPE_LABELS = {1: 'container', 2: 'swarm', 3: 'stack', 4: 'edge'}
 
@@ -53,11 +55,12 @@ def template_rank(t, pinned=None):
   priority, then completeness."""
   return (t['_source'] == pinned, -t['_priority'], template_score(t))
 
-def listed_sources():
-  """File names of the external sources in sources.csv, in the order listed."""
-  with open(os.path.join(BASE_DIR, 'sources.csv')) as f:
-    return [row[0].strip() + '.json' for row in csv.reader(f)
-            if len(row) > 1 and row[0].strip() and row[1].strip()]
+def source_priority(sources):
+  """File name -> dedup priority, where lower wins. An app source outranks every
+  collection, because its author maintains that one app and knows it best.
+  Within each kind, the order they're listed in sources.csv decides."""
+  ordered = [s for s in sources if s.is_app] + [s for s in sources if not s.is_app]
+  return {s.filename: i for i, s in enumerate(ordered, start=1)}
 
 def load_overrides():
   """Per-template fixes from overrides.json, keyed by (normalized title, type)."""
@@ -78,9 +81,9 @@ def pinned_source(overrides, key):
   prefer = overrides.get(key, {}).get('prefer')
   return f'{prefer}.json' if prefer else None
 
-def load_sources():
+def load_sources(sources):
   """Load and merge all template JSON files from sources/local/ and sources/external/."""
-  rank = {name: i for i, name in enumerate(listed_sources(), start=1)}
+  rank = source_priority(sources)
   templates = []
   local_dir = os.path.join(SOURCES_DIR, 'local')
   external_dir = os.path.join(SOURCES_DIR, 'external')
@@ -244,6 +247,14 @@ def normalize_template_fields(templates):
       log.warning(f'Skipping unnormalizable template: {t.get("title", "<no title>")} ({err})')
   return normalized
 
+def schema_errors(t):
+  """Schema violations in a single template. Checked per template so that one bad
+  entry from an upstream source gets dropped, rather than failing the whole build
+  at the final validation."""
+  probe = {k: v for k, v in t.items() if not k.startswith('_')}
+  probe.setdefault('id', 1)  # combine assigns real ids at write time
+  return [e.message for e in ITEM_VALIDATOR.iter_errors(probe)]
+
 def is_valid_template(t):
   """Check a template has the required fields for its type."""
   if not (isinstance(t.get('title'), str) and t['title'].strip()):
@@ -301,6 +312,11 @@ def deduplicate_and_normalize(templates, overrides=None):
     if not is_valid_template(t):
       log.warning(f'Skipping invalid template: {t.get("title", "<no title>")}')
       continue
+    problems = schema_errors(t)
+    if problems:
+      log.warning(f'Skipping template that fails the schema: {t["title"]} '
+                  f'from {t["_source"]} ({problems[0]})')
+      continue
     key = (normalize_string(t['title']), t.get('type', 1))
     pinned = pinned_source(overrides, key)
     if key not in best or template_rank(t, pinned) > template_rank(best[key], pinned):
@@ -375,18 +391,39 @@ def audit_overrides(templates, overrides):
     elif max(candidates, key=template_rank) is max(candidates, key=lambda t: template_rank(t, pinned)):
       log.warning(f'Redundant override: normal priority already picks {prefer} for {where}')
 
-def missing_sources():
-  """Names from sources.csv with no downloaded file in sources/external/."""
+def audit_app_sources(templates, sources):
+  """An author who publishes their own template, but whose app also still sits in
+  sources/local/, never gets their updates published: local wins every time.
+  Warn, so the stale local copy can be deleted."""
+  local = {}
+  for t in templates:
+    if t['_priority'] == 0 and isinstance(t.get('title'), str):
+      local.setdefault((normalize_string(t['title']), t.get('type', 1)), t['_source'])
+  app_files = {s.filename for s in sources if s.is_app}
+  for t in templates:
+    if t['_source'] not in app_files or not isinstance(t.get('title'), str):
+      continue
+    shadowed_by = local.get((normalize_string(t['title']), t.get('type', 1)))
+    if shadowed_by:
+      log.warning(f'{t["title"]} from the {t["_source"]} app source is shadowed by the '
+                  f'copy in sources/local/{shadowed_by}, so the author\'s updates never '
+                  'get published. Delete the local copy to let them through.')
+
+def missing_sources(sources, kind):
+  """Sources of one kind that sources.csv lists but nothing was downloaded for."""
   external_dir = os.path.join(SOURCES_DIR, 'external')
   present = set(os.listdir(external_dir)) if os.path.isdir(external_dir) else set()
-  return set(listed_sources()) - present
+  return sorted(s.name for s in sources if s.kind == kind and s.filename not in present)
 
 if __name__ == '__main__':
   banner('Combine', 'Merge, normalize + dedupe template sources into templates.json')
   overrides = load_overrides()
-  raw = normalize_template_fields(load_sources())
+  sources = sources_list.load()
+  raw = normalize_template_fields(load_sources(sources))
   sources_count = len({t.get('_source') for t in raw})
   log.info(f'Normalized {len(raw)} templates from {sources_count} sources')
+  if any(s.is_app for s in sources):
+    audit_app_sources(raw, sources)
   if overrides:
     log.info(f'Applying {len(overrides)} template overrides')
     audit_overrides(raw, overrides)
@@ -412,10 +449,15 @@ if __name__ == '__main__':
       previous = len(json.load(f)['templates'])
   except (OSError, ValueError, KeyError):
     previous = 0
-  # A failed source download must not silently shrink the published list
-  missing = missing_sources()
+  # An app source going away costs only its own app, so it just gets left out
+  unavailable = missing_sources(sources, sources_list.APP)
+  if unavailable:
+    log.warning(f'{len(unavailable)} app sources were not downloaded, so they are left '
+                f'out of this build: {", ".join(unavailable)}')
+  # A failed collection download must not silently shrink the published list
+  missing = missing_sources(sources, sources_list.COLLECTION)
   if missing and not os.environ.get('ALLOW_SHRINK'):
-    log.error(f'Refusing to write: missing external sources: {", ".join(sorted(missing))}. '
+    log.error(f'Refusing to write: missing external sources: {", ".join(missing)}. '
               'Set ALLOW_SHRINK=1 if this is intentional.')
     sys.exit(1)
   if len(templates) < previous * 0.9 and not os.environ.get('ALLOW_SHRINK'):

lib/download.py

@@ -1,10 +1,8 @@
 import os
-import csv
 import sys
-import time
-import requests
 import json
 
+import sources_list
 from log import get_logger, banner
 
 log = get_logger()
@@ -12,75 +10,58 @@ log = get_logger()
 dir = os.path.dirname(os.path.abspath(__file__))
 
 destination_dir = os.path.join(dir, '../sources/external')
-sources_list = os.path.join(dir, '../sources.csv')
 
-# Downloads the templates file from a given URL, to the local destination
-def download(url: str, filename: str, maintainer: str):
-    file_path = os.path.join(destination_dir, filename)
-    log.info(f'Downloading {url}')
-    sourceJson = None
-    for attempt in range(3):
-        try:
-            r = requests.get(url, timeout=30)
-            r.raise_for_status()
-            sourceJson = r.json()
-            break
-        except (requests.RequestException, ValueError) as err:
-            if attempt == 2:
-                log.warning(f'Skipping source due to an error: {url} ({err})')
-                return False
-            time.sleep(2 ** attempt)
+# Downloads the templates file for a given source, to the local destination
+def download(source):
+    log.info(f'Downloading {source.url}')
+    payload, error = sources_list.fetch_json(source.url)
+    if error:
+        log.warning(f'Skipping source due to an error: {source.url} ({error})')
+        return False
 
-    # Handle sources without valid data
-    if isinstance(sourceJson, list):
-        sourceJson = {'templates': sourceJson}
-    templates = sourceJson.get('templates') if isinstance(sourceJson, dict) else None
-    if not isinstance(templates, list) or not templates:
-        log.warning(f'Skipping source with no templates: {url}')
+    # An app source may publish a bare template object, since it's just the one
+    templates = sources_list.templates_in(payload, allow_bare=source.is_app)
+    if not templates:
+        log.warning(f'Skipping source with no templates: {source.url}')
+        return False
+
+    # 'app' means one author's own app, not a back door for an unreviewed list
+    if source.is_app and len(templates) > sources_list.MAX_APP_TEMPLATES:
+        log.warning(f'Skipping {source.name}: an app source may hold at most '
+                    f'{sources_list.MAX_APP_TEMPLATES} templates, but this one has '
+                    f'{len(templates)}, so it belongs in sources.csv as a collection')
         return False
 
     # Add maintainer field to each template
     for t in templates:
-        if isinstance(t, dict) and maintainer:
-            t['maintainer'] = maintainer
+        if isinstance(t, dict) and source.maintainer:
+            t['maintainer'] = source.maintainer
 
+    file_path = os.path.join(destination_dir, source.filename)
     log.debug(f'Saving to {os.path.abspath(file_path)}')
     with open(file_path, 'w') as f:
-        json.dump(sourceJson, f, indent=2, sort_keys=False)
+        json.dump({'templates': templates}, f, indent=2, sort_keys=False)
     return True
 
-# Gets list of URLs to download from CSV file
-def get_source_list():
-  sources = []
-  seen = set()
-  with open(sources_list, mode='r') as file:
-      csvFile = csv.reader(file)
-      for lines in csvFile:
-        row = [col.strip() for col in lines]
-        if len(row) < 2 or not row[0] or not row[1]:
-          if any(row):
-            log.warning(f'Skipping malformed sources.csv row: {lines}')
-          continue
-        if row[0] in seen:
-          log.warning(f'Skipping duplicate source: {row[0]}')
-          continue
-        seen.add(row[0])
-        sources.append(row)
-  return sources
-
 # Create destination folder if not yet present
 if not os.path.exists(destination_dir):
   os.makedirs(destination_dir)
 
 banner('Download', 'Fetch template sources listed in sources.csv')
 
-sources = get_source_list()
-failures = []
-for source in sources:
-  if not download(source[1], source[0] + '.json', source[2] if len(source) > 2 else ''):
-    failures.append(source[0])
+sources = sources_list.load()
+failures = [source for source in sources if not download(source)]
 
 log.info(f'Downloaded {len(sources) - len(failures)}/{len(sources)} sources')
-if failures:
-  log.error(f'Failed to download valid sources: {", ".join(failures)}')
+
+# A collection carries hundreds of apps, so losing one is worth stopping for.
+# An app source costs only its own app, and there are far more of them to go
+# stale, so the build goes on without it.
+failed_apps = [source.name for source in failures if source.is_app]
+failed_collections = [source.name for source in failures if not source.is_app]
+if failed_apps:
+  log.warning(f'Leaving out {len(failed_apps)} unavailable app '
+              f'sources: {", ".join(failed_apps)}')
+if failed_collections:
+  log.error(f'Failed to download valid sources: {", ".join(failed_collections)}')
   sys.exit(1)

lib/list.py

@@ -2,10 +2,10 @@ import json
 import html
 import urllib.parse
 import os
-import csv
 import re
 import sys
 
+import sources_list
 from log import get_logger, banner
 
 log = get_logger()
@@ -14,16 +14,12 @@ current_dir = os.path.dirname(os.path.abspath(__file__))
 project_dir = os.path.dirname(current_dir)
 readme_path = os.path.join(project_dir, '.github/README.md')
 templates_path = os.path.join(project_dir, 'templates.json')
-sources_path = os.path.join(project_dir, 'sources.csv')
+external_dir = os.path.join(project_dir, 'sources/external')
 
 def load_json_file(file_path):
     with open(file_path, 'r') as file:
         return json.load(file)
 
-def load_csv_file(file_path):
-    with open(file_path, 'r') as file:
-        return list(csv.reader(file))
-
 def slugify(title: str):
     baseUrl = 'https://portainer-templates.as93.net'
     slug = re.sub(r'\s+', '-', re.sub(r'[^a-z0-9 ]', '', title.lower()).strip())
@@ -50,25 +46,47 @@ def generate_app_list():
       markdown_content += f"{index+1}. {logo}**[{name}]({slugify(name)} '{description}')** {maintainer_md_link}\n"
   return markdown_content
 
-def generate_sources_list():
-    sources = load_csv_file(sources_path)
-    markdown_content = ''
+def github_user(*urls):
+    """The GitHub account behind a source, for its avatar and credit link. Falls back to
+    the maintainer link, since a source's JSON may be served from anywhere."""
+    for url in urls:
+        parsed = urllib.parse.urlparse(url or '')
+        path_parts = [p for p in parsed.path.split('/') if p]
+        if parsed.hostname in ('github.com', 'raw.githubusercontent.com') and path_parts:
+            return path_parts[0]
+    return None
 
-    count = 0
-    for source in sources:
-        if len(source) > 1 and source[1].strip():
-          count += 1
-          url = source[1].strip()
-          parsed_url = urllib.parse.urlparse(url)
-          path_parts = [p for p in parsed_url.path.split('/') if p]
-          if parsed_url.hostname in ('github.com', 'raw.githubusercontent.com') and path_parts:
-            username = path_parts[0]
-            avatar = f'<img src="https://github.com/{username}.png?size=40" width="26" height="26" />'
-            markdown_content += f"{count}. {avatar} [template]({url}) by [@{username}](https://github.com/{username})\n"
-          else:
-            markdown_content += f"{count}. [template]({url})\n"
+def source_credit(source, label):
+    """One line of a README credit list: avatar, link and the author's handle"""
+    username = github_user(source.url, source.maintainer)
+    if not username:
+        return f'[{label}]({source.url})'
+    avatar = f'<img src="https://github.com/{username}.png?size=40" width="26" height="26" />'
+    return (f'{avatar} [{label}]({source.url}) by '
+            f'[@{username}](https://github.com/{username})')
 
-    return markdown_content
+def app_label(source):
+    """An app source's own title, read back from what was downloaded, so the credit reads
+    "Cantinarr" rather than "cantinarr". Falls back to the source name when unfetched."""
+    try:
+        templates = load_json_file(os.path.join(external_dir, source.filename))['templates']
+        titles = [t['title'] for t in templates if isinstance(t, dict) and t.get('title')]
+    except (OSError, ValueError, KeyError):
+        titles = []
+    return titles[0] if titles else source.name
+
+def generate_sources_list(sources):
+    """The collections: other people's maintained lists of templates"""
+    return ''.join(f'{index}. {source_credit(source, "template")}\n'
+                   for index, source in enumerate(sources, start=1))
+
+def generate_app_sources_list(sources):
+    """The individual apps, each published and kept up to date by its own author"""
+    if not sources:
+        return ('_None yet - if you maintain a self-hosted app, '
+                '[add yours](CONTRIBUTING.md#option-1-publish-your-own-template-recommended)._')
+    return ''.join(f'{index}. {source_credit(source, app_label(source))}\n'
+                   for index, source in enumerate(sources, start=1))
 
 def insert_content_between_markers(file_path, start_marker, end_marker, content_to_insert):
     with open(file_path, 'r') as file:
@@ -95,8 +113,10 @@ def insert_content_between_markers(file_path, start_marker, end_marker, content_
 
 banner('List', 'Render app + source lists into the README')
 
-# Insert sources list into readme
-sources_md = generate_sources_list()
+all_sources = sources_list.load()
+
+# Insert the collection sources list into readme
+sources_md = generate_sources_list([s for s in all_sources if not s.is_app])
 insert_content_between_markers(
   readme_path,
   '<!-- auto-insert-sources:start -->',
@@ -105,6 +125,16 @@ insert_content_between_markers(
 )
 log.info(f'Rendered {sources_md.count(chr(10))} sources into README')
 
+# Insert the individual app sources list into readme
+app_sources = [s for s in all_sources if s.is_app]
+insert_content_between_markers(
+  readme_path,
+  '<!-- auto-insert-app-sources:start -->',
+  '<!-- auto-insert-app-sources:end -->',
+  generate_app_sources_list(app_sources),
+)
+log.info(f'Rendered {len(app_sources)} individual app sources into README')
+
 # Insert app list into readme
 apps_md = generate_app_list()
 insert_content_between_markers(

lib/sources_list.py

@@ -0,0 +1,133 @@
+"""Reading sources.csv, and fetching what it points at, in one place.
+
+Each row is: name, url, maintainer, kind
+
+  name        unique, lowercase; becomes sources/external/<name>.json
+  url         http(s) link to the raw JSON
+  maintainer  the source's home page, shown as its "Report issues" link
+  kind        'collection' for someone's maintained list of apps, 'app' for a
+              single app published by its own author. Blank means collection,
+              so the original three-column rows still read correctly.
+
+The build trusts the two kinds differently. An app source outranks every
+collection when the same app appears in both, since its author maintains that
+one app and knows it best. In exchange it may only carry a handful of
+templates, and if it stops resolving the build carries on without it.
+"""
+import csv
+import os
+import re
+import time
+from typing import NamedTuple
+from urllib.parse import urlparse
+
+from log import get_logger
+
+log = get_logger()
+
+SOURCES_CSV = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..', 'sources.csv')
+
+COLLECTION = 'collection'
+APP = 'app'
+KINDS = (COLLECTION, APP)
+
+# A source name becomes a file name, so keep it to something unambiguous
+NAME_PATTERN = re.compile(r'^[a-z0-9][a-z0-9_-]*$')
+
+# An app may reasonably ship a container and a stack, or a plain build and a
+# GPU one. Much past that is a collection, and belongs in one.
+MAX_APP_TEMPLATES = 5
+
+class Source(NamedTuple):
+  """One usable row of sources.csv"""
+  name: str
+  url: str
+  maintainer: str
+  kind: str
+
+  @property
+  def filename(self):
+    """Where download.py saves it, and the name combine.py ranks it by"""
+    return f'{self.name}.json'
+
+  @property
+  def is_app(self):
+    return self.kind == APP
+
+def rows(path=SOURCES_CSV):
+  """Every non-blank row, as (line number, stripped cells)"""
+  with open(path, newline='') as file:
+    for line, row in enumerate(csv.reader(file), start=1):
+      cells = [cell.strip() for cell in row]
+      if any(cells):
+        yield line, cells
+
+def row_problems(cells):
+  """Everything wrong with a row, as (level, message) pairs. Any problem at all
+  means load() skips the row; the level only decides how loudly to report it."""
+  if len(cells) < 2 or not cells[0] or not cells[1]:
+    return [('warning', f'malformed row, would be skipped: {cells}')]
+  problems = []
+  if not NAME_PATTERN.match(cells[0]):
+    problems.append(('error', f'name {cells[0]!r} becomes a file name, so it may only hold '
+                              "lowercase letters, digits, '_' and '-'"))
+  if urlparse(cells[1]).scheme not in ('http', 'https'):
+    problems.append(('error', f'{cells[0]} has a non-http(s) url: {cells[1]}'))
+  kind = cells[3] if len(cells) > 3 else ''
+  if kind and kind.lower() not in KINDS:
+    problems.append(('error', f'{cells[0]} has kind {kind!r}, expected '
+                              f'{" or ".join(KINDS)}'))
+  if len(cells) > 4 and any(cells[4:]):
+    problems.append(('error', f'{cells[0]} has {len(cells)} columns, expected at most 4 '
+                              '(name, url, maintainer, kind)'))
+  return problems
+
+def parse(cells):
+  """A Source from a row already known to be well-formed"""
+  kind = cells[3].lower() if len(cells) > 3 and cells[3] else COLLECTION
+  maintainer = cells[2] if len(cells) > 2 else ''
+  return Source(cells[0], cells[1], maintainer, kind)
+
+def load(path=SOURCES_CSV):
+  """Every usable source, in listed order. Rows with problems are skipped here
+  and reported by validate_sources.py, which is what gates a PR."""
+  sources, seen = [], set()
+  for line, cells in rows(path):
+    problems = row_problems(cells)
+    if problems:
+      for _, message in problems:
+        log.warning(f'Skipping sources.csv:{line}: {message}')
+      continue
+    if cells[0] in seen:
+      log.warning(f'Skipping sources.csv:{line}: duplicate source {cells[0]!r}')
+      continue
+    seen.add(cells[0])
+    sources.append(parse(cells))
+  return sources
+
+def fetch_json(url, attempts=3):
+  """Fetch and parse JSON from a url, retrying a couple of times on a blip.
+  Returns (payload, error message), of which exactly one is set."""
+  import requests  # only the fetching path needs it, so keep it off the parsers
+  for attempt in range(attempts):
+    try:
+      response = requests.get(url, timeout=30)
+      response.raise_for_status()
+      return response.json(), None
+    except (requests.RequestException, ValueError) as err:
+      if attempt == attempts - 1:
+        return None, str(err)
+      time.sleep(2 ** attempt)
+
+def templates_in(payload, allow_bare=False):
+  """The template list inside a downloaded source, whatever shape it arrived in,
+  or None if there isn't one. `allow_bare` accepts a single unwrapped template
+  object, which is the natural way to publish just the one app."""
+  if isinstance(payload, list):
+    return payload
+  if isinstance(payload, dict):
+    if isinstance(payload.get('templates'), list):
+      return payload['templates']
+    if allow_bare and isinstance(payload.get('title'), str):
+      return [payload]
+  return None

lib/validate_sources.py

@@ -1,13 +1,12 @@
 import copy
-import csv
 import json
 import os
 import sys
 from collections import Counter
-from urllib.parse import urlparse
 
 from jsonschema import Draft7Validator, FormatChecker
 
+import sources_list
 from combine import normalize_template, normalize_string
 from log import get_logger, banner
 
@@ -44,18 +43,8 @@ def quality_warnings(where, template):
                  for name, count in Counter(names).items() if count > 1]
     return warnings
 
-def check_source(path):
-    """Validate one local source: each template must normalize to a schema-valid entry"""
-    try:
-        data = load_json(path)
-    except (OSError, json.JSONDecodeError) as err:
-        return [f'{path}: could not read JSON ({err})'], []
-
-    templates = data.get('templates') if isinstance(data, dict) else None
-    if not isinstance(templates, list):
-        return [f'{path}: missing top-level "templates" array'], []
-
-    name = os.path.basename(path)
+def check_templates(name, templates):
+    """Validate a list of templates: each must normalize to a schema-valid entry"""
     errors, warnings, titles = [], [], Counter()
     if not templates:
         warnings.append(f'{name}: contains no templates')
@@ -84,6 +73,38 @@ def check_source(path):
                for title, count in titles.items() if count > 1]
     return errors, warnings
 
+def check_source(path):
+    """Validate one local source: each template must normalize to a schema-valid entry"""
+    try:
+        data = load_json(path)
+    except (OSError, json.JSONDecodeError) as err:
+        return [f'{path}: could not read JSON ({err})'], []
+
+    templates = data.get('templates') if isinstance(data, dict) else None
+    if not isinstance(templates, list):
+        return [f'{path}: missing top-level "templates" array'], []
+
+    return check_templates(os.path.basename(path), templates)
+
+def check_url(url):
+    """Fetch a remote source and check it the way the nightly build will, so an author
+    can test their own template before opening a PR"""
+    payload, error = sources_list.fetch_json(url)
+    if error:
+        return [f'{url}: could not fetch valid JSON ({error})'], []
+
+    templates = sources_list.templates_in(payload, allow_bare=True)
+    if templates is None:
+        return [f'{url}: found no templates. Publish a single template as a JSON object, '
+                'or several under a top-level "templates" array'], []
+
+    errors, warnings = check_templates(url, templates)
+    if len(templates) > sources_list.MAX_APP_TEMPLATES:
+        warnings.append(f'{url}: holds {len(templates)} templates, so it can only be listed '
+                        f'as a collection, not an app source (which may carry at most '
+                        f'{sources_list.MAX_APP_TEMPLATES})')
+    return errors, warnings
+
 def check_stack(path):
     """A compose stack file must be valid YAML with a non-empty services mapping"""
     try:
@@ -100,36 +121,43 @@ def check_stack(path):
     warnings = [f'{path}: obsolete top-level "version" key'] if 'version' in doc else []
     return [], warnings
 
+def local_source_names():
+    """File names (minus .json) in sources/local/. A source in sources.csv must not reuse
+    one, since combine.py tells sources apart by file name alone"""
+    local_dir = os.path.join(ROOT, 'sources', 'local')
+    if not os.path.isdir(local_dir):
+        return set()
+    return {os.path.splitext(f)[0] for f in os.listdir(local_dir) if f.endswith('.json')}
+
 def check_csv(path):
-    """Validate sources.csv: each row needs a unique name and an http(s) url"""
+    """Validate sources.csv: each row needs a unique, file-name-safe name, an http(s)
+    url, and a recognized kind"""
     try:
-        with open(path, newline='') as file:
-            rows = list(csv.reader(file))
+        rows = list(sources_list.rows(path))
     except OSError as err:
         return [f'{path}: could not read ({err})'], []
 
+    local_names = local_source_names()
     errors, warnings, seen = [], [], set()
-    for line, row in enumerate(rows, start=1):
-        cells = [cell.strip() for cell in row]
-        if not any(cells):
+    for line, cells in rows:
+        for level, message in sources_list.row_problems(cells):
+            (errors if level == 'error' else warnings).append(f'{path}:{line}: {message}')
+        name = cells[0]
+        if not name:
             continue
-        if len(cells) < 2 or not cells[0] or not cells[1]:
-            warnings.append(f'{path}:{line}: malformed row, would be skipped: {row}')
-            continue
-        name, url = cells[0], cells[1]
         if name in seen:
             errors.append(f'{path}:{line}: duplicate source name {name!r}')
         seen.add(name)
-        if urlparse(url).scheme not in ('http', 'https'):
-            errors.append(f'{path}:{line}: {name} has a non-http(s) url: {url}')
+        if name in local_names:
+            errors.append(f'{path}:{line}: name {name!r} collides with '
+                          f'sources/local/{name}.json, which would leave the two '
+                          'indistinguishable when picking between duplicate templates')
     return errors, warnings
 
-def source_names(path):
+def source_names():
     """Source names from sources.csv, used to check an override's "prefer" target exists"""
     try:
-        with open(path, newline='') as file:
-            return {row[0].strip() for row in csv.reader(file)
-                    if len(row) > 1 and row[0].strip() and row[1].strip()}
+        return {source.name for source in sources_list.load()}
     except OSError:
         return set()
 
@@ -145,7 +173,7 @@ def check_overrides(path):
         return [f'{path}: missing top-level "overrides" array'], []
 
     name = os.path.basename(path)
-    known = source_names(os.path.join(ROOT, 'sources.csv'))
+    known = source_names()
     errors, warnings, keys = [], [], Counter()
     for index, entry in enumerate(entries):
         where = f'{name}#{index}'
@@ -175,8 +203,14 @@ def check_overrides(path):
                for (title, kind), count in keys.items() if count > 1]
     return errors, warnings
 
+def is_url(target):
+    return target.startswith(('http://', 'https://'))
+
 def validate_path(path):
-    """Dispatch a path to the right checker, by file name first then extension"""
+    """Dispatch a target to the right checker: remote urls first, then by file name,
+    then by extension"""
+    if is_url(path):
+        return check_url(path)
     if os.path.basename(path) == 'overrides.json':
         return check_overrides(path)
     if path.endswith('.csv'):
@@ -189,7 +223,9 @@ def expand(targets):
     """Turn any directory argument into the source files it contains (recursively)"""
     paths = []
     for target in targets:
-        if os.path.isdir(target):
+        if is_url(target):
+            paths.append(target)
+        elif os.path.isdir(target):
             for root, _, names in os.walk(target):
                 paths += [os.path.join(root, n) for n in names if n.endswith(SOURCE_EXTENSIONS)]
         else:
@@ -203,13 +239,14 @@ def main():
 
     errors, warnings = [], []
     for path in expand(targets):
-        if not os.path.exists(path):
+        if not is_url(path) and not os.path.exists(path):
             log.warning(f'Skipping missing path: {path}')
             continue
         file_errors, file_warnings = validate_path(path)
         errors += file_errors
         warnings += file_warnings
-        log.info(f'Checked {os.path.relpath(path, ROOT)}: '
+        where = path if is_url(path) else os.path.relpath(path, ROOT)
+        log.info(f'Checked {where}: '
                  f'{len(file_errors)} errors, {len(file_warnings)} warnings')
 
     for warning in warnings:

sources.csv

@@ -1,11 +1,11 @@
-portainer_templates, https://raw.githubusercontent.com/portainer/templates/v3/templates.json, https://github.com/portainer/templates/
-technorabilia_templates, https://raw.githubusercontent.com/technorabilia/portainer-templates/main/lsio/templates/templates.json, https://github.com/technorabilia/portainer-templates/
-xneo1_templates, https://raw.githubusercontent.com/xneo1/portainer_templates/master/Template/template.json, https://github.com/xneo1/portainer_templates/
-mediadepot_templates, https://raw.githubusercontent.com/mediadepot/templates/master/portainer.json, https://github.com/mediadepot/templates/
-novaspirit_templates, https://raw.githubusercontent.com/novaspirit/pi-hosted/master/pi-hosted_template/template/portainer-v2.json, https://github.com/novaspirit/pi-hosted/
-qballjos_templates, https://raw.githubusercontent.com/Qballjos/portainer_templates/master/Template/template.json, https://github.com/Qballjos/portainer_templates/
-shmolf_templates, https://raw.githubusercontent.com/shmolf/portainer-templates/main/templates-2.0.json, https://github.com/shmolf/portainer-templates/
-mikestraney_templates, https://raw.githubusercontent.com/mikestraney/portainer-templates/master/templates.json, https://github.com/mikestraney/portainer-templates/
-selfhostedpro_templates, https://raw.githubusercontent.com/SelfhostedPro/selfhosted_templates/portainer-2.0/Template/template.json, https://github.com/SelfhostedPro/selfhosted_templates/
-dnburgess_templates, https://raw.githubusercontent.com/dnburgess/self-hosted-template/master/template.json, https://github.com/dnburgess/self-hosted-template/
-deployable_sh_templates, https://deployable.sh/portainer/templates.json, https://github.com/deployable-sh/stacks/
+portainer_templates, https://raw.githubusercontent.com/portainer/templates/v3/templates.json, https://github.com/portainer/templates/, collection
+technorabilia_templates, https://raw.githubusercontent.com/technorabilia/portainer-templates/main/lsio/templates/templates.json, https://github.com/technorabilia/portainer-templates/, collection
+xneo1_templates, https://raw.githubusercontent.com/xneo1/portainer_templates/master/Template/template.json, https://github.com/xneo1/portainer_templates/, collection
+mediadepot_templates, https://raw.githubusercontent.com/mediadepot/templates/master/portainer.json, https://github.com/mediadepot/templates/, collection
+novaspirit_templates, https://raw.githubusercontent.com/novaspirit/pi-hosted/master/pi-hosted_template/template/portainer-v2.json, https://github.com/novaspirit/pi-hosted/, collection
+qballjos_templates, https://raw.githubusercontent.com/Qballjos/portainer_templates/master/Template/template.json, https://github.com/Qballjos/portainer_templates/, collection
+shmolf_templates, https://raw.githubusercontent.com/shmolf/portainer-templates/main/templates-2.0.json, https://github.com/shmolf/portainer-templates/, collection
+mikestraney_templates, https://raw.githubusercontent.com/mikestraney/portainer-templates/master/templates.json, https://github.com/mikestraney/portainer-templates/, collection
+selfhostedpro_templates, https://raw.githubusercontent.com/SelfhostedPro/selfhosted_templates/portainer-2.0/Template/template.json, https://github.com/SelfhostedPro/selfhosted_templates/, collection
+dnburgess_templates, https://raw.githubusercontent.com/dnburgess/self-hosted-template/master/template.json, https://github.com/dnburgess/self-hosted-template/, collection
+deployable_sh_templates, https://deployable.sh/portainer/templates.json, https://github.com/deployable-sh/stacks/, collection