Skip to the content.

dev-links internals

How this repository works: what lives on which branch, how the site is built, and what each workflow and script does. For the links themselves, see README.md.

Contents


Overview

dev-links is a curated list of developer links, published with GitHub Pages at https://jurakovic.github.io/dev-links/. The site has these pages:

Page URL Source
dev-links / README.md on master
dev-blogs /blogs/ blogs/README.md on master
dev-news /news/ posts/ on dev-news, generated into news/README.md at build time
dev-tools /tools/ the dev-tools branch, copied into tools/ at build time
internals /INTERNALS.html this file, INTERNALS.md on master

The same Markdown files are readable on GitHub, with these differences:

How the pieces fit

cron-job.org, every 6 hours
   │ workflow_dispatch
   ▼
Update Posts workflow ◄─── blogs.json + src/ from master
   │                  ◄─── RSS / Atom feeds
   │ commit posts/*.json + README.md
   ▼
dev-news ───push───► Deploy GitHub Pages workflow ◄───push─── master
                       ▲          │
dev-tools ─────────────┘          ▼
                     jurakovic.github.io/dev-links

There are two loops:

  1. Content, by hand. You edit master: blogs.json, the READMEs, favicons. Scripts in src/ regenerate the generated parts. A push to master deploys the site.
  2. News, automatic. Every 6 hours the Update Posts workflow fetches every feed and commits the updated posts/*.json and a regenerated news page to dev-news. That push deploys the site again.

Branches

Branch Holds Written by
master everything edited by hand: READMEs, blogs.json, favicons, scripts, workflows, Jekyll config you, through PRs
dev-news generated data only: posts/<id>.json and the news page README.md github-actions[bot], “Update posts” commits
dev-tools standalone HTML tools you

master

The default branch.

.github/workflows/
  github_pages.yaml     build and deploy the Pages site
  update_posts.yaml     fetch feeds, update dev-news
_includes/              Jekyll includes: CSS, "Expand all blog posts" button
blogs/README.md         dev-blogs page: every blog in blogs.json
favicons/               16×16 icons, <icon>.png
news/README.md          placeholder, replaced at build time by the news page generated from dev-news
src/                    scripts (not published)
tools/README.md         placeholder, replaced at build time by the dev-tools branch
_config.yml             Jekyll config
blogs.json              every blog: the source of truth
blogs.opml              blogs.json as OPML, generated
INTERNALS.md            this file
README.md               dev-links page

dev-news

.github/workflows/github_pages.yaml   on push, calls master's Deploy GitHub Pages workflow
posts/<blog-id>.json                  the latest posts of one feed
README.md                             the news page as read on GitHub: the 100 newest posts, generated by gen_news.js

The data lives off master so the bot’s commits (several a day, over 1,300 so far) stay out of master’s history, and so news can update without touching content. dev-news holds no scripts. The update workflow takes them from master for each run and deletes them before committing.

The dev-news README.md is the news page as read on GitHub. The site doesn’t use it: the Pages build generates its own from posts/, with links within the site (see Deploy GitHub Pages).

Its links to master content (dev-links, dev-blogs, category sections, favicons) are absolute URLs to the content branch, the one blogs.json came from (Update Posts’ content_branch, normally master): https://github.com/jurakovic/dev-links/blob/<branch>/... and https://raw.githubusercontent.com/jurakovic/dev-links/refs/heads/<branch>/favicons/.... Those files aren’t on dev-news, so these links have to leave it. A link from one dev-news page to another is relative, so on GitHub it stays on the branch being read (dev-news or a copy of it), and on the site within the site.

dev-tools

Static pages with no build step: index.html (the tool list), encrypt/index.html and sp_executesql/index.html. The Pages build copies the branch into tools/.

dev-tools has no workflow, so a push to it does not deploy. Changes go live with the next deploy: the next news update, at most 6 hours later, or a manual run of Deploy GitHub Pages.


Data files

blogs.json (master)

The source of truth for every blog: an array of objects, indented with 4 spaces, ending with a newline. Order matters: within a category, blogs are rendered in file order.

{
    "id": "ardalis",
    "author": "Steve Smith",
    "title": "Ardalis",
    "description": "Founder of NimblePros, long-time Microsoft MVP ...",
    "url": "https://ardalis.com/blog",
    "feed": "https://ardalis.com/index.xml",
    "extras": ["https://blog.nimblepros.com"],
    "github": "https://github.com/ardalis",
    "category": "dotnet",
    "icon": "ardalis",
    "selected": true
}
Field Read by Meaning
id all scripts Unique key. Name of posts/<id>.json, the placeholder <div id>, and the feed name shown on dev-news.
author update_readme.js, gen_opml.js Shown before the title. Can be empty.
title update_readme.js, gen_opml.js Can be empty.
description none Notes about the blog. Not rendered anywhere.
url update_readme.js, gen_opml.js Main link.
feed update_posts.sh, update_readme.js, render_posts.js, gen_opml.js RSS or Atom URL. Empty: no posts are fetched and the blog gets no Latest Posts list.
extras update_readme.js More URLs, one line each.
github update_readme.js A repository URL (owner/repo) is shown inline after url as ([GitHub](...)); a profile URL gets its own line.
category update_readme.js, gen_news.js Section and dev-news tag, see Categories.
icon update_readme.js, gen_news.js favicons/<icon>.png. blank for none.
selected update_readme.js true: also listed on README.md. Every blog is listed on blogs/README.md.

Categories

A blog’s category decides which section of each README it goes into and which tag it gets on dev-news. Each section is wrapped in markers named after the category in upper case:

<!-- BEGIN DOTNET -->
...generated by update_readme.js...
<!-- END DOTNET -->

update_readme.js replaces everything between the markers. Everything outside them is written by hand.

category README.md section (selected blogs) blogs/README.md section (all blogs) dev-news tag, links to
dotnet-main DOTNET-MAIN, in .NET none .NET → README.md#net
sql-main SQL-MAIN, in SQL Server none SQL Server → README.md#sql-server
arch-main ARCH-MAIN, in Architecture none Architecture → README.md#architecture
ai-main AI-MAIN, in AI none AI → README.md#ai
ai AI AI AI → blogs/README.md#blogs-ai
dotnet DOTNET DOTNET .NET → #blogs-net
sql SQL SQL SQL Server → #blogs-sql
general GENERAL GENERAL General → #blogs-general
systems-data SYSTEMS-DATA SYSTEMS-DATA Systems & Data → #blogs-systems
management MANAGEMENT MANAGEMENT Management → #blogs-management
tech-companies none TECH-COMPANIES Tech Companies → #blogs-tech-companies

blogs.opml (master)

Every blog in blogs.json as OPML, for importing into a feed reader. Generated by gen_opml.js. Commit it with every blogs.json change.

posts/<id>.json (dev-news)

One file per blog with a feed, created on the first successful fetch. An array of at most 5 posts, newest first:

[
  {
    "title": "Quoting Kyle Kingsbury",
    "link": "https://simonwillison.net/2026/Nov/1/kyle-kingsbury/",
    "pubDate": "2026-11-01T16:49:04+00:00",
    "date": "2026-11-01T16:49:04+00:00"
  }
]
Field Meaning
title From the feed, refreshed on every run.
link From the feed. Identifies the post: a stored link keeps its stored dates.
pubDate The feed’s date, normalized to ISO 8601 UTC: the same instant, in one format. Kept as the raw feed value only if it can’t be parsed. Never changed after the post is first stored. Nothing displays it.
date What dev-news sorts and displays by, in ISO 8601 UTC. Every post has one. Usually the same as pubDate; the time the post was first seen when the feed’s date can’t be used as is (see fix_post_times.js).

Both dates are written in UTC with an explicit offset, for example 2026-10-28T19:27:07+00:00.

Written only by update_posts.sh, through fix_post_times.js. gen_news.js, render_posts.js and check_pubdates.js read only date.

The Pages build also copies these files into the site, so they are served at /dev-links/posts/<id>.json.


Workflows

Update Posts (dev-news)

File: .github/workflows/update_posts.yaml

Trigger. workflow_dispatch only. The GitHub schedule is commented out: since May 2026 an external cron-job.org job dispatches the workflow every 6 hours. To run it by hand: Actions → Update Posts (dev-news) → Run workflow.

Inputs:

Branches. The branch you run it from (github.ref_name, normally master) supplies only the scripts. Run from a feature branch, it tests that branch’s scripts with master’s blogs.json; set content_branch to the feature branch too to use its blogs.json.

Steps:

  1. Checkout target_branch with full history, using the DEV_LINKS_PAT secret. A push made with the default GITHUB_TOKEN does not start other workflows, so the PAT is what lets the bot’s push trigger the Pages deploy.
  2. Git Config. Bot name and email; core.fileMode false.
  3. Prepare Scripts. git restore --source copies blogs.json from the content branch, and src/gen_news.js, src/update_posts.sh and src/fix_post_times.js from the run’s branch, into the dev-news checkout, then chmod +x src/update_posts.sh.
  4. Build.
    1. ./src/update_posts.sh updates every feed.
    2. node src/gen_news.js <content_branch> regenerates README.md.
    3. The restored files are deleted.
    4. If anything changed, it commits “Update posts” and pushes. The step runs with shell: bash {0} (no -e), so the git diff --cached --quiet check can return non-zero without failing the step.

A feed that fails to download or parse is logged as [ERROR] Failed to process <id> (<feed>) and keeps its old file. The run still succeeds, so check the logs for these.

Deploy GitHub Pages

File: .github/workflows/github_pages.yaml

Triggers:

Branches. Each part of the build comes from one branch, resolved once in the build job’s env:

Part Variable Push to master Push to dev-news Run from a dev-news* branch Run from another branch
Pipeline: the workflow and src/ PIPELINE_REF master master master that branch
Content: pages, blogs.json, favicons, _config.yml, _includes/ CONTENT_BRANCH master master master content_branch
Posts DEV_NEWS_BRANCH dev-news dev-news that branch dev_news_branch
Tools DEV_TOOLS_BRANCH dev-tools dev-tools dev-tools dev_tools_branch

Run from a dev-news* branch (picked in the Run workflow dropdown), the run is that branch’s caller workflow, which calls master’s copy with no inputs. github.ref_name is then that branch, so the pipeline and the content are master’s and the posts are that branch’s. No branch name ends up in the site’s links: the news page is generated with relative links, so the site works the same whichever branches are picked.

Build job:

  1. Checkout content. Checks out the content branch with full history.
  2. Take scripts from the pipeline. git checkout origin/<pipeline> -- src, so the build runs the scripts of the branch the workflow runs from, even when the content comes from another branch.
  3. Fetch dev-news branch.

    git fetch origin <dev-news branch>
    git checkout origin/<dev-news branch> -- posts   # posts/<id>.json
    git reset                                        # unstage
    node src/gen_news.js --site                      # news/README.md
    

    The dev-news README.md isn’t used. The site’s news page is generated from the posts, with links relative to news/ (../README.md, ../favicons/…), so it doesn’t depend on the branch the dev-news copy links to.

    Three sed calls fix links in README.md for the site:

    • The dev-news links (in the description and on the “news updated” badge) change from the dev-news branch on GitHub to news/, the news page.
    • The “blogs” badge changes from blogs/README.md to blogs/. jekyll-relative-links turns [text](page.md) links into page URLs, but it skips links around an image, which is what a badge is.
    • The “OPML” badge changes from the raw file on GitHub to blogs.opml, the site’s own copy.
  4. Render latest posts. node src/render_posts.js fills the Latest Posts lists in README.md and blogs/README.md.
  5. Fetch dev-tools branch. Adds the tools branch as a worktree in /tmp/dev-tools, deletes the placeholder tools/README.md, and copies the branch’s files into tools/.
  6. Build with Jekyll (actions/jekyll-build-pages) from ./ into ./_site.
  7. Upload Pages artifact.

Deploy job: actions/deploy-pages publishes the artifact.

All of these changes stay in the build workspace and are never committed. There is no preview environment: every run publishes to the live site.

The deploy badge in README.md shows the latest run on dev-news. Those runs happen every 6 hours and also build master, so a broken master shows up there too.


Site (Jekyll)

_config.yml

Setting Purpose
remote_theme: jurakovic/cayman-blue The theme, a fork of Cayman.
google_analytics Analytics ID.
dark_theme, target_blank Options of the cayman-blue theme: dark mode, and opening links in the header and main content in a new tab.
exclude: [src] Scripts are not published.
defaults Layout default for every page. Pages under news/ and blogs/ get the titles dev-news and dev-blogs and show_header: false.

_includes

File Purpose
head-extras.html The theme’s hook into <head>. Includes head-custom-css.html on every page, and head-custom-js-0-root.html on / and /blogs/.
head-custom-css.html Hides .pages-hidden; styles the Latest Posts lists (.blog-posts); colors of the dev-news category tags (kbd.tag-<class>).
head-custom-js-0-root.html Adds an “Expand all blog posts” / “Collapse all blog posts” button to <div id="posts-button">, which opens or closes every Latest Posts list.

On GitHub, the category tags on the news page lose their class and show as plain <kbd> elements.


Scripts

All scripts are in src/ (excluded from the site). The Node scripts use only the standard library: there is no package.json and nothing to install. They resolve paths from the repo root (path.join(__dirname, '..')), so they work from any directory.

Script Runs Purpose
update_posts.sh Update Posts workflow, in a dev-news checkout Fetch every feed into posts/*.json
fix_post_times.js called by update_posts.sh Set each post’s dates
gen_news.js Update Posts workflow, in a dev-news checkout; Deploy GitHub Pages workflow (--site) Generate the news page
render_posts.js Deploy GitHub Pages workflow Fill the Latest Posts lists
update_readme.js by hand, on master Regenerate README sections from blogs.json
add_blog.js by hand, on master Add a blog to blogs.json
gen_opml.js by hand, on master Generate blogs.opml
check_pubdates.js by hand, on master Find posts dated later than the commit that stored them
transform_blogs.js by hand, one-off Rewrite every entry in blogs.json

update_posts.sh

bash src/update_posts.sh [blog-id ...]

Fetches the feeds of all blogs in blogs.json that have one, or only the given IDs, and writes posts/<id>.json.

Requires bash, curl, jq, node and the Go yq v4 (mikefarah/yq). The Python yq (a jq wrapper) does not support -p=xml and fails. It also needs posts/ next to src/, which only exists in a dev-news checkout with the scripts copied in, as the workflow does. The file is not executable in git, so run it with bash or chmod +x it first.

How it works:

  1. Changes to posts/.
  2. Reads the newest date across all posts/*.json, once, before any feed is updated. This “newest listed” date goes to fix_post_times.js to detect lagged posts.
  3. For each feed:
    1. Downloads it with curl -Ss -k -L and a browser User-Agent, so sites don’t block it as a bot. -k ignores certificate errors.
    2. Converts the XML to JSON with yq. Takes .rss.channel.item (RSS 2.0) or .feed.entry (Atom), keeps the first 5, and maps each to {title, link, pubDate}, trying the shapes different feeds use:
      • title: plain text, text next to attributes (+content), or XHTML.
      • link: Atom <link rel="alternate" href>, then <link href>, then RSS <link> text.
      • pubDate: RSS pubDate, then Atom published, then updated.
    3. Pipes the posts to fix_post_times.js together with the stored file and the newest listed date.
    4. Writes the result to posts/<id>.json. The output is captured into a variable first: redirecting straight into the file would truncate it before node reads it.
    5. On any failure, prints [ERROR] and leaves the file as it was.

fix_post_times.js

echo "$posts_json" | node src/fix_post_times.js posts/<id>.json [newest-listed]

Takes freshly fetched posts on stdin, compares them with the stored file, and prints the posts with pubDate normalized and date set.

The problem. Feed dates can’t be trusted as they are:

The rules, per incoming post:

Case pubDate date
Known: link already stored stored value (re-normalized) stored value
New, and its date is in the future, midnight today, lagged, or unparseable feed date, normalized (raw if unparseable) now: the time first seen
New, any other date feed date, normalized same as pubDate

The header comment in src/fix_post_times.js has the full rules with an example for each case.

gen_news.js

node src/gen_news.js [content-branch]   # README.md, read on GitHub; default: master
node src/gen_news.js --site             # news/README.md, for the site

Generates the news page, either as README.md in the repo root (in Update Posts, the dev-news checkout) or, with --site, as news/README.md (in the Pages build):

  1. Reads every posts/*.json, whether or not the blog is still in blogs.json.
  2. Sorts by date, newest first, and keeps 100.
  3. Writes a breadcrumb back to dev-links and dev-blogs, with the top anchor.
  4. Writes each post as two lines: favicon, date (YYYY-MM-DD), blog ID and category tag, then the title linked to the post. In titles, the Markdown characters *, _, `, [ and ] are escaped, and < and > become entities.
  5. Ends with an “↑ Top” link.

Links to master content (the breadcrumb, category tags and favicons) go through two bases, PAGE_BASE and FAVICON_BASE:

A link from one dev-news page to another is relative in both forms, so it stays on the same branch on GitHub and within the site on Pages.

Category tags come from CATEGORY_TAGS: a label, a CSS class (tag-<class>, colored in _includes/head-custom-css.html) and a link to the category’s section. A post whose blog has no entry there gets no tag; one whose blog isn’t in blogs.json also gets the blank icon.

render_posts.js

node src/render_posts.js

Runs in the Pages build, after the dev-news files are copied in. For README.md and blogs/README.md, replaces each <div id="<id>" class="blog-posts"></div> with that blog’s posts:

<div id="<id>" class="blog-posts"><blockquote><details><summary>Latest Posts</summary><div><ul><li><span>›</span><a href="..." target="_blank" rel="noopener noreferrer">2026-11-01 Title</a></li>...</ul></div></details></blockquote></div>

update_readme.js

node src/update_readme.js [--readme | --blogs | --all] [category]
Option Updates
--readme (default) README.md, selected blogs only
--blogs blogs/README.md, all blogs
--all both
category only that section; exits with an error if no blog has that category

Each blog is rendered as below, every line ending in two spaces (a Markdown line break), with a blank line between blogs:

![icon](favicons/<icon>.png)                ../favicons/ in blogs/README.md
<author> | <title>                          either one alone if the other is empty
<url> ([GitHub](<github>))                  GitHub link inline only for a repository URL
<extras[0]>                                 one line per entry in extras
<github>                                    a profile URL
<div id="<id>" class="blog-posts"></div>    only with a feed

Run it after every blogs.json change and check git diff. With an unchanged blogs.json it produces no diff.

add_blog.js

node src/add_blog.js <url> <feed> [github] [author] [category]
node src/add_blog.js https://third-bit.com/blog/ https://third-bit.com/atom.xml https://github.com/gvwilson "Greg Wilson" general
  1. Makes the ID from the host name without www. and the last part, with dots turned into dashes: martin.kleppmann.com → martin-kleppmann.
  2. Takes the title from the page’s <title>, with a 10-second timeout.
  3. Downloads a 16 px favicon from Google’s favicon service into favicons/<id>.png. On a 404 the icon is blank.
  4. Appends the entry to blogs.json with description: "", extras: [] and selected: false. The category defaults to general.
  5. Stops if the ID already exists. The favicon was already downloaded by then and has overwritten favicons/<id>.png.

gen_opml.js

node src/gen_opml.js

Writes blogs.opml with every blog, including those without a feed (empty xmlUrl). text is the author, or the title when there is no author; title is the reverse.

check_pubdates.js

git fetch origin dev-news
node src/check_pubdates.js [ref]   # default: origin/dev-news

Runs git blame on every posts/*.json at ref and reports each "date" later than the commit that wrote that line: a date that was in the future when it was stored. A check on fix_post_times.js. Prints No issues found. when there are none.

transform_blogs.js

node src/transform_blogs.js

Rewrites blogs.json with every entry’s fields in the standard order and selected: true on every blog. It was written to add the selected field (#28). Run as it is, it overwrites every selected flag; for another one-off change to every entry, edit the mapping first.


Common tasks

Add a blog

  1. node src/add_blog.js <url> <feed> [github] [author] [category]
  2. In blogs.json, fix the new entry: title, author, description, category, selected. Move it to where it should appear in its section.
  3. Check favicons/<id>.png. Use blank as the icon if there is none.
  4. node src/update_readme.js --all
  5. node src/gen_opml.js
  6. Commit and merge to master. The site deploys right away; the blog’s Latest Posts list appears after the next update run. To get it sooner, run Update Posts by hand and the deploy follows.

The first fetch of a new feed keeps the feed’s own dates, so its posts sort by their real dates on dev-news instead of all appearing at the top.

Change or remove a blog

Add a category

  1. Add a heading, a nav entry and <!-- BEGIN <CATEGORY> --> / <!-- END <CATEGORY> --> markers to README.md and/or blogs/README.md.
  2. Add the category to CATEGORY_TAGS in src/gen_news.js: label, CSS class and section URL.
  3. Add a .main-content kbd.tag-<class> color rule to _includes/head-custom-css.html.
  4. Set the category on the blogs and run node src/update_readme.js --all.

Update the news now

Actions → Update Posts (dev-news) → Run workflow, from master. The deploy runs after it.

Redeploy the site

Actions → Deploy GitHub Pages → Run workflow, from master, with the default inputs. Use this after a dev-tools change to skip waiting for the next news update.

Test changes to the update pipeline

  1. Push the change to a feature branch.
  2. Create a temporary test branch from dev-news, so the test run doesn’t write to dev-news itself. Start its name with dev-news (dev-news-test): Deploy GitHub Pages treats a branch named like that as a posts branch (see the Branches table under Deploy GitHub Pages), so picking it in the Run workflow dropdown works in step 4.

    git fetch origin dev-news
    git push origin origin/dev-news:refs/heads/<test-branch>
    
  3. Run Update Posts (dev-news) from the feature branch with target_branch set to <test-branch>. The scripts come from the feature branch, and blogs.json from content_branch: master, unless you set it to the feature branch. The result is committed to the test branch, where you can check the diff.
    • The news page’s links on GitHub point to the content branch.
    • The push to the test branch doesn’t deploy: the caller workflow it copied from dev-news only fires on pushes to dev-news.
  4. To see the test data on the site, run Deploy GitHub Pages with dev_news_branch: <test-branch>. Run it from master to see the data as master renders it, or from the feature branch to see the feature’s rendering, adding content_branch: <feature-branch> if the feature changes content. If the test branch’s name starts with dev-news, picking it in the Run workflow dropdown also works: its caller deploys its posts with master’s pipeline and content. With any other name, that run fails, since the branch is then taken as the pipeline and has no src/. This publishes to the live site; the next normal deploy puts it back.
  5. Delete the test branch when you’re done.

Run locally

Content scripts run on master from the repo root, as shown above.

The news pipeline, without pushing anything:

git fetch origin dev-news
git worktree add ../dev-links-news origin/dev-news
cd ../dev-links-news
git restore --source origin/master -- blogs.json src/gen_news.js src/update_posts.sh src/fix_post_times.js
bash src/update_posts.sh simonwillison   # one feed; no arguments for all feeds
node src/gen_news.js master
git diff --stat
cd - && git worktree remove --force ../dev-links-news

To test uncommitted script changes, copy the files from your working tree instead of using git restore.

The Pages build up to Jekyll, in a throwaway worktree because it overwrites files:

git fetch origin dev-news
git worktree add ../dev-links-site HEAD
cd ../dev-links-site
git checkout origin/dev-news -- posts && git reset
node src/gen_news.js --site
node src/render_posts.js
cd - && git worktree remove --force ../dev-links-site

The Jekyll step itself needs the github-pages gem and isn’t set up in this repo.


Known quirks


History

Date Change
Jun 2024 First commit.
Sep 2025 Latest Posts lists from RSS feeds (#19); news reader page (#20); news page generated as static Markdown (#21); posts moved to the dev-news branch, Deploy GitHub Pages workflow added (#22).
Oct 2025 Scripts moved from dev-news to master (#25).
Feb 2026 blogs.json, blogs.opml and scripted README updates (#27); dev-blogs page and selected (#28); fix for date-only feeds, fix_post_times.js (#31).
Mar 2026 Fix for future dates (#33).
May 2026 dev-tools added to the Pages deploy (#34, #35); update schedule moved to cron-job.org.
Oct 2026 Category tags on dev-news (#37); Latest Posts lists rendered at build time instead of fetched in the browser; fix for midnight dates with a non-UTC offset.
Nov 2026 Lagged posts moved to the top with their own date; pubDate stored normalized; scripts moved to src/ (#39); the Pages build generates the news page itself, and both workflows pick the pipeline, content and data branches separately.