headscale/CHANGELOG.md

CHANGELOG

Next

Security fix: OIDC changes in Headscale 0.24.0

Headscale v0.23.0 and earlier identified OIDC users by the "username" part of their email address (when strip_email_domain: true, the default) or whole email address (when strip_email_domain: false).

Depending on how Headscale and your Identity Provider (IdP) were configured, only using the email claim could allow a malicious user with an IdP account to take over another Headscale user's account, even when strip_email_domain: false.

This would also cause a user to lose access to their Headscale account if they changed their email address.

Headscale v0.24.0 now identifies OIDC users by the iss and sub claims. These are guaranteed by the OIDC specification to be stable and unique, even if a user changes email address. A well-designed IdP will typically set sub to an opaque identifier like a UUID or numeric ID, which has no relation to the user's name or email address.

This issue only affects Headscale installations which authenticate with OIDC.

Headscale v0.24.0 and later will also automatically update profile fields with OIDC data on login. This means that users can change those details in your IdP, and have it populate to Headscale automatically the next time they log in. However, this may affect the way you reference users in policies.

Migrating existing installations

Headscale v0.23.0 and earlier never recorded the iss and sub fields, so all legacy (existing) OIDC accounts from need to be migrated to be properly secured.

Headscale v0.24.0 has an automatic migration feature, which is enabled by default (map_legacy_users: true). This will be disabled by default in a future version of Headscale – any unmigrated users will get new accounts.

Headscale v0.24.0 will ignore any email claim if the IdP does not provide an email_verified claim set to true. What "verified" actually means is contextually dependent – Headscale uses it as a signal that the contents of the email claim is reasonably trustworthy.

Headscale v0.23.0 and earlier never checked the email_verified claim. This means even if an IdP explicitly indicated to Headscale that its email claim was untrustworthy, Headscale would have still accepted it.

What does automatic migration do?

When automatic migration is enabled (map_legacy_users: true), Headscale will first match an OIDC account to a Headscale account by iss and sub, and then fall back to matching OIDC users similarly to how Headscale v0.23.0 did:

• If strip_email_domain: true (the default): the Headscale username matches the "username" part of their email address.
• If strip_email_domain: false: the Headscale username matches the whole email address.

On migration, Headscale will change the account's username to their preferred_username. This could break any ACLs or policies which are configured to match by username.

Like with Headscale v0.23.0 and earlier, this migration only works for users who haven't changed their email address since their last Headscale login.

A successful automated migration should otherwise be transparent to users.

Once a Headscale account has been migrated, it will be unavailable to be matched by the legacy process. An OIDC login with a matching username, but non-matching iss and sub will instead get a new Headscale account.

Because of the way OIDC works, Headscale's automated migration process can only work when a user tries to log in after the update. Mass updates would require Headscale implement a protocol like SCIM, which is extremely complicated and not available in all identity providers.

Administrators could also attempt to migrate users manually by editing the database, using their own mapping rules with known-good data sources.

Legacy account migration should have no effect on new installations where all users have a recorded sub and iss.

What happens when automatic migration is disabled?

When automatic migration is disabled (map_legacy_users: false), Headscale will only try to match an OIDC account to a Headscale account by iss and sub.

If there is no match, it will get a new Headscale account – even if there was a legacy account which could have matched and migrated.

We recommend new Headscale users explicitly disable automatic migration – but it should otherwise have no effect if every account has a recorded iss and sub.

When automatic migration is disabled, the strip_email_domain setting will have no effect.

Special thanks to @micolous for reviewing, proposing and working with us on these changes.

Other OIDC changes

Headscale now uses the standard OIDC claims to populate and update user information every time they log in:

Headscale profile field	OIDC claim	Notes / examples
email address	email	Only used when "email_verified": true
display name	name	eg: Sam Smith
username	preferred_username	Varies depending on IdP and configuration, eg: ssmith, ssmith@idp.example.com, \\example.com\ssmith
profile picture	picture	URL to a profile picture or avatar

These should show up nicely in the Tailscale client.

This will also affect the way you reference users in policies.

BREAKING

• Remove dns.use_username_in_magic_dns configuration option #2020
  • Having usernames in magic DNS is no longer possible.
• Remove versions older than 1.56 #2149
  • Clean up old code required by old versions

Changes

• Improved compatibilty of built-in DERP server with clients connecting over WebSocket.
• Allow nodes to use SSH agent forwarding #2145
• Fixed processing of fields in post request in MoveNode rpc #2179
• Added conversion of 'Hostname' to 'givenName' in a node with FQDN rules applied #2198
• Fixed updating of hostname and givenName when it is updated in HostInfo #2199
• Fixed missing stable-debug container tag #2232

0.23.0 (2024-09-18)

This release was intended to be mainly a code reorganisation and refactoring, significantly improving the maintainability of the codebase. This should allow us to improve further and make it easier for the maintainers to keep on top of the project. However, as you all have noticed, it turned out to become a much larger, much longer release cycle than anticipated. It has ended up to be a release with a lot of rewrites and changes to the code base and functionality of Headscale, cleaning up a lot of technical debt and introducing a lot of improvements. This does come with some breaking changes,

Please remember to always back up your database between versions

Here is a short summary of the broad topics of changes:

Code has been organised into modules, reducing use of global variables/objects, isolating concerns and “putting the right things in the logical place”.

The new policy and mapper package, containing the ACL/Policy logic and the logic for creating the data served to clients (the network “map”) has been rewritten and improved. This change has allowed us to finish SSH support and add additional tests throughout the code to ensure correctness.

The “poller”, or streaming logic has been rewritten and instead of keeping track of the latest updates, checking at a fixed interval, it now uses go channels, implemented in our new notifier package and it allows us to send updates to connected clients immediately. This should both improve performance and potential latency before a client picks up an update.

Headscale now supports sending “delta” updates, thanks to the new mapper and poller logic, allowing us to only inform nodes about new nodes, changed nodes and removed nodes. Previously we sent the entire state of the network every time an update was due.

While we have a pretty good test harness for validating our changes, the changes came down to 284 changed files with 32,316 additions and 24,245 deletions and bugs are expected. We need help testing this release. In addition, while we think the performance should in general be better, there might be regressions in parts of the platform, particularly where we prioritised correctness over speed.

There are also several bugfixes that has been encountered and fixed as part of implementing these changes, particularly after improving the test harness as part of adopting #1460.

BREAKING

• Code reorganisation, a lot of code has moved, please review the following PRs accordingly #1473
• Change the structure of database configuration, see config-example.yaml for the new structure. #1700
  • Old structure has been remove and the configuration must be converted.
  • Adds additional configuration for PostgreSQL for setting max open, idle connection and idle connection lifetime.
• API: Machine is now Node #1553
• Remove support for older Tailscale clients #1611
  • The oldest supported client is 1.42
• Headscale checks that at least one DERP is defined at start #1564
  • If no DERP is configured, the server will fail to start, this can be because it cannot load the DERPMap from file or url.
• Embedded DERP server requires a private key #1611
  • Add a filepath entry to derp.server.private_key_path
• Docker images are now built with goreleaser (ko) #1716 #1763
  • Entrypoint of container image has changed from shell to headscale, require change from headscale serve to serve
  • /var/lib/headscale and /var/run/headscale is no longer created automatically, see container docs
• Prefixes are now defined per v4 and v6 range. #1756
  • ip_prefixes option is now prefixes.v4 and prefixes.v6
  • prefixes.allocation can be set to assign IPs at sequential or random. #1869
• MagicDNS domains no longer contain usernames
  • This is in preperation to fix Headscales implementation of tags which currently does not correctly remove the link between a tagged device and a user. As tagged devices will not have a user, this will require a change to the DNS generation, removing the username, see #1369 for more information.
  • use_username_in_magic_dns can be used to turn this behaviour on again, but note that this option will be removed when tags are fixed.
    • dns.base_domain can no longer be the same as (or part of) server_url.
    • This option brings Headscales behaviour in line with Tailscale.
• YAML files are no longer supported for headscale policy. #1792
  • HuJSON is now the only supported format for policy.
• DNS configuration has been restructured #2034
  • Please review the new config-example.yaml for the new structure.

Changes

• Use versioned migrations #1644
• Make the OIDC callback page better #1484
• SSH support #1487
• State management has been improved #1492
• Use error group handling to ensure tests actually pass #1535 based on #1460
• Fix hang on SIGTERM #1492 taken from #1480
• Send logs to stderr by default #1524
• Fix TS-2023-006 security UPnP issue #1563
• Turn off gRPC logging #1640 fixes #1259
• Added the possibility to manually create a DERP-map entry which can be customized, instead of automatically creating it. #1565
• Add support for deleting api keys #1702
• Add command to backfill IP addresses for nodes missing IPs from configured prefixes. #1869
• Log available update as warning #1877
• Add autogroup:internet to Policy #1917
• Restore foreign keys and add constraints #1562
• Make registration page easier to use on mobile devices
• Make write-ahead-log default on and configurable for SQLite #1985
• Add APIs for managing headscale policy. #1792
• Fix for registering nodes using preauthkeys when running on a postgres database in a non-UTC timezone. #764
• Make sure integration tests cover postgres for all scenarios
• CLI commands (all except serve) only requires minimal configuration, no more errors or warnings from unset settings #2109
• CLI results are now concistently sent to stdout and errors to stderr #2109
• Fix issue where shutting down headscale would hang #2113

0.22.3 (2023-05-12)

Changes

• Added missing ca-certificates in Docker image #1463

0.22.2 (2023-05-10)

Changes

• Add environment flags to enable pprof (profiling) #1382
  • Profiles are continuously generated in our integration tests.
• Fix systemd service file location in .deb packages #1391
• Improvements on Noise implementation #1379
• Replace node filter logic, ensuring nodes with access can see eachother #1381
• Disable (or delete) both exit routes at the same time #1428
• Ditch distroless for Docker image, create default socket dir in /var/run/headscale #1450

0.22.1 (2023-04-20)

Changes

• Fix issue where systemd could not bind to port 80 #1365

0.22.0 (2023-04-20)

Changes

• Add .deb packages to release process #1297
• Update and simplify the documentation to use new .deb packages #1349
• Add 32-bit Arm platforms to release process #1297
• Fix longstanding bug that would prevent "*" from working properly in ACLs (issue #699) #1279
• Fix issue where IPv6 could not be used in, or while using ACLs (part of #809) #1339
• Target Go 1.20 and Tailscale 1.38 for Headscale #1323

0.21.0 (2023-03-20)

Changes

• Adding "configtest" CLI command. #1230
• Add documentation on connecting with iOS to /apple #1261
• Update iOS compatibility and added documentation for iOS #1264
• Allow to delete routes #1244

0.20.0 (2023-02-03)

Changes

• Fix wrong behaviour in exit nodes #1159
• Align behaviour of dns_config.restricted_nameservers to tailscale #1162
• Make OpenID Connect authenticated client expiry time configurable #1191
  • defaults to 180 days like Tailscale SaaS
  • adds option to use the expiry time from the OpenID token for the node (see config-example.yaml)
• Set ControlTime in Map info sent to nodes #1195
• Populate Tags field on Node updates sent #1195

0.19.0 (2023-01-29)

BREAKING

• Rename Namespace to User #1144
  • BACKUP your database before upgrading
• Command line flags previously taking --namespace or -n will now require --user or -u

0.18.0 (2023-01-14)

Changes

• Reworked routing and added support for subnet router failover #1024
• Added an OIDC AllowGroups Configuration options and authorization check #1041
• Set db_ssl to false by default #1052
• Fix duplicate nodes due to incorrect implementation of the protocol #1058
• Report if a machine is online in CLI more accurately #1062
• Added config option for custom DNS records #1035
• Expire nodes based on OIDC token expiry #1067
• Remove ephemeral nodes on logout #1098
• Performance improvements in ACLs #1129
• OIDC client secret can be passed via a file #1127

0.17.1 (2022-12-05)

Changes

• Correct typo on macOS standalone profile link #1028
• Update platform docs with Fast User Switching #1016

0.17.0 (2022-11-26)

BREAKING

• noise.private_key_path has been added and is required for the new noise protocol.
• Log level option log_level was moved to a distinct log config section and renamed to level #768
• Removed Alpine Linux container image #962

Important Changes

• Added support for Tailscale TS2021 protocol #738
• Add experimental support for SSH ACL (see docs for limitations) #847
  • Please note that this support should be considered partially implemented
  • SSH ACLs status:
    • Support accept and check (SSH can be enabled and used for connecting and authentication)
    • Rejecting connections are not supported, meaning that if you enable SSH, then assume that all ssh connections will be allowed.
    • If you decided to try this feature, please carefully managed permissions by blocking port 22 with regular ACLs or do not set --ssh on your clients.
    • We are currently improving our testing of the SSH ACLs, help us get an overview by testing and giving feedback.
  • This feature should be considered dangerous and it is disabled by default. Enable by setting HEADSCALE_EXPERIMENTAL_FEATURE_SSH=1.

Changes

• Add ability to specify config location via env var HEADSCALE_CONFIG #674
• Target Go 1.19 for Headscale #778
• Target Tailscale v1.30.0 to build Headscale #780
• Give a warning when running Headscale with reverse proxy improperly configured for WebSockets #788
• Fix subnet routers with Primary Routes #811
• Added support for JSON logs #653
• Sanitise the node key passed to registration url #823
• Add support for generating pre-auth keys with tags #767
• Add support for evaluating autoApprovers ACL entries when a machine is registered #763
• Add config flag to allow Headscale to start if OIDC provider is down #829
• Fix prefix length comparison bug in AutoApprovers route evaluation #862
• Random node DNS suffix only applied if names collide in namespace. #766
• Remove ip_prefix configuration option and warning #899
• Add dns_config.override_local_dns option #905
• Fix some DNS config issues #660
• Make it possible to disable TS2019 with build flag #928
• Fix OIDC registration issues #960 and #971
• Add support for specifying NextDNS DNS-over-HTTPS resolver #940
• Make more sslmode available for postgresql connection #927

0.16.4 (2022-08-21)

Changes

• Add ability to connect to PostgreSQL over TLS/SSL #745
• Fix CLI registration of expired machines #754

0.16.3 (2022-08-17)

Changes

• Fix issue with OIDC authentication #747

0.16.2 (2022-08-14)

Changes

• Fixed bugs in the client registration process after migration to NodeKey #735

0.16.1 (2022-08-12)

Changes

• Updated dependencies (including the library that lacked armhf support) #722
• Fix missing group expansion in function excludeCorrectlyTaggedNodes #563
• Improve registration protocol implementation and switch to NodeKey as main identifier #725
• Add ability to connect to PostgreSQL via unix socket #734

0.16.0 (2022-07-25)

Note: Take a backup of your database before upgrading.

BREAKING

• Old ACL syntax is no longer supported ("users" & "ports" -> "src" & "dst"). Please check the new syntax.

Changes

• Drop armhf (32-bit ARM) support. #609
• Headscale fails to serve if the ACL policy file cannot be parsed #537
• Fix labels cardinality error when registering unknown pre-auth key #519
• Fix send on closed channel crash in polling #542
• Fixed spurious calls to setLastStateChangeToNow from ephemeral nodes #566
• Add command for moving nodes between namespaces #362
• Added more configuration parameters for OpenID Connect (scopes, free-form parameters, domain and user allowlist)
• Add command to set tags on a node #525
• Add command to view tags of nodes #356
• Add --all (-a) flag to enable routes command #360
• Fix issue where nodes was not updated across namespaces #560
• Add the ability to rename a nodes name #560
  • Node DNS names are now unique, a random suffix will be added when a node joins
  • This change contains database changes, remember to backup your database before upgrading
• Add option to enable/disable logtail (Tailscale's logging infrastructure) #596
  • This change disables the logs by default
• Use [Prometheus]'s duration parser, supporting days (d), weeks (w) and years (y) #598
• Add support for reloading ACLs with SIGHUP #601
• Use new ACL syntax #618
• Add -c option to specify config file from command line #285 #612
• Add configuration option to allow Tailscale clients to use a random WireGuard port. kb/1181/firewalls #624
• Improve obtuse UX regarding missing configuration (ephemeral_node_inactivity_timeout not set) #639
• Fix nodes being shown as 'offline' in tailscale status #648
• Improve shutdown behaviour #651
• Drop Gin as web framework in Headscale 648 677
• Make tailnet node updates check interval configurable #675
• Fix regression with HTTP API #684
• nodes ls now print both Hostname and Name(Issue #647 PR #687)

0.15.0 (2022-03-20)

Note: Take a backup of your database before upgrading.

BREAKING

• Boundaries between Namespaces has been removed and all nodes can communicate by default #357
  • To limit access between nodes, use ACLs.
• /metrics is now a configurable host:port endpoint: #344. You must update your config.yaml file to include:

  metrics_listen_addr: 127.0.0.1:9090
  

Features

• Add support for writing ACL files with YAML #359
• Users can now use emails in ACL's groups #372
• Add shorthand aliases for commands and subcommands #376
• Add /windows endpoint for Windows configuration instructions + registry file download #392
• Added embedded DERP (and STUN) server into Headscale #388

Changes

• Fix a bug were the same IP could be assigned to multiple hosts if joined in quick succession #346
• Simplify the code behind registration of machines #366
  • Nodes are now only written to database if they are registered successfully
• Fix a limitation in the ACLs that prevented users to write rules with * as source #374
• Reduce the overhead of marshal/unmarshal for Hostinfo, routes and endpoints by using specific types in Machine #371
• Apply normalization function to FQDN on hostnames when hosts registers and retrieve information #363
• Fix a bug that prevented the use of tailscale logout with OIDC #508
• Added Tailscale repo HEAD and unstable releases channel to the integration tests targets #513

0.14.0 (2022-02-24)

**UPCOMING ### BREAKING From the **next** version (0.15.0), all machines will be able to communicate regardless of if they are in the same namespace. This means that the behaviour currently limited to ACLs will become default. From version 0.15.0, all limitation of communications must be done with ACLs.

This is a part of aligning headscale's behaviour with Tailscale's upstream behaviour.

BREAKING

• ACLs have been rewritten to align with the bevaviour Tailscale Control Panel provides. NOTE: This is only active if you use ACLs
  • Namespaces are now treated as Users
  • All machines can communicate with all machines by default
  • Tags should now work correctly and adding a host to Headscale should now reload the rules.
  • The documentation have a fictional example that should cover some use cases of the ACLs features

Features

• Add support for configurable mTLS docs #297

Changes

• Remove dependency on CGO (switch from CGO SQLite to pure Go) #346

0.13.0 (2022-02-18):

Features

• Add IPv6 support to the prefix assigned to namespaces
• Add API Key support
  • Enable remote control of headscale via CLI docs
  • Enable HTTP API (beta, subject to change)
• OpenID Connect users will be mapped per namespaces
  • Each user will get its own namespace, created if it does not exist
  • oidc.domain_map option has been removed
  • strip_email_domain option has been added (see config-example.yaml)

Changes

• ip_prefix is now superseded by ip_prefixes in the configuration #208
• Upgrade tailscale (1.20.4) and other dependencies to latest #314
• fix swapped machine<->namespace labels in /metrics #312
• remove key-value based update mechanism for namespace changes #316

0.12.4 (2022-01-29):

Changes

• Make gRPC Unix Socket permissions configurable #292
• Trim whitespace before reading Private Key from file #289
• Add new command to generate a private key for headscale #290
• Fixed issue where hosts deleted from control server may be written back to the database, as long as they are connected to the control server #278

0.12.3 (2022-01-13)

Changes

• Added Alpine container #270
• Minor updates in dependencies #271

0.12.2 (2022-01-11)

Happy New Year!

Changes

• Fix Docker release #258
• Rewrite main docs #262
• Improve Docker docs #263

0.12.1 (2021-12-24)

(We are skipping 0.12.0 to correct a mishap done weeks ago with the version tagging)

BREAKING

• Upgrade to Tailscale 1.18 #229
  • This change requires a new format for private key, private keys are now generated automatically:
    1. Delete your current key
    2. Restart headscale, a new key will be generated.
    3. Restart all Tailscale clients to fetch the new key

Changes

• Unify configuration example #197
• Add stricter linting and formatting #223

Features

• Add gRPC and HTTP API (HTTP API is currently disabled) #204
• Use gRPC between the CLI and the server #206, #212
• Beta OpenID Connect support #126, #227

0.11.0 (2021-10-25)

BREAKING

• Make headscale fetch DERP map from URL and file #196