Overview Link to heading

How to access remote Git repositories securely via command line? And where to store the secrets?

These questions led me down a few unexpected rabbit holes and I discovered many things about Git authentication and many reasons for and against each. I was bound to forget all of this if I didn’t write it down. So this is as much a note to myself as it may be useful to others. I am grateful to the many bloggers that have written posts on Git authentication, particularly this post by Danielle Navarro.

To make it short, username/password authentication cannot be used for Git API access on GitHub since 2021 and the major alternatives are personal access tokens (PAT), secure shell (SSH) and web authentication (OAuth). I settled on PATs because they can be given an expiration date, offer fine-grained access control and reflect security-by-isolation principles that I find persuasive.

PATs need to be stored securely. Among the multiverse of Git credential helpers I prefer using the system keyring on my local machine. On Debian Linux this requires some non-obvious setup of libsecret and compilation of the associated credential helper. On Windows and MacOS it is supposed to work right out of the box, but I haven’t worked on Windows for over a decade and have never had a Mac, so I wouldn’t know. The Linux system keyring works well in practice and integrates nicely with my Emacs workflow.

The Git cache is a good alternative for remote deployment servers. The timeout can be customized, something like 30 to 60 minutes seems reasonable. It is important to cache only tightly scoped PATs on remote machines. If the scope is set to single repository and read-only, the damage of token loss should not exceed that of machine compromise.

Above all else, remember AviD’s Rule of Usability:

Security at the expense of usability comes at the expense of security.

In other words: if security is too much of a pain, people will prefer insecurity (including myself).

Table of Contents Link to heading

The Problem Link to heading

Accessing remote Git repositories hosted on services like GitHub to pull, push or do other things programmatically via API requires authentication. Git-the-software (not GitHub-the-service) originally envisioned username/password authentication as the default, but with the advent of more stringent security practices username/password authentication for API access fell into disrepute and was disallowed on GitHub in 2021. Additionally, two-factor authentication has become the norm (and is enforced for open source contributors such as myself) so the old Git authentication flow which can’t handle a second factor wouldn’t work anyway.

Currently there are three dominant authentication options for Git repository access via command line:

• Personal access tokens (PAT)
• Secure shell (SSH)
• Open Authorization (OAuth)

Personal access tokens (PAT) are very long strings of characters that act as a combined username/password replacement, are only used for API access, can be given an expiry date and support very fine-grained access control. PATs are supposed to be transient and replaced often, so that the loss of a token does not compromise the integrity of the entire account. They could be described as temporary limited passwords, albeit with similar secure storage requirements. PATs are an example of the security-by-isolation principle.

Secure Shell (SSH) authentication is a popular alternative to PAT-based authentication. Instead of using a symmetric transient password-like token, SSH authentication is based on asymmetric public/private key pairs. The private key always remains on the machine of the user, the public key is uploaded to the server. The private key proves the identity of the user, the public key can only be used to verify it, nothing else. SSH is primarily a means of controlling a remote server via command line, but it is possible to use it to manage a remote Git repository instead.

Open Authorization (OAuth) is the third major access route to GitHub repositories. At the beginning of a standard OAuth flow a browser window is opened and the user must login to GitHub with their username/password and second factor. Following this a time-limited OAuth token is created and used for subsequent non-interactive authentication.

PATs are very long, are supposed to change regularly and are impractical to memorize, so they must be stored securely — somewhere. The same is true for SSH keys, the private key must also be kept safe. Even with OAuth the OAuth token must be stored safely during its validity period.

This is the crux of the problem: where to store the secrets?

The Multiverse of Git Credential Helpers Link to heading

Over the years the Git ecosystem has added a bewildering array of credential helpers to make it easier to supply Git with the appropriate authentication credentials. Credential helpers don’t store the secrets themselves, but are interfaces to storage options that do.

Below is a list of almost all of the credentials helpers mentioned in the Git documentation, minus the Azure/Netlify specific ones, which are irrelevant to my use case:¹

• Native to Git
  • Git Credential Store
  • Git Cache
• Platform Specific Keyrings
  • Linux GNOME Keyring or KDE Wallet
  • MacOS Keychain
  • Windows Credential Manager
• OAuth
  • Git Credential Manager (GCM)
  • Git Credential OAuth
• Password Managers
  • Gopass
  • Lastpass
  • 1Password
  • KeepassXC

I’ve decided to use Git Cache and the Linux keyring in my personal workflow. This essay includes setup instructions for cache and keyring. My daily workflow looks something like this:

I use the KeepassXC password manager to store all my PATs, so adding a KeepassXC credential helper interface might be useful. However, my specific system setup makes this impractical. I do mention it because others might find this option useful.

Not sure where to fit in the Git Credential Manager (GCM) here. The git-credential-oauth software seems a much better and focused option (400 lines of code vs 40,000 for GCM) for OAuth and GCM secrets storage for PATs appears to be handled by the system keyrings anyway, so might as well use those and save on bloat. I pretty much ignored GCM, although others seem to like it.

I decided against using the native Git Credential Store (cleartext storage! on the disk!) or SSH (many reasons, complicated). Your use case and security considerations may differ, but I’ve written down my reasons below. I’m interested in OAuth, but haven’t gotten around to trying it.

Realistic Option 1: Git Cache Link to heading

The native Git Cache is one of my preferred credential helpers. It can be set up with just a single command:

1$ git config --global credential.helper cache

When an action requires authentication and the PAT is entered, Git stores the PAT in memory, never on disk. By default the timeout is 15 minutes.

The timeout can be configured separately, by adding the number of seconds until timeout in the same command. The following sets a 1-hour timeout:

1$ git config --global credential.helper 'cache --timeout=3600'

Relying on the Git Cache is far more secure than storing the PAT in cleartext on disk, but one should assume that keyloggers would still be able to capture the PAT, maybe even extract it from memory, if the machine is compromised.

This is why it remains important to set an appropriate cache timeout, set the PAT scope narrowly and include a reasonable PAT expiry date to limit damage. Temporal and thematic limitations of the PAT are more important than the cache timeout.

Personally I use the Git Cache for pulling code to remote machines if the project requires more resources than I have available locally. I set the Git Cache timeout to between 30 to 60 minutes.

Nevertheless, the most important security measure is to set tight access controls on the PAT. If the PAT is used on a remote server, I set it to single repository scope and read-only permissions. For the code in the repository to be executed it must be located on the remote machine in cleartext anyway, so losing the PAT should not cause more damage than the machine compromise itself.

Realistic Option 2: System Keyring Link to heading

The keyring is a popular system feature on many platforms that stores usernames and passwords in secure encrypted storage and can only be accessed with the proper credentials, often a primary password, but more recently also biometric features or hardware tokens.

The advantage of the system keyring is much greater convenience compared to the Git Cache in daily usage, since it stores a PAT permanently in encrypted storage. Once the keyring is unlocked, the PAT stays in memory until logout (on Linux).²

The system keyring is also more pleasant to use with Emacs, my preferred text editor. Pushing from Emacs with the git-credential-libsecret helper will open a system prompt to log-in with the keyring, whereas pushing with Git Cache simply fails and forces me to break workflow and enter the PAT on the command line at least once.

The disadvantage of the system keyring on Linux is that once unlocked it stays unlocked until the session is ended. It appears possible to set the keyring timeout somewhere, but so far I haven’t found managed to figure out how. In the meantime this useful command restarts the keyring and flushes the cache:

1$ gnome-keyring-daemon -rd

Configuring the System Keyring Link to heading

Configuring Git to use the system keyring is supposed to be very easy on Windows and Mac. I think it even works out of the box, since credential helpers for the Windows Credential Manager and MacOS keychain are included with Git. I haven’t worked on anything but Linux for a long time, so I wouldn’t know. I do hope it works that way.

Libsecret is an interface between the Linux keyring (the storage for secrets) and other applications that want to access the keyring. There are three steps to installing and configuring Libsecret as the Git credential manager on Linux. It turns out that it isn’t actually difficult, just very much non-obvious.

The three steps are:

1. Install packages
2. Compile credential manager
3. Configure git to use libsecret

Step 1: Install dependencies Link to heading

These are the packages required to make it work, including make and gcc to build from source, git in case it’s not already installed and libsecret.

1$ sudo apt install make gcc git libsecret-1-0 libsecret-1-dev libglib2.0-dev

Step 2: Compile Git-Libsecret-Connector Link to heading

This is the non-obvious part. Git technically already comes with a credential helper that connects to libsecret, but on Debian and Ubuntu it is shipped as pure source code, not as a ready-to-use binary.

It therefore needs to be compiled first. Fortunately, this is very easy if you know the magic words.³

1$ sudo make --directory=/usr/share/doc/git/contrib/credential/libsecret

This compiles the C source code into a usable binary.

Step 3: Configure git to use Libsecret Link to heading

The following line sets the Git configuration file to use libsecret to access the system keyring. It works just like the Cache configuration line above.

1$ git config --global credential.helper /usr/share/doc/git/contrib/credential/libsecret/git-credential-libsecret

Now it should automatically trigger the keyring whenever you push, pull or do something that requires authentication!

Why not Git Credential Store? Link to heading

The integrated Git Credential Store is the easiest credential helper to set up and the most convenient to use, but also the most insecure. This is because it saves the PAT in cleartext on disk. Yes, in cleartext with no security whatsoever. I mention the command only because it is the one I choose to avoid:

1$ git config credential.helper store

Anyone who can access the disk can access the stored PAT. The most likely breach scenario would be some kind of drive-by infection with a trojan distributed by an ad-network (ads are everywhere on the internet, after all) that searches the disk for plain-text credentials and steals them.

So, this is a hard no.

Why not Secure Shell (SSH)? Link to heading

SSH is pretty neat, but it does have a a number of problems that decided me against relying on SSH for managing Git repositories: 1) the private key is stored on disk, 2) the key does not expire, 3) the key access controls cannot be scoped (deployment keys are a way around this), 4) SSH forwarding is insecure and 5) the SSH port 22 is often blocked on large managed networks, for example university networks.

Problem 1: SSH key stored on disk Link to heading

In the classic Git Credential Store example the SSH private key is an ordinary file stored on disk. This means that pretty much all the security considerations that apply to cleartext passwords also apply to SSH keys. If the disk is compromised (e.g. drive-by trojan from an infected ad-network) then all SSH keys stored on disk will be collected as fast as any cleartext passwords.

Password for the SSH private key? While it is possible to set a password for the SSH key, the additional protection is limited. It is quite likely that on a compromised machine all command line input, including the password to the SSH key when entered, would be collected and exfiltrated as well.

Storing the SSH private key in the system keyring? Of course it is possible to store the SSH key in the system keyring, same as the PAT. This, however, requires setting up the system keyring and negates any convenience benefits of accessing remote repositories with the simple SSH setup. If the system keyring is in use one might as well go with a PAT.

Problem 2: SSH key does not expire Link to heading

SSH keys do not expire. I don’t think it is possible to set an expiry date on a local SSH keypair and I have found nothing to the contrary so far.

It is possible to set an expiry date for an uploaded public key, if the server supports this. However, GitHub does not support expiry dates for uploaded SSH public keys.

If the key is compromised, this means the key will be compromised for a very, very long time. Which can be very, very bad. The recently uncovered XZ Utils Backdoor was planted by a covert operative who embedded himself in an Open Source project for several years. It’s not unthinkable that a compromised SSH key could be used for mischief many years down the line.

A special problem arises with SSH deploy keys, which are not linked to particular users and do not expire when the user is removed from an organization.

Problem 3: Cannot scope SSH permissions Link to heading

Regular SSH access to Github does not permit fine-grained access control. The SSH connection to Github is allowed to do everything that can be done via API, which can be somewhere between bad and catastrophic if the key is stolen.

Deploy keys are a partial solution to the scoping problem, but they can only be created with single-repository scope. This can be a problem for some use cases, but in my own workflow it would be a workable alternative. That being said, I think many developers use unrestricted SSH keys in their regular workflows.

PATs on the other hand always permit scoping of permissions and can be tailored to specific use cases. It is possible to create a more encompassing PAT for a local secure machine and to create additional limited PATs, for example with read-only access to single repositories for use on a deployment server.

Problem 4: SSH Agent Forwarding is insecure Link to heading

Normally a the private key needs to be present on the machine accessing the remote Git repository. Storing an unrestricted private SSH key on the disk of a local machine is a medium risk, on remote machines it is an unacceptable risk. Deployment keys with single repository scope are an alternative, but may be impractical during development.

SSH Agent Forwarding solves this issue by forwarding the functionality of the SSH agent, but not the private key itself. The SSH Agent Forwarding workflow looks something like this:

The deployment server is the place where the code stored in the repository is executed, usually some kind of VM in the cloud. If the deployment server is compromised, then the access rights of the SSH agent can be misused for the duration of the SSH connection. This is much better than losing the SSH key permanently, but it is still quite bad because of the universal access rights of regular SSH keys on GitHub.

In terms of security tradeoffs, compromised SSH Agent Forwarding should have a smaller temporal breach scope than losing a PAT. An SSH connection should not be active for more than a few hours at a time, but it might be compromised on more than one occasion if the breach isn’t noticed. A compromised PAT will be valid until its expiry, usually a maximum of 30 days.

On the other hand, a breached PAT with narrow scope (e.g. only the code that is being deployed on the remote server) would cause much smaller losses than the near-full-account access of SSH connections with an unrestricted SSH key. Repository-linked deployment keys are an alternative.

Problem 5: SSH port 22 often blocked on managed networks Link to heading

Accessing remote Git repositories via API is usually done over two protocols: HTTPS or SSH. HTTPS is the regular web protocol and runs on port 443. SSH is normally used for remotely administering servers and runs on port 22.

The main difference: HTTPS is used by regular people, SSH by power users.

Port 443 needs to be open to access any kind of normal webpage, a ton of REST APIs that run over HTTPS and for many other reasons. Blocking it is like blocking the internet, so this is simply not done on large managed networks (e.g. universities, corporations, etc.).

Port 22 on the other hand is used by very few people, usually those with advanced computer skills, usually those who can wreck things on the network. So it is often blocked for security reasons.

Now, SSH can be tunnelled over the HTTPS port, but this requires extra setup and firewall rules or proxy servers may still interfere. Since I spend a good amount of time connected to university networks, PATs over HTTPS are easier.

Why not OAuth? Link to heading

I still need to try the OAuth credential helper. It seems to be an interesting combination of two-factor authentication combined with the Git Cache.

I do wonder if it is more convenient than storing a PAT in the system keyring. I also wonder if it is more secure than a tightly scoped PAT (I didn’t see any scoping options while reading up on it, but may have missed them).

Something to try in the future!

Conclusion Link to heading

This concludes my reflections on a problem I had some time ago: how to access remote Git repositories securely via command line and where to store the secrets.

I use Git Cache and the Linux keyring in my personal workflow. I’m interested in OAuth, but haven’t gotten around to trying it. The essay contains setup instructions for cache and keyring.

I use the KeepassXC password manager to store all my PATs, so adding a credential helper interface might be useful. Unfortunately my specific system setup makes this impractical. It does seem to be a good option, though.

I decided against using the native Git Credential Store because it stores PATs in cleartext on the disk. I decided against SSH because the private key is stored on disk, the key does not expire, the key cannot be scoped as well as a PAT, SSH forwarding is insecure and SSH port 22 is often blocked on large managed networks.

Hopefully this was useful to someone, if only to my future self wondering why I spent all that time thinking about Git authentication.