This was really thought provoking — it made me realize that Git just happens to use a filesystem for persistence, but doesn’t necessarily have to. A POSIX filesystem might not even be the best way to store a git repo. Makes me wonder: what else could speak Git + POSIX? Redis? Postgres? IPFS is a fun one — it’s already content addressed.
supriyo-biswas 18 minutes ago [-]
Great work, though I wish some of this work could be upstreamed to Gitea instead?
xena 15 minutes ago [-]
Author of the post here. I'll talk with someone I know at Gitea. I don't think this is viable to upstream into Gitea, but there's only one way to find out!
nolist_policy 1 hours ago [-]
If you want to store a git repo on S3, you can that with git-annex[1] today. It can do client side encryption and large files as well.
Came here for a five-gallon bucket hooked to Dulwich (archiving rain?), Slightly disappointed :)
Go Git and Dulwich and friends are indeed fun tech.
colechristensen 25 minutes ago [-]
I did something similar, though a full reimplementation of a git and git-lfs library in Elixir. Still a work in progress though as the S3 backend isn't quite complete and there are performance problems doing some git things through S3.
The documentation isn't quite correct, but it's getting there
Eikon 2 hours ago [-]
Most of the pain here is the typical set of issues people run into trying to make S3 a filesystem as-is, common with S3FS-family approaches.
ZeroFS (https://github.com/Barre/zerofs) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed.
Different point on the spectrum, but less square-peg, also most probably much, much faster (it works great on linux-sized repos) :)
supriyo-biswas 11 minutes ago [-]
Would probably be great to have ZeroFS backed PVs for the K8s folks :)
znnajdla 35 minutes ago [-]
I wouldn’t call it gymnastics. The surprising part of the article was that Git itself is an object store that happens to use a filesystem for persistence, but an S3 bucket might actually be more suitable than a .git directory on POSIX.
21 minutes ago [-]
23 minutes ago [-]
xena 2 hours ago [-]
Author of the article here. I'm aware of ZeroFS and other similar approaches (such as something internal at Tigris that will become public at a later date), this was more of an experiment to see how far you can get with stuff I already had "on the shelf". I am going to be improving this a fair bit; I just need to plan out what I'm gonna work on and figure out the best times to stream it, etc.
[1] https://git-annex.branchable.com
Go Git and Dulwich and friends are indeed fun tech.
https://anvil.fangorn.io/fangorn/ex_git_objectstore
The documentation isn't quite correct, but it's getting there
ZeroFS (https://github.com/Barre/zerofs) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed.
Different point on the spectrum, but less square-peg, also most probably much, much faster (it works great on linux-sized repos) :)