Accelerate by years part I - Inlang directories as lix repositories

This is part 1 of a 4-part series of memos that propose steps to accelerate lix and inlang by multiple years.

Accelerate by years part I - Inlang directories as lix repositories

Accelerate by years part II - Self-contained inlang files

Accelerate by years part III - Lix on SQLite

Accelerate by years part IV - The prototype

LOOM

https://www.loom.com/share/1cddb195df2f41ed98c320fab6001b33?sid=ea253213-e96f-4b89-a8e3-faf99738c4ea

TL;DR

If we find a way to “store” lix repositories in git that push and pull from a lix hosting service, we no longer have to host inlang directories on GitHub. If we don’t have to host inlang directories on GitHub, we will control the entire lix workflow (push, pull, commit, merge, ci/cd, etc) and don’t have to hack around Git and GitHub anymore. 

image

Context

Inlang directories depend on external git repositories, which means:

• We spent months building lix with little return in getting closer to differentiable lix features (the hosting platform, semantic merging, or third-party lix apps). 

• Scalability requirements are high because inlang apps operate on an entire git repo instead of just the inlang directory.

image

Proposal

Turn inlang directories into self-contained repositories to become independent of git. 

• Speed up the path to differentable lix features lix by years. Self-contained inlang repositories have lower requirements for hosting, network capabilities, security, and other aspects than touching source code. 

• Eliminate bugs caused by the lack of a repository [paraglide-js#44, cli#5, paraglide–js#51])

image

Benefits

Scalability requirements are much lower

Hosting inlang “files as repositories” leads to substantially smaller repositories, reducing the scalability requirements for networking, querying, and performance. 

We don’t need to support 100MB monorepos from day one. An inlang “file as repository” is <1MB large. 

image

Lix SDK controls push, pull, commit, etc.

Inlang apps, powered by the lix SDK, gain full control over change control workflows like merging, push, pull, etc. 

We unblock the path for the lix SDK to start building the semantic merge API, auth, permissions, ci/cd, etc., because we do not have to consider the parent git repository anymore. Commit, push, and pull happens via inlang apps, not the git CLI.

image

Ideal workflow

Web app perspective (Parrot, Fink)

1. Create a new inlang project that is stored in local storage.

2. Opt-in set up remote to push and pull from (lixnet)

Dev app perspective (Sherlock, Paraglide)

1. Create a new inlang project stored in a git repo

2. Changes are conducted via dev apps (full control via lix) and synced via the git repo

3. Opt-in to a remote sync via lixnet to collaborate with non-devs

Next steps

• Research the possibilities of turning inlang directories into self-contained repositories with an acceptable developer experience.

  • submodules?

  • subtrees?

  • something else?

Additional information

Things we invested time into that would be redundant

Lix lazy checkout (6 months effort)**

Inlang files (& other first lix use cases) are small enough and do not need lazy checkout. We could have invested 6 months of 1 full-time lix person on the lix hosting platform. 

Fink GitHub auth, pull requests, rate limits shenanigans (recurring effort ≈2-4 months)

Fink is crippled by GitHub. Constant problems with auth, pull requests, and lately even rate limits (wtf). If we had the hosting platform, we could have invested the recurring work and pain of dealing with GitHub auth and pull requests into lix auth and change proposals. 

Ninja (2 months effort)**

If we had the hosting platform, would we have built Ninja? Probably not. We would have invested the time into lix automation pipelines (CI/CD). 

Inlang slot file storage (3-5 months effort)

We decided to ignore lix while designing the inlang storage format because: 

Having a generalizable lix merge api seems out of scope as long as we have to be git compatible (aka as long as lixnet doesn't exist and is widely adopted)

Samuel Stroschein, https://github.com/opral/inlang-message-sdk/issues/74#issuecomment-2138329130

If we wouldn’t need to hack around git, we likely would have decided to invest the time into the lix file extension API instead of inventing a slot file storage to work around git. 3-5 months of work that could have been invested into a generalizable lix solution instead of a custom inlang workaround.