Accelerate by years part III - Lix on SQLite

This is part 3 of a 4-part series of memos that propose steps to accelerate lix and inlang by multiple years.

Accelerate by years part I - Inlang directories as lix repositories

Accelerate by years part II - Self-contained inlang files

Accelerate by years part III - Lix on SQLite

Accelerate by years part IV - The prototype

Context

We might have the possibility to accelerate lix even further by not building lix on Git. 

1. Previous memos of the “accelerating lix by years” series made the case that compatibility with external git repos (outside of the inlang file) is not required. 

2. We know that git is unsuited for applications. So if compatibility with external git repos is not required, is git required at all to build lix? 

Insight: Git is unsuited for applications

1. Applications have to store data in text files to be mergeable by git. Storing application data as text files leads to the massive downstream complexity of re-inventing a database: 

   1. A git-friendly custom storage layer must be developed [inlang-message-sdk#74].

   2. The custom storage layer needs to be connected to an in-memory database/query layer, which is either re-invented or a lacking off-the-shelf solution like RxDB is adopted [inlang-message-sdk#101] as a “throwaway MVP” because: 

      1. The entire database must be loaded into memory, which eventually needs to be optimized, aka re-inventing database optimizations [inlang-message-sdk#83,  inlang-message-sdk#97]. 

      2. An insufficient query API. Joins don’t exist, forcing us to create one “mega document” type MessageBundle that contains all data like messages, variants, lint reports, and more to accommodate the query API. A flaw that has a high computational cost and brings complexity into the data model. A migration to “something else” or custom optimizations (that reinvent what databases already have) are given once large organizations adopt inlang. 

2. Using git’s file-based storage model of changes (snapshots, commits, and pack files) will lead to re-inventing a database for the lix SDK for the same reason the inlang SDK had to re-invent a database: 

   1. Performant and complex (!) query APIs about changes need to exist. Git has no API. We would need to invent our own API. 

   2. We will likely need a custom storage layer over git or hacking of git that stores semantic changes.    

Opportunity 

Build lix 1.0 with the learnings from scratch based on SQLite and the semantic merging API.  

The requirements for lix v1 boil down to 1) a semantic merge() function based on 2) a plugin that inlang provides to merge .inlang files. Only two requirements for lix v1 if we drop support for git; that’s it. Inlang apps will be able to provide collaboration by calling the merge function. Wild.  

Benefits:

• We don’t re-invent a database in inlang and lix. We can store all inlang & lix data in SQLite databases. Concurrency, locking, and atomicity are all solved. Read “SQLite as an application file format”.

• SQL for applications and lix change control data, connectable via joins. 

• Third parties can start writing lix applications. 

• Remove unnecessary complexities of git:

  • No isomorphic-git fork

  • No filesystem complexities 

  • No need for a git server like Gitea for v1 of a hosting service

  • No redundant git features that might not be needed for apps (push, pull, staging, subtree, submodules, branches, etc.), or if needed, we will likely need to design them differently because of semantic merging. 

Downsides: 

• Auf Wiedersehen git (but so what). 

• We must build tooling around inlang files for merging e.g., git merge drivers, GitHub actions (that should be substantially easier than staying git compatible though!) 

Demonstration 

Query API: 

• Joins between Bundles, Messages, and Variants are possible :O 

• Querying the history of a message via lix is a breeze.

• Out-of-the-box JS query builder with Kysely.

```

const message = await project.query

  .selectFrom(‘message’)

  .innerJoin(‘bundle’, ‘bundle.messageId’, ‘message.id’)

  .innerJoin(‘lix.history’, ‘lix.history.messageId’, ‘message.id’)

  .select(“*”)

  .executeTakeFirst()

message.bundle

>> { human_id: “blabla”, … }

message.lix.history

>> [{ commit_id: “”, commit_author: “”, … }]

```

Example inlang file format: 

• The lix SQLite and inlang SQLite are kept separate to allow lix and inlang, respectively, to keep control of their data. 

• The file can be an SQLite archive to take full control over the fs and avoid platform-specific leaks [lix-sdk#3].

```

project.inlang                             ← (zip or a SQLite archive) 

• .lix                                  ← lix sqlite

• db.sqlite                         ← inlang sqlite   

```

Conclusion

A once-in-a-product lifetime opportunity to save years in building lix. 

We know that the minimum requirements for inlang are a merge function and a plugin API to provide lix with semantic understanding. Coincidentally, those two requirements are also the requirements for third-party developers to start building lix apps. We only have to figure out how the semantic plugin API for lix works. Figuring out how the plugin API looks like is surely easier than re-inventing databases, and … the differentiating part of lix. Exciting.