Project Toolchain

Project Toolchain is a set of API tools that were introduced to standardize the code-generation aspect of Prisma Util. This page will provide an in-depth view of the inner workings of this addition.

What is this article?

This article provides a more in-depth look at Project Toolchain and how it operates under the hood. This is NOT a necessary concept and is quite an advanced topic. We recommend skipping this page if you're new to Prisma Util.

If you wish to skip this article, click here to go to the next concept.

While developing Prisma Util, we've stumbled across a problem: we needed to edit the PrismaClient code (both generated and installed via @prisma/client), but we didn't have an easy way of doing so. And that's where Project Toolchain started.

The idea behind Project Toolchain was to create an interface that can be easily maintained and allows modification of code assets easily. We also wanted to provide an easy-to-use API for developers using Prisma Util that were relying on configuring each generation path manually. We fixed this issue by standardizing everything under an optional module which is currently known as Project Toolchain.

In this article, we'll talk about how Project Toolchain edits assets and generates new ones. We'll cover internal handlers and how we made use of strategic transactions to make sure that we don't end up with duplicate code.

Defining Terms

During this article, we'll make heavy use of the terms defined below. While some names might match up with other programming paradigms, they are not to be used as definitions for said paradigms, but strictly as what they are defined here.

Asset

An asset is a file that can be edited using Project Toolchain.

Edit Block

Changes within a file are always split between multiple blocks to ensure that no duplicates are added. An edit block is an instruction that stipulates how a section of an asset should be edited.

Transaction

A transaction is a collection of edit blocks that will be executed for a specific asset. A transaction can only make changes in a single file. Changes in multiple files are always split between multiple transactions and added to the generator queue.

Transaction / Generator Queue

The transaction queue represents a list of Transactions that will be executed synchronously to avoid index modification in files.

Duplicate & Index Safety

Before executing an edit block, the Generation API will ensure that the asset being edited hasn't been modified already. This check will run the replaced value against the current state of the asset. If the new content has been found in the asset, this code block is marked as skipped and Project Toolchain continues to the next block.

Due to the nature of the Generation Toolchain, the new state of an asset is determined by splitting the current content and appending the modified lines. While this system works great, it comes with a small caveat: every time an edit block is ran, the current state has to be refreshed. To handle this, Project Toolchain will create an in-memory state of the asset and modify it directly to ensure that there's only one commit to the file: the final version. By doing this, we have essentially minimized generation times by keeping an in-memory state of the current asset being changed by the transaction to avoid double edits.

Handling Middleware

The most common use-case for developers using Prisma Util's Project Toolchain module is to import middleware generated by the Project Toolchain Middleware API. The current system works great because it provides a much-needed level of abstraction that works like this:

Scope Splitting

To make importing middleware easier for developers, we've decided to scope them based on the feature name. By creating this clear distinction between features, we are able to create a folder for each feature to structure the code better. The file structure for each feature is explained here:

File Structure

[FEATURE_NAME]
    generated
        [MIDDLEWARE_NAME].js
    index.js
    index.d.ts

As you can see, the file structure is quite compact and provides everything needed for a great developer experience. The next sections will explain why we chose this file structure.

Optimizing Writes

To make the generation process easier to understand, we've prepared this example for the pg_trgm Support feature. This feature defines a folder called pgtrgm and exports a middleware named ftsIndexes. When running your generators, this execution chain will occur:

1. The Project Toolchain Middleware API will receive a request for a feature called pgtrgm that wants to create a middleware named ftsIndexes with the code CODE.

2. Project Toolchain will define an internal flag for this feature called existing. This flag will be used later on in the process and has the default value of true.

3. Project Toolchain will check for the prisma-util/toolchain/middleware/pgtrgm folder. If this folder doesn't exist, it is created now and the existing flag is set to false.

4. If the existing flag is set to false, Project Toolchain will create a folder called generated inside of prisma-util/toolchain/middleware/pgtrgm. Then, in this folder, it will create a file called ftsIndexes.js and set the CODE as the content of this file. If the existing flag is set to true, Project Toolchain will:

   1. Check the current list of middleware to generate.

   2. Get a list of all of the files in the generated folder and remove all files that haven't been requested for generation during this run.

   3. Create a sha256 hash for each of the files in the generated folder that have been marked for generation during this turn. This hash will be mentioned as HASH1 in the following steps.

   4. Create a sha256 hash for the CODE of each middleware that will be generated during this run. This will ensure that no unnecessary writes are done. This hash will be mentioned as HASH2 in the following steps.

   5. Check if the corresponding pairs of hashes (HASH1 and HASH2 for each generation entry) match. Based on the result of this check, Project Toolchain will perform different actions. If the pair for a file matches, the file is left unmodified. If the pair for a file doesn't match, the content of the file is replaced with the CODE.

5. The index.js and index.d.ts files will be generated as stipulated in the following sections. These files are created after every run to ensure validity.

The index.js and index.d.ts Files

These files provide the entry point to the middleware generated by Project Toolchain. To make everything easier to import, the index.d.ts file will be generated like this:

index.d.ts

import { PrismaClient } from '@prisma/client';
declare type AvailableMiddleware = "middleware1" | "middleware2";
declare type ExportedMiddlewareBase = {
    [middlewareName in AvailableMiddleware]: (prisma: PrismaClient) => (params: any, next: (args: any) => Promise<any>) => Promise<any>;
} & {
    /**
     * Add all middleware defined in this folder.
     * 
     * @param prisma The instance of the PrismaClient that will be modified.
     * @returns The instance of the PrismaClient passed as an argument.
     */
    all: (prisma: PrismaClient) => PrismaClient;
};
declare const middlewareBase: ExportedMiddlewareBase;

// These comments will not be added to the final file.
// If there is only one middleware available, the following line will be added:
export default [MIDDLEWARE_NAME];
// If there are more than two middleware available, the following lines will be 
// added instead:
export default middlewareBase;
import [MIDDLEWARE_NAME_1] from "./generated/[MIDDLEWARE_NAME_1]";
import [MIDDLEWARE_NAME_2] from "./generated/[MIDDLEWARE_NAME_2]";
export { [MIDDLEWARE_NAME_1], [MIDDLEWARE_NAME_2] };

This file will always be re-generated to ensure that it is up-to-date with the current list of middleware available to use. Along with this file, the index.js file will always be generated like this:

index.js

// These comments will not be added to the final file.
// If there is only one middleware available, the following lines will be added:
import [MIDDLEWARE_NAME] from "./generated/[MIDDLEWARE_NAME]";
export default [MIDDLEWARE_NAME];
// If there are more than two middleware available, the following lines will be 
// added instead:
import [MIDDLEWARE_NAME_1] from "./generated/[MIDDLEWARE_NAME_1]";
import [MIDDLEWARE_NAME_2] from "./generated/[MIDDLEWARE_NAME_2]";

const middlewareBase = {
    all: (prisma) => {
        prisma.$use([MIDDLEWARE_NAME_1](prisma));
        prisma.$use([MIDDLEWARE_NAME_2](prisma));
        return prisma;
    },
    [MIDDLEWARE_NAME_1],
    [MIDDLEWARE_NAME_2]
};

export default middlewareBase;
export { [MIDDLEWARE_NAME_1], [MIDDLEWARE_NAME_2] };

Restrictions

To ensure the proper functioning of Project Toolchain, the following rules apply:

• A middleware can't be named all, because it is reserved by the all function.

Future Plans

We plan on releasing the Project Toolchain internal APIs to developers willing to make custom generators for Prisma Util, but this will come at a later stage.