Customize the Build Pipeline in C3 AI Release Management
A build pipeline has a default set of steps to download a repository, generate artifacts for all the packages, and run tests for those packages. However, users can customize the pipeline in a few different ways:
- Overwrite existing steps in the default pipeline
- Insert custom steps into the default pipeline
- Author a completely new pipeline
Registering branch groups
To enable customizability, a path to custom lambdas representing pipeline steps (jarvisFilePath) must be specified for a group of branches (JarvisService.BranchGroup). Navigate to the URL for the static console of the jarvisservice app https://<cluster>/<env>/jarvisservice/console/index.html to make this change by running the following command:
// Initialize variables based on the branch group you registered
var repositoryUrl = 'HTTPS Repository URL';
var packagesPath = 'Path to the parent folder containing your packages';
var regex = 'The regex for the branches you registered';
var customLambdasPath = 'Path to the folder containing your custom lambda files';
var branchGroup = JarvisService.BranchGroup.fetch({
filter: Filter.eq('repository.url', repositoryUrl).and().eq('packagesPath', packagesPath).and().eq('regex', regex)
}).objs[0];
branchGroup.withJarvisFilePath(customLambdasPath).merge();If the folder specified by jarvisFilePath is empty or does not exist, the default pipeline runs as usual.
To register a branch group and configure the jarvisFilePath field through the C3 AI Studio UI, refer to Register, Update, and Delete a Group of Branches. The jarvisFilePath can also be set through the Jarvis service console as shown in the example above.
Create a custom lambda
A custom lambda should be a .js file located in the jarvisFilePath and should follow the following format:
data = {
name: "stepName",
hookStep: "stepToHookInto",
hookStrategy: "hookStrategy",
packageName: "packageName",
packageVersion: "packageVersion",
value: function(step) {
// Lambda logic goes in here
},
};
| Variable | Description | Required |
|---|---|---|
name | Specify a name for the custom step. If the name is the same as the name of a step that exists in the default pipeline, this custom lambda overwrites the lambda of the original step. | Yes |
hookStep | The step for the custom step to "hook into". When the hook step is complete, this custom step is run afterward. When authoring an entirely new pipeline, this field can be left empty. If this field is provided, hookStrategy must also be provided. | Optional |
hookStrategy | How the custom step should be run after the hookStep. 2 strategies are currently supported: "blocking" and "async". Imagine an original pipeline with 3 steps A -> B -> C. If custom step D is hooked into step B with an "async" strategy, C and D are run concurrently. Otherwise, D has to finish first before C is allowed to run. | Optional |
packageName | The package that the custom step should be run in. If packageVersion is also specified, Release Management configures the package from ArtifactHub with the specified version, spins up a new app with the root package as packageName, and runs the custom step in that new app. | Optional |
packageVersion | If neither packageName nor packageVersion is provided, the custom step is run as normal in the jarvisExecutor app and only has access to regular platform Types. If the step is run inside a specified app, all the Types for that app are loaded in and you can use the Types/functions you want. | Optional |
value | The lambda that we want to run. | Yes |
Overwrite existing steps in the default pipeline
To overwrite an existing step, a user can create a .js file in the format shown above and specify the name of the step they wish to overwrite. The main steps for the pipeline are as follows:
| Variable | Description |
|---|---|
buildInit | This step is the first step of the pipeline that initializes the rest of the build steps. |
stashRepo | This step downloads the source control repository being tested as a ZIP file and stashes it for later use in the pipeline. Other steps can now download the repository content, depending on the source control service. |
generatePackageSteps | This step 1. analyzes the downloaded repository for packages that live in the specified packagesPath, 2. constructs and persists artifacts for all packages found, and 3. adds generateUiBundles (if necessary) and testPackages steps that should be run afterwards. |
testPackages | This step runs all tests assigned to a partition. If nothing was configured, this step runs all tests for a single package. |
buildSummary | This step is the last step of the pipeline that summarizes the build. |
Although overwriting existing steps is possible, it requires a solid understanding of what each step is doing and how it links to other steps. For example, the stashRepo and generatePackageSteps steps use Jarvis.addSteps to add other pipeline steps such as testPackages and buildSummary so the overwritten step may need to incorporate similar logic.
Insert custom steps into the default pipeline
Rather than overwriting an existing step, a user may want to insert a new step somewhere in the pipeline. In this case, a user can give their step a new name that does not overlap with any other step names and, at the minimum, specify values for the hookStep and hookStrategy fields. If a user wants to create a custom step that runs after stashRepo but before generatePackageSteps, which is added by stashRepo, they would specify hookStep as stashRepo and hookStrategy as "blocking".
The other two optional fields can be specified if the step should be run within a particular app. If packageVersion is not specified, it will try to start an app using the artifact created in the current build. However, if the artifact has not been created yet (by default, during the generatePackageSteps step), an error will be thrown.
How to call the Jarvis service within a custom app
Regular steps are run inside the jarvisExecutor app, which can talk directly to the Jarvis service. However, this is not necessarily true for a custom app. Thus, for a custom app to be able to communicate with the Jarvis service, its package will have to list jarvis as one of its dependencies in its .c3pkg.json file.
A user can then persist reports in one custom step by calling Jarvis.fileReports and then fetch them in another step with Jarvis.reportForId. Build-level configs can also be retrieved during a step with Jarvis.buildConfigValue. Refer to Create and View Build Configurations for C3 AI Release Management for further information.
Author a completely new pipeline
A user can author a completely new pipeline by modifying the firstStepName field in Jarvis.BranchGroup.TriggerOptions for the desired JarvisService.BranchGroup. Note that the buildInit and stashRepo steps will still be run since they are necessary for downloading the repository and reading the custom lambdas. The firstStepName is by default generatePackageSteps. Assuming you have already fetched the necessary branch group, you can run the following script in the static console of the jarvisservice app https://<cluster>/<env>/jarvisservice/static/console/index.html like before:
var firstStepName = 'DESIRED FIRST STEP NAME HERE';
// Use the previous script for setting jarvisFilePath to initialize branchGroup variable
var triggerOptions = branchGroup.get().triggerOptions;
branchGroup.withTriggerOptions(Jarvis.BranchGroup.TriggerOptions.make(triggerOptions).withFirstStepName(firstStepName)).merge();