Here’s a basic GitLab CI/CD YAML setup demonstrating the use of the include: configuration option, along with an explanation.

The include: keyword in GitLab CI/CD is a powerful feature that allows you to break down your pipeline configuration into smaller, manageable, and reusable pieces. You can include external YAML files, which can be from your local project, a remote URL, another project, or predefined templates provided by GitLab.

This example will show two common use cases:

1. include:local: Including a YAML file from the same repository.
2. include:template: Including a predefined CI/CD template provided by GitLab.

Example Structure:

Let’s assume you have the following file structure in your GitLab repository:

.
├── .gitlab-ci.yml         # Main CI/CD configuration file
└── .gitlab-ci/            # Directory for CI/CD related files (optional, good practice)
    └── common-build-job.yml # A reusable build job definition
Code language: PHP (php)

File 1: .gitlab-ci/common-build-job.yml

This file will contain a reusable job definition. It could be a job template (starting with a .) that other jobs can extend, or a complete job. Here, we’ll define a job template for a common build step.

YAML

# .gitlab-ci/common-build-job.yml

.common_build_template: # Note the '.' making this a hidden job/template
  image: alpine:latest
  tags:
    - docker
  before_script:
    - echo "Setting up build environment from common template..."
  script:
    - echo "This is a common build step defined in common-build-job.yml"
    - sleep 10 # Simulate some work
  after_script:
    - echo "Cleaning up build environment from common template."
Code language: PHP (php)

Explanation of .gitlab-ci/common-build-job.yml:

• .common_build_template:: This defines a “hidden” job template. Hidden jobs are not run directly but can be extended by other jobs. The leading . (dot) signifies this.
• image, tags, before_script, script, after_script: These are standard job keywords defining how this template behaves.

File 2: .gitlab-ci.yml (Main CI/CD file)

This is your main pipeline configuration file that will use include: to pull in the configurations from other files.

YAML

# .gitlab-ci.yml

include:
  - local: '.gitlab-ci/common-build-job.yml' # Includes a local file
  - template: Jobs/Secret-Detection.gitlab-ci.yml # Includes a GitLab-provided template

stages:
  - build
  - test
  - security # Stage for the included Secret Detection job

variables:
  GLOBAL_VARIABLE: "This is available everywhere"

# Job extending the local template
build_application:
  stage: build
  extends: .common_build_template # Extends the template from the included file
  script:
    # The script from .common_build_template will run first
    - echo "Building the main application..."
    - echo "Using variable: $GLOBAL_VARIABLE"
    # The after_script from .common_build_template will run after this

# A regular job defined in the main file
unit_tests:
  stage: test
  image: node:18
  script:
    - echo "Running unit tests..."
    - npm install
    - npm test

# The Secret-Detection job will be automatically added from the included template
# It typically runs in a 'test' or 'security' stage.
# We added 'security' stage above to accommodate it if it uses that.
Code language: PHP (php)

Explanation of .gitlab-ci.yml:

1. include: Section:
   • This top-level keyword is used to list external YAML files whose contents will be merged into this main configuration.
   • local: '.gitlab-ci/common-build-job.yml':
     • This tells GitLab to find the common-build-job.yml file within the same repository at the specified path.
     • The configurations (like the .common_build_template) from this file become available as if they were written directly in .gitlab-ci.yml.
   • template: Jobs/Secret-Detection.gitlab-ci.yml:
     • This includes one of GitLab’s predefined CI/CD templates. GitLab offers many templates for common tasks like security scanning (Secret Detection, SAST, DAST), deployments, etc.
     • The Secret-Detection.gitlab-ci.yml template automatically adds a job to your pipeline that scans your code for inadvertently committed secrets.
2. stages:
   • Defines the execution order for jobs. Note that included templates might also define their own stages or expect certain stages to exist. The Secret-Detection.gitlab-ci.yml template, for example, often adds jobs to a test or a dedicated security stage. I’ve added security here for clarity.
3. variables:
   • Defines a global variable.
4. build_application: Job:
   • stage: build: Assigns the job to the build stage.
   • extends: .common_build_template: This is a key feature used with included templates. It tells this job to inherit all the configurations from the .common_build_template (which was loaded via the include:local statement).
     • The image, tags, before_script, and after_script from .common_build_template will be used.
     • The script from .common_build_template will run before the script defined in build_application.
   • script:: Adds specific commands for this job.
5. unit_tests: Job:
   • This is a standard job definition, independent of the included files (though it could also extend templates if needed).
6. Implicit Secret Detection Job:
   • You don’t see a job named secret_detection explicitly defined in this main file, but because you included the Jobs/Secret-Detection.gitlab-ci.yml template, GitLab will automatically add the job(s) defined in that template to your pipeline.

How include Works and Merging:

• GitLab fetches the included files and performs a deep merge of their contents with the main .gitlab-ci.yml configuration.
• Order Matters: If multiple included files define the same job or global keyword, the last one included usually takes precedence, but this can be complex. Job definitions are generally additive.
• Global Keywords: Keywords like stages, variables, default, and workflow from included files are merged. For stages, the list of stages is the union of all defined stages. For variables and default, later definitions usually override earlier ones.
• Jobs: Jobs from all included files are added to the pipeline. If a job with the same name exists in multiple files (or in the main file and an included one), the behavior can be tricky; usually, the one defined in the main .gitlab-ci.yml or the last included file that defines it takes precedence, effectively overriding others. It’s generally better to ensure job names are unique or use the extends mechanism with templates.

Benefits of Using include:

• Modularity: Break down large, complex pipeline configurations into smaller, focused files.
• Reusability: Define common job configurations (like build steps, test setups, deployment scripts) once and reuse them across multiple pipelines or projects (using include:project or include:remote).
• Maintainability: Easier to manage and update specific parts of your CI/CD logic without navigating a massive single YAML file.
• Standardization: Enforce standards by providing common templates for teams to include (e.g., security scanning, compliance checks).
• Leverage GitLab Templates: Quickly add best-practice CI/CD capabilities for things like security scanning, dependency checking, etc., using GitLab’s curated templates.

Other include Methods (Briefly):

• include:project: 'namespace/project'
  • file: '/path/to/file.yml'
  • ref: 'branch_or_tag_or_commit'
  • Includes a file from another project within the same GitLab instance.
• include:remote: 'https_url_to_raw_yaml_file'
  • Includes a YAML file from a publicly accessible URL.

This example should give you a good starting point for using include: to make your GitLab CI/CD pipelines more organized and efficient!