【2022鐵人賽】重構YAML範本：加入更多彈性

YAML範本設計終於來到了尾聲，前一篇連變數都把它範本化了，還有什麼地方可以優化的嗎？

有的！如果回頭檢視之前設計的YAML內容，肯定還是可以發現有什麼地方可以改得更好，只是一開始沒發現或是暫時沒時間弄而已。然而實際在我的工作上其實已經針對使用的YAML重構到了第三個版本，雖然沒有完全寫在鐵人賽的文章內容中，但是大致上的精髓也是十之八九，就來看看還有什麼東西可以調整的吧！

加入stepList型態參數

之前在認識Build Pipeline的參數(Parameters)與變數(Variables)這篇文章中有提到參數的型態，除了常用的string、boolean這種常值類型的以外，還有對應stage、job、step與它們的集合類型的參數，而stepList就是可以加入許多額外的step的參數類型。

如果你是寫過程式的開發者，我想應該都有事件驅動的概念，如果你曾經寫過.Net WinForm的程式，應該更能理解我在這裡要加入的概念。

還記得我們的Job有分成三個吧？分別是BuildCode、BuildImage、DeployCloudRun，在這邊我希望能夠在這些Job實際執行動作「之前」與「之後」加入一些能夠讓外部動態增加一些step的彈性。這裡的外部不一定指的是範本以外的專案，也可以是範本中不同層級的YAML範本。

BuildCode增加的彈性

在BuildCode的範本裡面我打算加入codePreBuildSteps、codePostBuildSteps這兩個參數，分別代表程式碼在編譯之前與編譯之後可以額外加入step的區段。

為了這麼做，首先必須在BuildCode.yaml中加入對應的參數設定如下：

# jobs/buildCode.yaml

  #======= Extensibility ========
  - name: codePreBuildSteps
    type: stepList
    default: []
  - name: codePostBuildSteps
    type: stepList
    default: []

接著在使用dotnet-sdk-in-linux-container.yaml之前加入codePreBuildSteps的參數，在結尾的部份加入codePostBuildSteps參數，YAML內容部份如下：

# jobs/buildCode.yaml

      - ${{ parameters.codePreBuildSteps }}
      - template: ../steps/dotnet-sdk-in-linux-container.yaml
        parameters:
          srcPath: ${{ parameters.sourcePath }}
          slnOrCsprojName: ${{ parameters.slnOrCsprojName }}
          dotnetCommand: publish
          dotnetCommandArgs: '-c ${{ parameters.buildConfiguration }}'
      - ${{ parameters.codePostBuildSteps }}

因為buildCode.yaml是job層級，上面還有個stage template，所以在stages/dotnet-build-stage.yaml中也必須同樣加上上面的參數設定，然後在使用buildCode.yaml的部份加上對應的參數值。

# stages/dotnet-build-stage.yaml

    - template: ../jobs/buildCode.yaml
      parameters:
        sourcePath: ${{ parameters.sourcePath }}
        slnOrCsprojName: ${{ parameters.slnOrCsprojName }}
        codePreBuildSteps: ${{ parameters.codePreBuildSteps }}
        codePostBuildSteps: ${{ parameters.codePostBuildSteps }}

buildImage增加的彈性

以相同的概念在buildImage的部份則是加上imagePreBuildSteps、imagePostBuildSteps，不過這邊要加入的檔案有三個，分別是stages/dotnet-build-stage.yaml、jobs/buildImage.yaml、steps/docker-build.yaml。

參數設定如下：

  #======= Extensibility ========
  - name: imagePreBuildSteps
    type: stepList
    default: []
  - name: imagePostBuildSteps
    type: stepList
    default: []

加了參數設定之後，接下來稍微有點不太一樣，在buildImage.yaml加入了上面的參數之後，是在使用docker-build.yaml的地方設定參數值，真正使用到參數值的地方則是在docker-build.yaml裡面，會不一樣主要還是裡面的steps差異，如果不影響就可以放外面，會影響就放裡面。

# steps/docker-build.yaml

  - ${{ parameters.imagePreBuildSteps }}
  - task: Docker@2
    displayName: Build image
    inputs:
      repository: '${{ parameters.imgRepository }}'
      command: 'build'
      Dockerfile: ${{ parameters.buildDockerfile }}
      buildContext: ${{ parameters.buildContext }}
      ${{ if eq(parameters.buildArgs, '') }}:
        arguments: '--no-cache'
      ${{ else }}:
        arguments: '--no-cache --build-arg ${{ parameters.buildArgs }}'
      tags: ${{ parameters.imgTags }}
  - ${{ parameters.imagePostBuildSteps }}

deployCloudRun增加的彈性

在deployCloudRun的部份因為沒有細到step template，所以直接就是在deployCloudRun.yaml中加入參數設定與使用參數值了(同一份檔案就直接全貼內容了)。不過參數設定的部份還是要另外複製一份到stages/deploy-cloudrun-stage.yaml檔案中。

# jobs/deployCloudRun.yaml

parameters:
  - name: cloudRunProjectId
    type: string
  - name: imgRepository
    type: string
  - name: cloudRunRegion
    type: string
  - name: cloudRunServiceName
    type: string
  - name: templateResourceName
    type: string
    default: templates
  - name: pipelineResourceName
    type: string
    default: pipelines

  #======= Extensibility ========
  - name: preDeploySteps
    type: stepList
    default: []
  - name: postDeploySteps
    type: stepList
    default: []

  - name: jobName
    type: string
    default: DeployCloudRun

jobs:
  - job: ${{ parameters.jobName }}
    steps:
      - script: |    
          echo '${{ convertToJson(parameters) }}' >> parameters.json
          cat parameters.json
        displayName: Print template parameters
      - checkout: ${{ parameters.templateResourceName }}
        path: ${{ parameters.templateResourceName }}
        clean: true
      - checkout: ${{ parameters.pipelineResourceName }}
        path: ${{ parameters.pipelineResourceName }}
        clean: true
      - ${{ parameters.preDeploySteps }}
      - template: ../steps/merge-cloudRun-envVars.yaml
        parameters:
          templateResourceName: ${{ parameters.templateResourceName }}
          pipelineResourceName: ${{ parameters.pipelineResourceName }}
          projectEnvVarsFile: variables/cloudrun-envVars.yaml
          sharedEnvVarsFile: variables/cloudrun-envVars.yaml
          mergedNewFileFullPath: $(Agent.BuildDirectory)/mergedEnvVars.yaml
      - task: Bash@3
        displayName: Deploy docker image to cloudrun
        inputs:
          targetType: 'inline'
          script: |
            docker run --rm \
            -v $(Agent.BuildDirectory)/${{ parameters.templateResourceName }}/ironman2022-gcp-key.json:/gcp/cloudKey.json \
            -v $(Agent.BuildDirectory)/mergedEnvVars.yaml:/gcp/env-vars.yaml \
            asia.gcr.io/google.com/cloudsdktool/google-cloud-cli:latest \
            bash -c "gcloud auth login --cred-file=/gcp/cloudKey.json && gcloud run deploy ${{ parameters.cloudRunServiceName }} --env-vars-file /gcp/env-vars.yaml --image ${{ parameters.imgRepository }} --region ${{ parameters.cloudRunRegion }} --project ${{ parameters.cloudRunProjectId }} --allow-unauthenticated"
      - ${{ parameters.postDeploySteps }}

stages/deploy-cloudrun-stage.yaml檔案中的內容則是下面這段：

# stages/deploy-cloudrun-stage.yaml

parameters:
  # deployCloudRun.yaml parameters
  - name: cloudRunProjectId
    type: string
  - name: imgRepository
    type: string
  - name: cloudRunRegion
    type: string
  - name: cloudRunServiceName
    type: string
  - name: templateResourceName
    type: string
    default: templates
  - name: pipelineResourceName
    type: string
    default: pipelines

  #======= Extensibility ========
  - name: preDeploySteps
    type: stepList
    default: []
  - name: postDeploySteps
    type: stepList
    default: []

  - name: stageName
    type: string
    default: DeployCloudRun
    
stages:
- stage: ${{ parameters.stageName }}
  displayName: 佈署Cloud Run
  jobs:
    - template: ../jobs/deployCloudRun.yaml
      parameters:
        cloudRunProjectId: ${{ parameters.cloudRunProjectId }}
        imgRepository: ${{ parameters.imgRepository }}
        cloudRunRegion: ${{ parameters.cloudRunRegion }}
        cloudRunServiceName: ${{ parameters.cloudRunServiceName }}
        templateResourceName: ${{ parameters.templateResourceName }}
        pipelineResourceName: ${{ parameters.pipelineResourceName }}
        preDeploySteps: ${{ parameters.preDeploySteps }}
        postDeploySteps: ${{ parameters.postDeploySteps }}

將jobName、stageName、dependsOn設為參數

如果你有仔細的看deployCloudRun.yaml和deploy-cloudrun-stage.yaml這兩個檔案的內容的話，應該會發現除了前面提到的stepList類型的參數之外，還另外增加了jobName或stageName參數。

這個修改就是讓外部可以重新設定job或stage的名稱(也就是識別ID)，如果沒有特別設定的話則是使用預設值，也就是原本的名稱。這麼做的用意在哪呢？

主要是為了dependsOn屬性，也就是Job的依賴關係。

其實在鐵人賽的文章中，我只有以.Net的程式為範例來舉例，所以編譯程式碼的Job YAML直接就叫作buildCode.yaml，但是實際上若不是完全只有使用一種程式語言的環境，那麼應該會分成好幾種不同程式語言的YAML檔，例如：dotnet-build.yaml、ios-build.yaml、node-build.yaml之類的，也就是在buildImage的部份如果dependsOn固定設定一個名稱，可能在範本使用上不太妥當，所以我將jobName、stageName和dependsOn都設為參數。

如何使用？

要使用到參數設定的stepList，就是在引用範本的parameters底下對應的參數直接使用task類型的YAML語法就可以了，如果是在範本中使用到之後又要開放給外部(上一層)使用，則是繼續使用參數值即可。

例如在buildImage之前要做一些事，像是額外下載需要的檔案，或是更改檔名或設定值之類的…，用stages/dotnet-build-stage.yaml的部份內容來舉例如下：

    - template: ../jobs/buildImage.yaml
      parameters:
        templateResourceName: ${{ parameters.templateResourceName }}
        dependsOn: BuildCode
        artifactName: ${{ parameters.artifactName }}
        unzip: ${{ parameters.unzip }}
        zipFileName: ${{ parameters.zipFileName }}
        unzipToFolderPath: ${{ parameters.unzipToFolderPath }}
        imgRepository: ${{ parameters.imgRepository }}
        imgTags: ${{ parameters.imgTags }}
        buildDockerfile: ${{ parameters.buildDockerfile }}
        buildContext: ${{ parameters.buildContext }}
        containerRegistry: ${{ parameters.containerRegistry }}
        buildArgs: "TARGET_FILE=${{ parameters.startFileName }}"
        imagePreBuildSteps: 
        - script: |
            echo "這是使用的範例1"
          displayName: 範例啦
        - task: Bash@3
          displayName: 還是範例啦
          inputs:
            targetType: 'inline'
            script: |
              echo "這是使用的範例2"
        - ${{ parameters.imagePreBuildSteps }}
        imagePostBuildSteps: ${{ parameters.imagePostBuildSteps }}

細部來看的話，就只有下面這段：

        imagePreBuildSteps: 
        - script: |
            echo "這是使用的範例1"
          displayName: 範例啦
        - task: Bash@3
          displayName: 還是範例啦
          inputs:
            targetType: 'inline'
            script: |
              echo "這是使用的範例2"
        - ${{ parameters.imagePreBuildSteps }}

最後的 – ${{ parameters.imagePreBuildSteps }}是讓範本設定同名的參數能延續使用，也就是不會讓外部設定的內容失效，簡單來說就是接下去的意思。

這次修改過的YAML檔案內容

下面就是這篇文章修改過的完整YAML檔案內容，

# steps/docker-build.yaml

parameters:
  - name: imgRepository
    type: string
  - name: imgTags
    type: object
  - name: buildDockerfile
    type: string
    default: Build.Dockerfile
  - name: buildContext
    type: string
  - name: containerRegistry
    type: string
  - name: buildArgs
    type: string
    default: ''

  #======= Extensibility ========
  - name: imagePreBuildSteps
    type: stepList
    default: []
  - name: imagePostBuildSteps
    type: stepList
    default: []

steps:
  - ${{ parameters.imagePreBuildSteps }}
  - task: Docker@2
    displayName: Build image
    inputs:
      repository: '${{ parameters.imgRepository }}'
      command: 'build'
      Dockerfile: ${{ parameters.buildDockerfile }}
      buildContext: ${{ parameters.buildContext }}
      ${{ if eq(parameters.buildArgs, '') }}:
        arguments: '--no-cache'
      ${{ else }}:
        arguments: '--no-cache --build-arg ${{ parameters.buildArgs }}'
      tags: ${{ parameters.imgTags }}
  - ${{ parameters.imagePostBuildSteps }}
  - task: Docker@2
    displayName: "Login to Container Registry"
    inputs:
      command: login
      containerRegistry: ${{ parameters.containerRegistry }}
  - task: Bash@3
    displayName: Push docker image
    inputs:
      targetType: 'inline'
      script: |
        docker push -a ${{ parameters.imgRepository }}

# jobs/buildCode.yaml

parameters:
  - name: sourcePath
    type: string
    default: $(Build.SourcesDirectory)
  - name: slnOrCsprojName
    type: string
    default: Pipeline.sln
  - name: buildConfiguration
    type: string
    default: release
    values:
    - release
    - debug

  - name: jobName
    type: string
    default: BuildCode

  #======= Extensibility ========
  - name: codePreBuildSteps
    type: stepList
    default: []
  - name: codePostBuildSteps
    type: stepList
    default: []

jobs:
  - job: ${{ parameters.jobName }}
    steps:
      - checkout: sources
        clean: true
      - template: ../steps/publish-commit-sha.yaml
        parameters:
          sourcePath: ${{ parameters.sourcePath }}
      - ${{ parameters.codePreBuildSteps }}
      - template: ../steps/dotnet-sdk-in-linux-container.yaml
        parameters:
          srcPath: ${{ parameters.sourcePath }}
          slnOrCsprojName: ${{ parameters.slnOrCsprojName }}
          dotnetCommand: publish
          dotnetCommandArgs: '-c ${{ parameters.buildConfiguration }}'
      - ${{ parameters.codePostBuildSteps }}
      - template: ../steps/publish-pipeline-artifacts.yaml
        parameters:
          publishPath: '$(Build.ArtifactStagingDirectory)/zipFiles'
          artifactName: '$(pipelineArtifact)'

# jobs/buildImage.yaml

parameters:
  - name: artifactName
    type: string
    default: OutputFiles
  - name: unzip
    type: boolean
    default: false
  - name: zipFileName
    type: string
    default: ''
  - name: unzipToFolderPath
    type: string
    default: ''

  # - name: pipelineResourceName
  #   type: string
  #   default: pipelines
  - name: templateResourceName
    type: string
    default: templates

  - name: imgRepository
    type: string
  - name: imgTags
    type: object
  - name: buildDockerfile
    type: string
    default: Build.Dockerfile
  - name: buildContext
    type: string
  - name: containerRegistry
    type: string
  - name: buildArgs
    type: string
    default: ''

  #======= Extensibility ========
  - name: imagePreBuildSteps
    type: stepList
    default: []
  - name: imagePostBuildSteps
    type: stepList
    default: []
  - name: dependsOn
    type: object
    default: []

  - name: jobName
    type: string
    default: BuildImage

jobs:
  - job: ${{ parameters.jobName }}
    dependsOn: ${{ parameters.dependsOn }}
    steps:
    - checkout: ${{ parameters.templateResourceName }}
      path: ${{ parameters.templateResourceName }}
      clean: true
    - template: ../steps/download-pipeline-artifacts.yaml
      parameters:
        artifactName: ${{ parameters.artifactName }}
        unzip: ${{ parameters.unzip }}
        zipFileName: ${{ parameters.zipFileName }}
        unzipToFolderPath: ${{ parameters.unzipToFolderPath }}
    - template: ../steps/docker-build.yaml
      parameters:
        imgRepository: ${{ parameters.imgRepository }}
        imgTags: ${{ parameters.imgTags }}
        buildDockerfile: $(Agent.BuildDirectory)/${{ parameters.templateResourceName }}/Dockerfile
        buildContext: ${{ parameters.buildContext }}
        containerRegistry: ${{ parameters.containerRegistry }}
        ${{ if ne(parameters.buildArgs, '') }}:
          buildArgs: ${{ parameters.buildArgs }}
        imagePreBuildSteps: ${{ parameters.imagePreBuildSteps }}
        imagePostBuildSteps: ${{ parameters.imagePostBuildSteps }}

# jobs/deployCloudRun.yaml

parameters:
  - name: cloudRunProjectId
    type: string
  - name: imgRepository
    type: string
  - name: cloudRunRegion
    type: string
  - name: cloudRunServiceName
    type: string
  - name: templateResourceName
    type: string
    default: templates
  - name: pipelineResourceName
    type: string
    default: pipelines

  #======= Extensibility ========
  - name: preDeploySteps
    type: stepList
    default: []
  - name: postDeploySteps
    type: stepList
    default: []

  - name: jobName
    type: string
    default: DeployCloudRun

jobs:
  - job: ${{ parameters.jobName }}
    steps:
      - script: |    
          echo '${{ convertToJson(parameters) }}' >> parameters.json
          cat parameters.json
        displayName: Print template parameters
      - checkout: ${{ parameters.templateResourceName }}
        path: ${{ parameters.templateResourceName }}
        clean: true
      - checkout: ${{ parameters.pipelineResourceName }}
        path: ${{ parameters.pipelineResourceName }}
        clean: true
      - ${{ parameters.preDeploySteps }}
      - template: ../steps/merge-cloudRun-envVars.yaml
        parameters:
          templateResourceName: ${{ parameters.templateResourceName }}
          pipelineResourceName: ${{ parameters.pipelineResourceName }}
          projectEnvVarsFile: variables/cloudrun-envVars.yaml
          sharedEnvVarsFile: variables/cloudrun-envVars.yaml
          mergedNewFileFullPath: $(Agent.BuildDirectory)/mergedEnvVars.yaml
      - task: Bash@3
        displayName: Deploy docker image to cloudrun
        inputs:
          targetType: 'inline'
          script: |
            docker run --rm \
            -v $(Agent.BuildDirectory)/${{ parameters.templateResourceName }}/ironman2022-gcp-key.json:/gcp/cloudKey.json \
            -v $(Agent.BuildDirectory)/mergedEnvVars.yaml:/gcp/env-vars.yaml \
            asia.gcr.io/google.com/cloudsdktool/google-cloud-cli:latest \
            bash -c "gcloud auth login --cred-file=/gcp/cloudKey.json && gcloud run deploy ${{ parameters.cloudRunServiceName }} --env-vars-file /gcp/env-vars.yaml --image ${{ parameters.imgRepository }} --region ${{ parameters.cloudRunRegion }} --project ${{ parameters.cloudRunProjectId }} --allow-unauthenticated"
      - ${{ parameters.postDeploySteps }}

# stages/deploy-cloudrun-stage.yaml

parameters:
  # deployCloudRun.yaml parameters
  - name: cloudRunProjectId
    type: string
  - name: imgRepository
    type: string
  - name: cloudRunRegion
    type: string
  - name: cloudRunServiceName
    type: string
  - name: templateResourceName
    type: string
    default: templates
  - name: pipelineResourceName
    type: string
    default: pipelines

  #======= Extensibility ========
  - name: preDeploySteps
    type: stepList
    default: []
  - name: postDeploySteps
    type: stepList
    default: []

  - name: stageName
    type: string
    default: DeployCloudRun
    
stages:
- stage: ${{ parameters.stageName }}
  displayName: 佈署Cloud Run
  jobs:
    - template: ../jobs/deployCloudRun.yaml
      parameters:
        cloudRunProjectId: ${{ parameters.cloudRunProjectId }}
        imgRepository: ${{ parameters.imgRepository }}
        cloudRunRegion: ${{ parameters.cloudRunRegion }}
        cloudRunServiceName: ${{ parameters.cloudRunServiceName }}
        templateResourceName: ${{ parameters.templateResourceName }}
        pipelineResourceName: ${{ parameters.pipelineResourceName }}
        preDeploySteps: ${{ parameters.preDeploySteps }}
        postDeploySteps: ${{ parameters.postDeploySteps }}

# stages/dotnet-build-stage.yaml

parameters:
  # buildCode.yaml parameters
  - name: sourcePath
    type: string
    default: $(Build.SourcesDirectory)
  - name: slnOrCsprojName
    type: string
    default: Pipeline.sln

  # buildImage.yaml parameters
  - name: artifactName
    type: string
    default: OutputFiles
  - name: unzip
    type: boolean
    default: false
  - name: zipFileName
    type: string
    default: ''
  - name: unzipToFolderPath
    type: string
    default: ''
  - name: imgRepository
    type: string
  - name: imgTags
    type: object
  - name: buildDockerfile
    type: string
    default: Build.Dockerfile
  - name: buildContext
    type: string
  - name: containerRegistry
    type: string
  - name: startFileName
    type: string
  - name: templateResourceName
    type: string
    default: templates

  #======= Extensibility ========
  - name: imagePreBuildSteps
    type: stepList
    default: []
  - name: imagePostBuildSteps
    type: stepList
    default: []
  
  - name: codePreBuildSteps
    type: stepList
    default: []
  - name: codePostBuildSteps
    type: stepList
    default: []

  - name: stageName
    type: string
    default: BuildSourceAndImage

stages:
- stage: ${{ parameters.stageName }}
  displayName: 編譯程式碼和建立Docker Image
  jobs:
    - template: ../jobs/buildCode.yaml
      parameters:
        sourcePath: ${{ parameters.sourcePath }}
        slnOrCsprojName: ${{ parameters.slnOrCsprojName }}
        codePreBuildSteps: ${{ parameters.codePreBuildSteps }}
        codePostBuildSteps: ${{ parameters.codePostBuildSteps }}
    - template: ../jobs/buildImage.yaml
      parameters:
        templateResourceName: ${{ parameters.templateResourceName }}
        dependsOn: BuildCode
        artifactName: ${{ parameters.artifactName }}
        unzip: ${{ parameters.unzip }}
        zipFileName: ${{ parameters.zipFileName }}
        unzipToFolderPath: ${{ parameters.unzipToFolderPath }}
        imgRepository: ${{ parameters.imgRepository }}
        imgTags: ${{ parameters.imgTags }}
        buildDockerfile: ${{ parameters.buildDockerfile }}
        buildContext: ${{ parameters.buildContext }}
        containerRegistry: ${{ parameters.containerRegistry }}
        buildArgs: "TARGET_FILE=${{ parameters.startFileName }}"
        imagePreBuildSteps: ${{ parameters.imagePreBuildSteps }}
        # - script: |
        #     echo "這是使用的範例1"
        #   displayName: 範例啦
        # - task: Bash@3
        #   displayName: 還是範例啦
        #   inputs:
        #     targetType: 'inline'
        #     script: |
        #       echo "這是使用的範例2"
        # - ${{ parameters.imagePreBuildSteps }}
        imagePostBuildSteps: ${{ parameters.imagePostBuildSteps }}