Container image scan & code scan with BlackDuck

By | March 6, 2024
0 Comment

Black Duck Software, şirketlerin açık kaynak kodlu yazılım bileşenlerinin kullanımını ve uyumluluğunu yönetmelerine yardımcı olan bir yazılım ve hizmet sağlayıcısıdır. Black Duck’un en popüler ürünü olan “Black Duck Hub”, açık kaynak kodlu bileşenlerin tarama ve yönetimini kolaylaştıran bir araçtır. Bu araç, yazılım geliştiricilerinin yazılım projelerinde kullandıkları açık kaynak kodlu bileşenleri izlemelerine, güvenlik açıkları ve lisans uyumluluğu gibi önemli faktörleri değerlendirmelerine olanak tanır.

Bu makalede yazdığım ve çok yoğun şekilde kullandığım hem image repository üzerindeki hem de code seviyesinde scan yapmanızı sağlayan scriptimi sizinle paylaşmak istiyorum. Burada iki adet script bulunuyor. Ana script ile detect scriptini belli fonsiyonlar ile çağırıp scan işlemlerini yapacağız. Çağıracağımız script detect8 isimli bir script olacak. Synopsys detect’e ait Blackduck ile uyum içinde çalışan bu bileşenin black duck ile ilişkisini biraz daha açalım.

Synopsys Detect ve Black Duck arasındaki ilişki, Synopsys Detect’in Black Duck’un bileşen veritabanına ve analiz teknolojisine dayanarak açık kaynak bileşenlerin tarama ve yönetimini gerçekleştirmesidir. Bu entegrasyon sayesinde, geliştiriciler açık kaynak kodlu bileşenlerin güvenlik açıkları, lisans uyumluluğu ve diğer risklerini tespit edebilir ve bu bilgileri geliştirme sürecinde kullanabilirler.

Kısacası, Synopsys Detect, Black Duck’un sağladığı veritabanı ve analiz teknolojisi üzerine inşa edilmiş bir araçtır ve açık kaynak bileşenlerin güvenlik ve uyumluluk yönetimini kolaylaştırmak için kullanılır.

Şimdi ana scriptimiz ile aynı dizinde bulunan detect8.sh isimli scriptimizi oluşturalım.

#!/bin/bash
# downloaded on 2023-04-25 from:
# https://detect.synopsys.com/detect8.sh

echo "Detect Shell Script ${SCRIPT_VERSION}"

get_path_separator() {
  # Performs a check to see if the system is Windows based.
  if [[ `uname` == *"NT"* ]] || [[ `uname` == *"UWIN"* ]]; then
    echo "\\"
  else
    echo "/"
  fi
}

# DETECT_LATEST_RELEASE_VERSION should be set in your
# environment if you wish to use a version different
# from LATEST.
DETECT_RELEASE_VERSION=${DETECT_LATEST_RELEASE_VERSION}

# To override the default version key, specify a
# different DETECT_VERSION_KEY in your environment and
# *that* key will be used to get the download url from
# artifactory. These DETECT_VERSION_KEY values are
# properties in Artifactory that resolve to download
# urls for the detect jar file. As of 2022-12-07, the
# available DETECT_VERSION_KEY values are:
#
# Every new major version of detect will have its own
# DETECT_LATEST_X key.
DETECT_VERSION_KEY=${DETECT_VERSION_KEY:-DETECT_LATEST_8}

# You can specify your own download url from
# artifactory which can bypass using the property keys
# (this is mainly for QA purposes only)
DETECT_SOURCE=${DETECT_SOURCE:-}

# To override the default location of $HOME/synopsys-detect, specify
# your own DETECT_JAR_DOWNLOAD_DIR in your environment and
# *that* location will be used.
# *NOTE* We currently do not support spaces in the
# DETECT_JAR_DOWNLOAD_DIR.
DEFAULT_DETECT_JAR_DOWNLOAD_DIR="${HOME}$(get_path_separator)synopsys-detect$(get_path_separator)download"
if [[ -z "${DETECT_JAR_DOWNLOAD_DIR}" ]]; then
	# If new name not set: Try old name for backward compatibility
    DETECT_JAR_DOWNLOAD_DIR=${DETECT_JAR_PATH:-${DEFAULT_DETECT_JAR_DOWNLOAD_DIR}}
fi
DETECT_JAR_DOWNLOAD_DIR=${DETECT_JAR_DOWNLOAD_DIR:-${DEFAULT_DETECT_JAR_DOWNLOAD_DIR}}

# To control which java detect will use to run, specify
# the path in in DETECT_JAVA_PATH or JAVA_HOME in your
# environment, or ensure that java is first on the path.
# DETECT_JAVA_PATH will take precedence over JAVA_HOME.
# JAVA_HOME will take precedence over the path.
# Note: DETECT_JAVA_PATH should point directly to the
# java executable. For JAVA_HOME the java executable is
# expected to be in JAVA_HOME/bin/java
DETECT_JAVA_PATH=${DETECT_JAVA_PATH:-}

# If you want to pass any java options to the
# invocation, specify DETECT_JAVA_OPTS in your
# environment. For example, to specify a 6 gigabyte
# heap size, you would set DETECT_JAVA_OPTS=-Xmx6G.
DETECT_JAVA_OPTS=${DETECT_JAVA_OPTS:-}

# If you want to pass any additional options to
# curl, specify DETECT_CURL_OPTS in your environment.
# For example, to specify a proxy, you would set
# DETECT_CURL_OPTS=--proxy http://myproxy:3128
DETECT_CURL_OPTS=${DETECT_CURL_OPTS:-}

# If you only want to download the appropriate jar file set
# this to 1 in your environment. This can be useful if you
# want to invoke the jar yourself but do not want to also
# get and update the jar file when a new version releases.
DETECT_DOWNLOAD_ONLY=${DETECT_DOWNLOAD_ONLY:-0}

SCRIPT_ARGS=""
for NEXT_ARG in "$@"
do
    SCRIPT_ARGS+="\"${NEXT_ARG}\" "
done

LOGGABLE_SCRIPT_ARGS=""

# This provides a way to get the script version (via, say, grep/sed). Do not change.
SCRIPT_VERSION=3.0.1

echo "Detect Shell Script ${SCRIPT_VERSION}"

DETECT_BINARY_REPO_URL=https://sig-repo.synopsys.com

for i in $*; do
  if [[ $i == --blackduck.hub.password=* ]]; then
    LOGGABLE_SCRIPT_ARGS="$LOGGABLE_SCRIPT_ARGS --blackduck.hub.password=<redacted>"
  elif [[ $i == --blackduck.hub.proxy.password=* ]]; then
    LOGGABLE_SCRIPT_ARGS="$LOGGABLE_SCRIPT_ARGS --blackduck.hub.proxy.password=<redacted>"
  elif [[ $i == --blackduck.hub.api.token=* ]]; then
    LOGGABLE_SCRIPT_ARGS="$LOGGABLE_SCRIPT_ARGS --blackduck.hub.api.token=<redacted>"
  elif [[ $i == --blackduck.password=* ]]; then
    LOGGABLE_SCRIPT_ARGS="$LOGGABLE_SCRIPT_ARGS --blackduck.password=<redacted>"
  elif [[ $i == --blackduck.proxy.password=* ]]; then
    LOGGABLE_SCRIPT_ARGS="$LOGGABLE_SCRIPT_ARGS --blackduck.proxy.password=<redacted>"
  elif [[ $i == --blackduck.api.token=* ]]; then
    LOGGABLE_SCRIPT_ARGS="$LOGGABLE_SCRIPT_ARGS --blackduck.api.token=<redacted>"
  elif [[ $i == --polaris.access.token=* ]]; then
    LOGGABLE_SCRIPT_ARGS="$LOGGABLE_SCRIPT_ARGS --polaris.access.token=<redacted>"
  else
    LOGGABLE_SCRIPT_ARGS="$LOGGABLE_SCRIPT_ARGS $i"
  fi
done

run() {
  get_detect
  if [[ ${DETECT_DOWNLOAD_ONLY} -eq 0 ]]; then
    run_detect
  fi
}

get_detect() {
  PATH_SEPARATOR=$(get_path_separator)
  USE_LOCAL=0
  LOCAL_FILE="${DETECT_JAR_DOWNLOAD_DIR}${PATH_SEPARATOR}synopsys-detect-last-downloaded-jar.txt"
  if [[ -z "${DETECT_SOURCE}" ]]; then
    if [[ -z "${DETECT_RELEASE_VERSION}" ]]; then
      VERSION_CURL_CMD="curl ${DETECT_CURL_OPTS} --silent --header \"X-Result-Detail: info\" '${DETECT_BINARY_REPO_URL}/api/storage/bds-integrations-release/com/synopsys/integration/synopsys-detect?properties=${DETECT_VERSION_KEY}'"
      VERSION_EXTRACT_CMD="${VERSION_CURL_CMD} | grep \"${DETECT_VERSION_KEY}\" | sed 's/[^[]*[^\"]*\"\([^\"]*\).*/\1/'"
      DETECT_SOURCE=$(eval ${VERSION_EXTRACT_CMD})
      if [[ -z "${DETECT_SOURCE}" ]]; then
        echo "Unable to derive the location of ${DETECT_VERSION_KEY} from response to: ${VERSION_CURL_CMD}"
        USE_LOCAL=1
      fi
    else
      DETECT_SOURCE="${DETECT_BINARY_REPO_URL}/bds-integrations-release/com/synopsys/integration/synopsys-detect/${DETECT_RELEASE_VERSION}/synopsys-detect-${DETECT_RELEASE_VERSION}.jar"
    fi
  fi

  if [[ USE_LOCAL -eq 0 ]]; then
    echo "Will look for : ${DETECT_SOURCE}"
  else
    echo "Will look for : ${LOCAL_FILE}"
  fi

  if [[ USE_LOCAL -eq 1 ]] && [[ -f "${LOCAL_FILE}" ]]; then
    echo "Found local file ${LOCAL_FILE}"
    DETECT_FILENAME=`cat ${LOCAL_FILE}`
  elif [[ USE_LOCAL -eq 1 ]]; then
    echo "${LOCAL_FILE} is missing and unable to communicate with a Detect source."
    exit -1
  else
    DETECT_FILENAME=${DETECT_FILENAME:-$(awk -F "/" '{print $NF}' <<< $DETECT_SOURCE)}
  fi
  DETECT_DESTINATION="${DETECT_JAR_DOWNLOAD_DIR}${PATH_SEPARATOR}${DETECT_FILENAME}"

  USE_REMOTE=1
  if [[ USE_LOCAL -ne 1 ]] && [[ ! -f "${DETECT_DESTINATION}" ]]; then
    echo "You don't have the current file, so it will be downloaded."
  else
    echo "You have already downloaded the latest file, so the local file will be used."
    USE_REMOTE=0
  fi

  if [ ${USE_REMOTE} -eq 1 ]; then
    echo "getting ${DETECT_SOURCE} from remote"
    TEMP_DETECT_DESTINATION="${DETECT_DESTINATION}-temp"
    curlReturn=$(curl ${DETECT_CURL_OPTS} --silent -w "%{http_code}" -L -o "${TEMP_DETECT_DESTINATION}" --create-dirs "${DETECT_SOURCE}")
    if [[ 200 -eq ${curlReturn} ]]; then
      mv "${TEMP_DETECT_DESTINATION}" "${DETECT_DESTINATION}"
      if [[ -f ${LOCAL_FILE} ]]; then
        rm "${LOCAL_FILE}"
      fi
      echo "${DETECT_FILENAME}" >> "${LOCAL_FILE}"
      echo "saved ${DETECT_SOURCE} to ${DETECT_DESTINATION}"
    else
      echo "The curl response was ${curlReturn}, which is not successful - please check your configuration and environment."
      exit -1
    fi
  fi
}

set_detect_java_path() {
  PATH_SEPARATOR=$(get_path_separator)

  if [[ -n "${DETECT_JAVA_PATH}" ]]; then
    echo "Java Source: DETECT_JAVA_PATH=${DETECT_JAVA_PATH}"
  elif [[ -n "${JAVA_HOME}" ]]; then
    DETECT_JAVA_PATH="${JAVA_HOME}${PATH_SEPARATOR}bin${PATH_SEPARATOR}java"
    echo "Java Source: JAVA_HOME${PATH_SEPARATOR}bin${PATH_SEPARATOR}java=${DETECT_JAVA_PATH}"
  else
    echo "Java Source: PATH"
    DETECT_JAVA_PATH="java"
  fi
}

run_detect() {
  set_detect_java_path

  JAVACMD="\"${DETECT_JAVA_PATH}\" ${DETECT_JAVA_OPTS} -jar \"${DETECT_DESTINATION}\""
  echo "running Detect: ${JAVACMD} ${LOGGABLE_SCRIPT_ARGS}"

  eval "${JAVACMD} ${SCRIPT_ARGS}"
  RESULT=$?
  echo "Result code of ${RESULT}, exiting"
  exit ${RESULT}
}

run

Detect8.sh isimli scriptimizi oluştutup kaydettikten sonra bunu çağıracağımız fonksiyonları içeren main scriptimiz scan_microservice.sh’ı oluşturalım.

#!/bin/bash

function prepare_environment() {
  if [[ -z "${BLACK_DUCK_TOKEN}" ]]; then
    echo "black duck api token was not provided, aborting"
    exit 1
  fi
  if [[ -z "${GITHUB_USER}" ]]; then
    echo "github user was not provided, aborting"
    exit 1
  fi
  if [[ -z "${GITHUB_PERSONAL_ACCESS_TOKEN}" ]]; then
    echo "github personal access token was not provided, aborting"
    exit 1
  fi
  
  Green='\033[0;32m'
  NC='\033[0m'
  
  readonly BLACK_DUCK_PROJECT_NAME="<black-duck-project-name>"
  readonly BLACK_DUCK_TIMEOUT="300"
  readonly BLACK_DUCK_URL="<black-duck-url>"
  readonly DOCKER_REGISTRY_URL="<registry-url>"
}

function execute_docker_image_scan() {
  declare -a DOCKER_IMAGES_MICROSERVICES=(
    "docker-image-1:latest"
    "docker-image-2:latest"
  )
  for DOCKER_IMAGE in "${DOCKER_IMAGES_MICROSERVICES[@]}"; do
    echo -e "${Green}****************** BlackDuck scan has been started for docker image: '${DOCKER_IMAGE}' ******************${NC}"
    DOCKER_IMAGE_VERSION=$(cut -d ":" -f2 <<<"${DOCKER_IMAGE}")
    bash ./detect8.sh \
      --blackduck.url="${BLACK_DUCK_URL}" \
      --blackduck.api.token="${BLACK_DUCK_TOKEN}" \
      --detect.project.name="${BLACK_DUCK_PROJECT_NAME}" \
      --detect.project.version.name="${DOCKER_IMAGE_VERSION}" \
      --detect.timeout="${BLACK_DUCK_TIMEOUT}" \
      --detect.tools=DOCKER \
      --detect.docker.image="${DOCKER_REGISTRY_URL}/${DOCKER_IMAGE}" \
      --detect.detector.search.continue=true
    echo -e "${Green}****************** BlackDuck scan has been finished for docker image: '${DOCKER_IMAGE}' ******************${NC}"
    echo ""
  done
}

function execute_code_scan() {
  declare -a MICROSERVICES_WITH_VERSIONS=(
    "docker-image-1:latest"
    "docker-image-2:latest"
  )
  readonly GITHUB_URL="<write github url here with token>"

  for MICROSERVICE_WITH_VERSION in "${MICROSERVICES_WITH_VERSIONS[@]}"; do
    MICROSERVICE_NAME=$(cut -d ":" -f1 <<<"${MICROSERVICE_WITH_VERSION}")
    MICROSERVICE_VERSION=$(cut -d ":" -f2 <<<"${MICROSERVICE_WITH_VERSION}")
    GIT_REPO_DIR="repo-to-scan"
    
    echo -e "${Green}****************** BlackDuck scan has been started for code of service: '${MICROSERVICE_NAME}' ******************${NC}"
    git clone "${GITHUB_URL}/${MICROSERVICE_NAME}.git" "${GIT_REPO_DIR}"
    bash ./detect8.sh \
      --blackduck.url="${BLACK_DUCK_URL}" \
      --blackduck.api.token="${BLACK_DUCK_TOKEN}" \
      --detect.project.name="${BLACK_DUCK_PROJECT_NAME}" \
      --detect.project.version.name="${MICROSERVICE_VERSION}" \
      --detect.timeout="${BLACK_DUCK_TIMEOUT}" \
      --detect.tools=DETECTOR \
      --detect.source.path="${GIT_REPO_DIR}" \
      --detect.code.location.name="${GIT_REPO_DIR}" \
      --detect.detector.search.continue=true \
      --detect.maven.build.command='--settings=actions-settings.xml'

    rm -rf "${GIT_REPO_DIR}"
    echo -e "${Green}****************** BlackDuck scan has been finished for code of service: '${MICROSERVICE_NAME}' ******************${NC}"
    echo ""
  done
}

while getopts "p:t:u:" arg; do
  case "${arg}" in
  p)
    GITHUB_PERSONAL_ACCESS_TOKEN="${OPTARG}"
    ;;
  t)
    BLACK_DUCK_TOKEN="${OPTARG}"
    ;;
  u)
    GITHUB_USER="${OPTARG}"
    ;;
  *)
    print_usage
    exit 1
    ;;
  esac
done
shift $((OPTIND - 1))

prepare_environment
execute_docker_image_scan
execute_code_scan

Burada dikkat edilecek nokta şudur. Script için gerekli değişiklikleri kendi ortamınıza ilişkin bilgiler ile değiştirilmelidir.

readonly BLACK_DUCK_PROJECT_NAME="<black-duck-project-name>"
readonly BLACK_DUCK_TIMEOUT="300"
readonly BLACK_DUCK_URL="<black-duck-url>"
readonly DOCKER_REGISTRY_URL="<registry-url>"

Ek olarak kodun bulunduğu github url’i de token’lı olacak şekilde değişkene değer olarak verilmelidir.

readonly GITHUB_URL="<write github url here with token>"

Ardından script tetiklendiğinde aşağıdaki 3 fonksiyon tetiklenerek repository ve code base tarafında black duck tarafından scan işlemleri gerçekleştirilerek bir rapor oluşturulacaktır.

Herkes için oldukça faydalı olacak bir güvenlik tool’un olan Blackduck için bu scripti pipeline’larınıza da kolaylıkla entegre edebilirsiniz.

Script düzgün şekilde çalıştırıldığında çıktı aşağıdaki gibi olacaktır.

Cheers.

Leave a Reply

Your email address will not be published. Required fields are marked *