Deployments

Images

Each Sourcegraph service is provided as a Docker image. Every commit to main in sourcegraph/sourcegraph pushes updated Docker images for all of our services to Docker Hub as part of our CI pipeline (i.e. if CI is green, then Docker images have been pushed). Images are first built as “candidate” images that are pushed to GCR to with the tag format <commit-hash>_<build-number>_candidate. The pipeline then runs a series of tests and checks against the images. If all pipeline steps pass the images are “promoted” and pushed to DockerHub with the tag format <build-number>_<date>_<commit-hash>. These are used by Sourcegraph Cloud.

When a new semver release is cut the pipelines, will build a release image with the same tag as the latest release version as well. These are used by customer deployments.

For pushing custom images, refer to building Docker images for specific branches.

Renovate

Renovate is a tool for updating dependencies. deploy-sourcegraph-* repositories with Renovate configured check for updated Docker images about every hour. If it finds new Docker images then it opens and merges a PR (Sourcegraph.com example) to update the image tags in the deployment configuration. This is usually accompanied by a CI job that deploys the updated images to the appropriate deployment.

Renovate configurations are committed in their respective deploy-sourcegraph-* repositories as renovate.json5.

ArgoCD

ArgoCD is a continuous delivery tool for Kubernetes applications. Sourcegraph’s ArgoCD instance is available at argocd.sgdev.org.

ArgoCD currently handles deployments for k8s.sgdev.org.

Infrastructure

The cloud resources (including clusters, DNS configuration, etc.) on which are deployments run should be configured in the infrastructure repository, even though Kubernetes deployments are managed by various deploy-sourcegraph-* repositories. For information about how our infrastructure is organized, refer to Infrastructure.

Merging changes from deploy-sourcegraph

We have two Sourcegraph Kubernetes cluster installations that we manage ourselves:

• deploy-sourcegraph-cloud
• deploy-sourcegraph-dogfood-k8s-2

This section describes how to merge changes from deploy-sourcegraph (referred to as upstream) into deploy-sourcegraph-cloud. The deploy-sourcegraph-dogfood-k8s-2 configuration is automatically updated with the latest deploy-sourcegraph changes.

The process is similar to the process we recommend our customers use to merge changes from upstream. The differences in process originate from the dual purpose of these two installations: they are genuine installations used by us and outside users (in the case of dot-com) and in addition to that they are test installations for new changes in code and in deployment. For that reason they are not pinned down to a version and the docker images are automatically updated to the latest builds.

Note: What is said about deploy-sourcegraph-cloud also applies to deploy-sourcegraph-dogfood-k8s unless otherwise specified.

Merging upstream deploy-sourcegraph into deploy-sourcegraph-cloud

1. Clone this repository and cd into it.
2. If you have not already, git remote add upstream https://github.com/sourcegraph/deploy-sourcegraph
3. git checkout master && git pull upstream/master
   • master of this repository strictly tracks master of deploy-sourcegraph, so there should be no merge conflicts.
   • If there are any merge conflicts in this step, you may git checkout master && git rev-parse HEAD && git reset --hard upstream/master && git push -f origin master which should always be 100% safe to do.
4. git checkout release && git checkout -B merge_upstream to create a branch where you will perform the merge.
5. git merge upstream/master to merge deploy-sourcegraph@master into merge_upstream
   • This will give you conflicts which you should address manually:
     • On docker image tags conflicts (image:), choose the insiders tag to allow renovate to deploy new builds.
     • On script conflicts (create-new-cluster.sh, kubectl-apply-all.sh, etc.), look for comments like This is a custom script for dot-com that indicate you should choose the current state over incoming changes.
   • If new services have been added (these generally show up as created files in base/), make sure that:
     • namespace: prod is applied to all new resource metadata.
   • Use kubectl apply --dry-run --validate --recursive -f base/ to validate your work.
   • Before you commit, ensure the commit message indicates which files had conflicts for reviewers to look at.
     • Using the default merge commit message, you can copy or uncomment the lines following Conflicts.
6. Send a PR to this repository for merging merge_upstream into release.
7. Reviewers should review:
   • The conflicting files.
   • If there are any risks associated with merging that should be watched out for / addressed, such as documented manual migrations that will need to be performed as part of merging the PR.
8. If there are any manual migrations needed, coordinate with the distribution team and apply those first.
   • For example, new services that require elevated permissions might not be deployed by Buildkite - this must be done manually.
9. Once approved, squash merge your PR so it can be easily reverted if needed.
   • In general, it might be a good idea to avoid doing this at the end of a PDT workday - if something goes wrong, it is easier to get help if other people are around.
10. Watch CI, which will deploy the change automatically.
11. Check the deployment is healthy afterwards (kubectl get pods should show all healthy, searching should work).