Upgrade HAMi
This guide only applies to upgrades from HAMi v2.10 to the new Chart. Following these instructions directly from other versions is not recommended. If you are running an earlier version, first follow the v2.10 upgrade guide to upgrade HAMi to v2.10, then continue with this guide.
Upgrade Impact
Upgrading HAMi replaces hami-scheduler and hami-device-plugin and regenerates the device configuration, but does not affect running GPU workloads, so there is no need to stop or migrate existing tasks for the upgrade. However, workloads using HAMi-core do not automatically switch to the new version. HAMi-core is injected when a Pod is created, and running processes do not reload the library. For workloads that need the new HAMi-core version, perform a rolling restart or recreate the Pods after the upgrade.
During the upgrade, we recommend pausing submission of new HAMi GPU workloads until hami-scheduler, hami-device-plugin, and the admission webhook have finished updating. Rolling updates may cause brief unavailability or configuration version mismatches, which may lead to scheduling failures or injection errors for new tasks.
Device plugins from other vendors are outside the scope of this guide. See Devices Supported by HAMi.
Before You Upgrade
Configure Variables
Set the variables needed for deployment. Adjust the following example to match your environment.
export namespace=kube-system
export release=hami
export hami_chart_dir="${HOME}/hami-upgrade/HAMi"
Check the HAMi Version
Before upgrading, confirm that the currently installed version is HAMi 2.10:
helm list -n "${namespace}" --filter "^${release}$"
Back Up the Current Configuration
Before upgrading, save the user configuration for the current release:
helm get values "${release}" -n "${namespace}" -o yaml > previous-values.yaml
Manual changes to the device ConfigMap are not recorded in Helm values. Before upgrading, also save its data.device-config.yaml to migrate those changes and check the configuration after a rollback.
The following command uses hami-scheduler-device as an example. If you changed the release name or configured naming overrides, replace it with the actual ConfigMap name:
kubectl get configmap hami-scheduler-device -n "${namespace}" \
--allow-missing-template-keys=false \
-o jsonpath='{.data.device-config\.yaml}' > previous-device-config.yaml
Confirm that previous-device-config.yaml contains the current device configuration and retain this backup. During migration, extract only the settings that need to be retained. Do not restore the entire backup as the new configuration.
Obtain the Chart
The Chart containing the migration changes has not yet been published to the Helm repository. Use HAMi commit bf61e91 or a later commit that includes this change to ensure the additional validation for old fields is available. The following commands use bf61e91 as an example. All subsequent validation and upgrade commands use the Chart in ${hami_chart_dir}:
git clone https://github.com/Project-HAMi/HAMi.git "${hami_chart_dir}"
git -C "${hami_chart_dir}" checkout bf61e91763b51105f821c573002baca4e0025329
The Chart at this commit is development code, not a published release. Its version and appVersion may still refer to the previous release. Pinning the Chart commit does not pin the latest image tag used below.
Migrate and Create Values
When upgrading from v2.10 to the latest Chart, all device-related fields move under devices.<vendor>. First create my-values.yaml from the backed-up values, migrate the fields, and validate rendering before performing the upgrade.
Create values.yaml
Create my-values.yaml from previous-values.yaml as a starting point for preparing the upgrade configuration:
cp previous-values.yaml my-values.yaml
This guide uses HAMi images tagged latest. The Chart defaults to the IfNotPresent pull policy, so changing only the tag will continue to use old images cached on the nodes. Set the tag in my-values.yaml and change the pull policy for the scheduler extender, HAMi device plugin, and monitor to Always:
global:
imageTag: latest
scheduler:
extender:
image:
pullPolicy: Always
devicePlugin:
image:
pullPolicy: Always
monitor:
image:
pullPolicy: Always
With Always, the container runtime queries the image registry to resolve the tag when a container starts. The latest tag remains mutable, so containers started at different times are not guaranteed to use the same build. Updating the tag in the registry does not automatically restart existing Pods.
If the old values explicitly set scheduler.extender.image.tag, devicePlugin.image.tag, or devicePlugin.monitor.image.tag, remove those overrides.
After copying the file, update the device fields as described in the next section. Do not use the entire old configuration directly for the upgrade. Include any other custom settings that you need to retain in my-values.yaml. Do not restore the entire old ConfigMap after the upgrade, as this would overwrite the new default configuration.
Migrate Device Fields
The standard resource names for Enflame, Kunlun, Vastai, and Biren are now automatically added to the scheduler extender from the corresponding device fields. For these four device types, customresources contains only additional resources and defaults to an empty list. Remove any standard resource entries copied from the old values.
Before migrating, review previous-values.yaml and any manually modified device configuration in previous-device-config.yaml. Use the defaults from the Chart version installed before the upgrade as the baseline. Include only settings that differ from those defaults and still need to be retained after the upgrade in the new values.
- For device fields in the old values, use the vendor-specific mappings below to migrate them to the new paths and remove the old paths.
- For configuration modified directly in the ConfigMap, find the corresponding Helm values fields before migrating. Do not copy the entire ConfigMap contents into values. The old fields in the tables below refer to old Helm values paths, not ConfigMap field paths.
- Remove device fields that match the old version's defaults from
my-values.yaml. There is no need to add them at the corresponding new paths; let the new Chart use its defaults.
Do not decide whether a value needs migration solely because it is 0, false, or an empty string; compare it with the old version's defaults. Remove the old fields listed below after migration. The Chart rejects old fields covered by its validation checks, but fields not covered may be silently ignored. Successful rendering alone does not confirm that migration is complete.
NVIDIA
This includes NVIDIA resource fields and NVIDIA Device Plugin configuration fields:
| v2.10 Field | Latest Field |
|---|---|
resourceName | devices.nvidia.resourceCountName |
resourceMem | devices.nvidia.resourceMemoryName |
resourceMemPercentage | devices.nvidia.resourceMemoryPercentageName |
resourceCores | devices.nvidia.resourceCoreName |
resourcePriority | devices.nvidia.resourcePriorityName |
devicePlugin.deviceSplitCount | devices.nvidia.deviceSplitCount |
devicePlugin.deviceMemoryScaling | devices.nvidia.deviceMemoryScaling |
devicePlugin.deviceCoreScaling | devices.nvidia.deviceCoreScaling |
devicePlugin.preConfiguredDeviceMemory | devices.nvidia.preConfiguredDeviceMemory |
devicePlugin.enableNumaTopology | devices.nvidia.enableNumaTopology |
devicePlugin.runtimeClassName | devices.nvidia.runtimeClassName |
devicePlugin.createRuntimeClass | devices.nvidia.createRuntimeClass |
Cambricon
| v2.10 Field | Latest Field |
|---|---|
mluResourceName | devices.cambricon.resourceCountName |
mluResourceMem | devices.cambricon.resourceMemoryName |
mluResourceCores | devices.cambricon.resourceCoreName |
Hygon
HAMi#2969 renamed DCU to HCU for Hygon devices across resource names, configuration, and annotations. We recommend upgrading the HCU (formerly DCU) device plugin to the latest official version of k8s-hcu-device-plugin provided by Hygon. For details about the changes and configuration, see Enable Hygon HCU Sharing.
| v2.10 Field | v2.10 Default Value | Latest Field | Latest Default Value |
|---|---|---|---|
dcuResourceName | hygon.com/dcunum | devices.hygon.resourceCountName | hygon.com/hcunum |
dcuResourceMem | hygon.com/dcumem | devices.hygon.resourceMemoryName | hygon.com/hcumem |
dcuResourceCores | hygon.com/dcucores | devices.hygon.resourceCoreName | hygon.com/hcucores |
After removing old fields that match the v2.10 defaults, the Chart uses the new hygon.com/hcu* resource names. During the upgrade, check the resource names actually exposed by the HCU device plugin and update resource requests in subsequently submitted workloads to match the Chart configuration. Helm rendering validation alone does not confirm that Hygon device migration is complete.
MetaX
| v2.10 Field | Latest Field |
|---|---|
metaxResourceName | devices.metax.resourceVCountName |
metaxResourceCore | devices.metax.resourceVCoreName |
metaxResourceMem | devices.metax.resourceVMemoryName |
metaxsGPUTopologyAware | devices.metax.sgpuTopologyAware |
In v2.10, metaxsGPUTopologyAware uses a string and defaults to "false". The new field, devices.metax.sgpuTopologyAware, uses a boolean and defaults to false. The Chart still accepts the strings "true" and "false", but unquoted boolean values are recommended when migrating. If the old value is the default "false", simply remove the old field. If it is "true", set the new field to true.
Enflame
| v2.10 Field | Latest Field |
|---|---|
enflameResourceNameDRSGCU | devices.enflame.resourceNameDRSGCU |
enflameResourceNameGCUMemory | devices.enflame.resourceNameGCUMemory |
enflameResourceNameGCUCore | devices.enflame.resourceNameGCUCore |
Kunlunxin
| v2.10 Field | Latest Field |
|---|---|
kunlunResourceName | devices.kunlun.resourceCountName |
kunlunResourceVCountName | devices.kunlun.resourceVCountName |
kunlunResourceVMemoryName | devices.kunlun.resourceVMemoryName |
Vastai
| v2.10 Field | Latest Field |
|---|---|
vastaiResourceName | devices.vastai.resourceCountName |
Biren
| v2.10 Field | Latest Field |
|---|---|
birenResourceName | devices.biren.resourceCountName |
Migration Example
Suppose the previous installation used override-values.yaml with the following contents:
resourceName: nvidia.com/gpu
devicePlugin:
deviceSplitCount: 20
disablecorelimit: "true"
Here, resourceName matches the v2.10 default and does not need to be retained. deviceSplitCount was changed from the default of 10 to 20 and needs to be migrated. disablecorelimit is a custom setting whose path has not changed, so retain it as is.
Also suppose that the following field in data.device-config.yaml in the hami-scheduler-device ConfigMap was changed from the default of 0 to 4096 (only the relevant snippet is shown):
nvidia:
defaultMemory: 4096
The v2.10 Chart template hardcoded defaultMemory to 0, so it could not be changed through a dedicated values field. HAMi#3152 exposed these settings as values fields, so this manual change can now be migrated to devices.nvidia.defaultMemory.
The resulting my-values.yaml is:
devices:
nvidia:
deviceSplitCount: 20
defaultMemory: 4096
devicePlugin:
disablecorelimit: "true"
This file retains only the settings that need overrides: migrated device fields, custom configuration extracted from the ConfigMap, and custom settings whose paths have not changed. Neither the old path for resourceName nor its corresponding new path needs to be included; the new Chart provides the default.
scheduler.overwriteEnv and other devicePlugin settings still use their original paths, including enabled, image settings, deviceListStrategy, migStrategy, disablecorelimit, and nodeConfiguration. Retain any custom values at their original paths.
If you previously used device-config.content or files/device-config.yaml in the Chart, they replace the entire generated device configuration rather than merging with devices.*. To switch to managing configuration through devices.*, first review and migrate the settings that need to be retained, then remove the existing full configuration override.
Validate the Migrated Configuration
Render the migrated configuration using the local Chart obtained during preparation:
helm template "${release}" "${hami_chart_dir}/charts/hami" \
--namespace "${namespace}" \
--values my-values.yaml
Perform the Upgrade
Render the release again using the migrated values and the Chart defaults:
helm upgrade "${release}" "${hami_chart_dir}/charts/hami" \
--namespace "${namespace}" \
--reset-values \
--values my-values.yaml
--reset-values discards the old values stored in the release. All overrides that need to be retained must be included in my-values.yaml or supplied through other --values files.
Do not use --reuse-values or --reset-then-reuse-values for this migration. These options carry removed fields from the old values into the new Chart, which may cause rendering to fail. They do not make old fields that are not covered by validation take effect again.
Post-Upgrade Verification
Check Workloads
kubectl get pods -n "${namespace}" \
-l "app.kubernetes.io/instance=${release}"
kubectl rollout status deployment -n "${namespace}" \
-l "app.kubernetes.io/instance=${release},app.kubernetes.io/component=hami-scheduler" \
--timeout=5m
kubectl rollout status daemonset -n "${namespace}" \
-l "app.kubernetes.io/instance=${release},app.kubernetes.io/component=hami-device-plugin" \
--timeout=5m
Pods should be in the Running state, and the Deployment and DaemonSet should have completed their rolling updates.
Test Device Allocation
Use a small Pod to verify GPU allocation:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: gpu-test-pod
namespace: default
spec:
containers:
- name: test
image: nvidia/cuda:12.2.0-runtime-ubuntu22.04
command: ["nvidia-smi"]
resources:
limits:
nvidia.com/gpu: 1
restartPolicy: Never
EOF
kubectl logs gpu-test-pod
kubectl delete pod gpu-test-pod
Then check node resources and HAMi annotations:
kubectl describe nodes | grep -A 5 "Allocated resources"
kubectl get nodes -o yaml | grep -A 10 "hami.io"
Rollback
First, view the release history:
helm history "${release}" -n "${namespace}"
Roll back to the previous version:
helm rollback "${release}" -n "${namespace}"
You can also specify a revision:
helm rollback "${release}" <revision-number> -n "${namespace}"
Helm rollback uses the manifests saved in the release history, which do not include records of direct changes to ConfigMaps in the cluster. If you manually modified the device configuration before upgrading, compare it with previous-device-config.yaml after rollback and restore the necessary fields. Do not overwrite the entire ConfigMap. Retain the migrated my-values.yaml and use its prepared configuration when upgrading again.
If workloads still behave abnormally after rollback, first restore the GPU workloads from before the maintenance window, then check the scheduler, device plugin, and node logs.
Getting Help
If you encounter issues during the upgrade:
- Check the troubleshooting guide
- Review scheduler and device plugin logs
- Check GitHub Issues
- Ask in the community discussions
