Clustering provider in Orleans

(Jin Qing’s Column, Nov. 3, 2021)

When deployed to a cluster of nodes,
Orleans internally implements a protocol to manage it’s silos,
including discovery, failure and reconfigure,
which is called cluster membership management.

Orleans has clustering membership providers for: Azure, SQL server, Zookeeper.

Clustering provider is one of key aspects of silo configuration.

OrleansContrib/Orleans.Clustering.Kubernetes
is a clustering provider for running Orleans cluster on Kubernetes.

Tell silo to use Kubernetes as the Cluster Membership Provider:

var silo = new SiloBuilder()
        ...
        .UseKubeMembership()
        ...
        .Build();

UseKubeMembership() instantiates a KubeMembershipTable which implements
IMembershipTable.

    public interface IMembershipTable
    {
        Task InitializeMembershipTable(bool tryInitTableVersion);
        Task DeleteMembershipTableEntries(string clusterId);
        Task CleanupDefunctSiloEntries(DateTimeOffset beforeDate);
        Task ReadRow(SiloAddress key);
        Task ReadAll();
        Task InsertRow(MembershipEntry entry, TableVersion tableVersion);
        Task UpdateRow(MembershipEntry entry, string etag, TableVersion tableVersion);
        Task UpdateIAmAlive(MembershipEntry entry);
    }

KubeMembershipTable
access Kubernetes API server to read and write silo entry CRD.

• InitializeMembershipTable()
  • TryInitClusterVersion()
    • _kubeClient.GetNamespacedCustomObjectAsync
    • _kubeClient.CreateNamespacedCustomObjectAsync
• DeleteMembershipTableEntries(string clusterId)
  • _kubeClient.DeleteNamespacedCustomObjectAsync
• InsertRow(…)
  • _kubeClient.GetNamespacedCustomObjectAsync
  • _kubeClient.ReplaceNamespacedCustomObjectAsync
  • _kubeClient.CreateNamespacedCustomObjectAsync
• ReadAll()
  • GetClusterVersion()
    • _kubeClient.ListNamespacedCustomObjectAsync
  • GetSilos()
    • _kubeClient.ListNamespacedCustomObjectAsync
• ReadRow(SiloAddress key)
  • _kubeClient.GetNamespacedCustomObjectAsync
• UpdateIAmAlive(MembershipEntry entry)
  • _kubeClient.GetNamespacedCustomObjectAsync
  • _kubeClient.ReplaceNamespacedCustomObjectAsync
• UpdateRow(…)
  • _kubeClient.ReplaceNamespacedCustomObjectAsync
• CleanupDefunctSiloEntries(DateTimeOffset beforeDate)
  • _kubeClient.DeleteNamespacedCustomObjectAsync

The operators to NamespacedCustomObject are:

• Delete
• Get
• Create
• Replace
• List

Two CRDs, ClusterVersion and Silo are defined in files:

• ClusterVersionCRD.yaml
• SiloEntryCRD.yaml

Custom resource objects are stored in etcd of Kubernetes.