Skip to main content

Ilum Core

Ilum Core helm chart config.

TL;DR​

$ helm repo add ilum https://charts.ilum.cloud
$ helm install ilum-core ilum/ilum-core

Installing the Chart​

To install the chart with the release name ilum-core:

$ helm install ilum-core ilum/ilum-core

The command deploys ilum-core on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.

Uninstalling the Chart​

To uninstall/delete the ilum-core deployment:

$ helm delete ilum-core

The command removes all the Kubernetes components associated with the chart and deletes the release.

Parameters​

Common parameters​

NameDescriptionValue
nameOverrideString to partially override ilum-core.fullname template (will maintain the release name)""
replicaCountNumber of ilum-core replicas1
fullnameOverrideString to fully override ilum-core.fullname template""
nodeSelectorilum-core node selection constraint
tolerationsilum-core pods tolerations[]
affinityilum-core node affinity

ilum-core deployment parameters​

NameDescriptionValue
imageilum-core imageilum/core:6.2.0
pullPolicyilum-core image pull policyIfNotPresent
imagePullSecretsilum-core image pull secrets[]

ilum-core communication parameters​

NameDescriptionValue
communication.typeilum-core communication type with spark jobs, available options: grpc, kafkagrpc

ilum-core service parameters​

NameDescriptionValue
service.typeilum-core service typeClusterIP
service.portilum-core service port9888
service.nodePortilum-core service node port - required when type is LoadBalancer or NodePort""
service.clusterIPilum-core service cluster IP - required when type is ClusterIP""
service.loadBalancerIPilum-core service load balancer IP - required when type is LoadBalancer""
service.annotationsilum-core service annotations{}

ilum-core ingress parameters​

NameDescriptionValue
ingress.enabledilum-core ingress enabled flagfalse
ingress.versionilum-core ingress versionv1
ingress.classNameilum-core ingress class name""
ingress.hostilum-core ingress hosthost
ingress.tlsilum-core ingress tls configuration[]
ingress.tls[x].secretNameilum-core ingress secret name to apply for a single tls configuration entry
ingress.tls[x].hostsilum-core ingress hosts list to apply for a single tls configuration entry
ingress.annotationsilum-core ingress annotations in yaml formatnginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/proxy-body-size: "600m"
nginx.org/client-max-body-size: "600m"
ingress.pathilum-core ingress path/(.*)
ingress.pathTypeilum-core ingress pathTypePrefix

ilum-core livenessProbe/readinessProbe parameters​

NameDescriptionValue
readinessProbeilum-core readinessProbe configuration
readinessProbe:
  failureThreshold: 3
  httpGet:
    path: /api/dev/reactive/health
    port: http
    scheme: HTTP
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1
livenessProbeilum-core livenessProbe configuration
livenessProbe:
  failureThreshold: 3
  httpGet:
    path: /api/dev/reactive/health
    port: http
    scheme: HTTP
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 1

ilum-core mongo parameters​

NameDescriptionValue
mongo.instancesilum-core mongo instances to connect tomongo:27017
mongo.replicaSetNameilum-core mongo replica set namers0
mongo.statusProbe.enanbledilum-core mongo healthcheck flagtrue
mongo.statusProbe.imageilum-core mongo healthcheck imageilum/mongodb:6.0.5

ilum-core kafka parameters​

NameDescriptionValue
kafka.addressilum-core kafka address to connect tokafka:9092
kafka.requestSizeilum-core kafka max.request.size parameter for ilum jobs kafka producers20000000
kafka.maxPollRecordsilum-core kafka max.poll.records parameter for ilum jobs kafka consumers500
kafka.maxPollIntervalilum-core kafka max.poll.interval.ms parameter for ilum jobs kafka consumers60000
kafka.statusProbe.enanbledilum-core kafka healthcheck flagtrue
kafka.statusProbe.imageilum-core kafka healthcheck imagebitnami/kafka:3.4.1

ilum-core grpc service parameters​

NameDescriptionValue
grpc.service.typeilum-core grpc service typeClusterIP
grpc.service.portilum-core grpc service port9999
grpc.service.nodePortilum-core grpc service node port - required when type is LoadBalancer or NodePort""
grpc.service.clusterIPilum-core grpc service cluster IP - required when type is ClusterIP""
grpc.service.loadBalancerIPilum-core grpc service load balancer IP - required when type is LoadBalancer""
grpc.service.annotationsilum-core grpc service annotations{}

ilum-core grpc parameters for spark job​

NameDescriptionValue
grpc.job.hostilum-core grpc host for spark job to connect toilum-grpc
grpc.job.portilum-core grpc port for spark job to connect to9999

ilum-core kubernetes cluster initializer parameters​

NameDescriptionValue
kubernetes.initClusterOnStartupilum-core default kubernetes cluster initialization flagtrue
kubernetes.upgradeClusterOnStartupilum-core default kubernetes cluster upgrade from values in config map flagfalse
kubernetes.api.urlilum-core default kubernetes cluster api urlhttps://kubernetes.default.svc
kubernetes.container.imageilum-core default kubernetes cluster container imageilum/spark:3.5.2-delta
kubernetes.sparkNamespaceilum-core default kubernetes cluster namespace to store spark resources{{ .Release.Namespace }}
kubernetes.storage.typeilum-core default kubernetes cluster storage type, available options: s3, gcs, wasbs, hdfss3

Default cluster reconciliation​

The two flags above govern how the bundled default Spark cluster is reconciled from chart values. The default cluster is bootstrap-once. When kubernetes.initClusterOnStartup is true (the default), the cluster is created on first startup if it does not already exist. When a cluster named default is already present, its stored specification is re-synced from chart values only if kubernetes.upgradeClusterOnStartup is true. That flag defaults to false, so a routine helm upgrade followed by a pod rollout does not push changed kubernetes.* values onto an existing default cluster. This preserves edits made through the UI or the dev API, at the cost of chart values no longer being authoritative once the cluster exists.

When kubernetes.upgradeClusterOnStartup is enabled, the reconciliation is partial:

Re-synced from chart valuesNever chart-driven on an existing cluster
sparkImage, defaultApplicationConfig, environment variables, namespace, serviceAccountName, attached presetsscalaVersion, TLS certificates, credentials, cluster id and description

The fields in the right-hand column are always preserved from the stored cluster, even with the flag enabled. They can only be changed by editing the cluster through the UI or the dev API.

Metastore is reconciled independently

The bundled default metastore row (metastore.type and the corresponding metastore..config) is reconciled unconditionally on every startup, independent of kubernetes.upgradeClusterOnStartup. A hive to nessie switch therefore takes effect on a plain pod rollout.

To apply changed default-cluster values, either enable reconciliation:

ilum-core:
  kubernetes:
    upgradeClusterOnStartup: "true"

and run helm upgrade followed by kubectl rollout restart deploy/ilum-core, or delete the existing cluster and re-bootstrap it from current chart values.

After the default cluster is deleted from the Clusters list under Workloads, it can be rebuilt from chart values without restarting ilum-core: while no clusters remain, the Clusters list shows a Restore Default Cluster button that re-creates it from chart values. The button is offered only when no default cluster exists; it never overwrites a cluster that is still present.

Restarting ilum-core after the delete has the equivalent effect, since the startup initializer recreates the missing cluster:

kubectl -n ilum rollout restart deploy/ilum-core

Enabling kubernetes.upgradeClusterOnStartup re-applies chart values on every boot, which overwrites subsequent UI or dev-API edits to the default cluster, and still does not migrate scalaVersion, certificates, or credentials.

s3 kubernetes storage parameters​

NameDescriptionValue
kubernetes.s3.hostilum-core default kubernetes cluster S3 storage host to store spark resourcess3
kubernetes.s3.portilum-core default kubernetes cluster S3 storage port to store spark resources7000
kubernetes.s3.sparkBucketilum-core default kubernetes cluster S3 storage bucket to store spark resourcesilum-files
kubernetes.s3.dataBucketilum-core default kubernetes cluster S3 storage bucket to store ilum tablesilum-tables
kubernetes.s3.accessKeyilum-core default kubernetes cluster S3 storage access key to store spark resources""
kubernetes.s3.secretKeyilum-core default kubernetes cluster S3 storage secret key to store spark resources""

gcs kubernetes storage parameters​

NameDescriptionValue
kubernetes.gcs.clientEmaililum-core default kubernetes cluster GCS storage client email""
kubernetes.gcs.sparkBucketilum-core default kubernetes cluster GCS storage bucket to store spark resources"ilum-files"
kubernetes.gcs.dataBucketilum-core default kubernetes cluster GCS storage bucket to store ilum tables"ilum-tables"
kubernetes.gcs.privateKeyilum-core default kubernetes cluster GCS storage private key to store spark resources""
kubernetes.gcs.privateKeyIdilum-core default kubernetes cluster GCS storage private key id to store spark resources""

wasbs kubernetes storage parameters​

NameDescriptionValue
kubernetes.wasbs.accountNameilum-core default kubernetes cluster WASBS storage account name""
kubernetes.wasbs.accessKeyilum-core default kubernetes cluster WASBS storage access key to store spark resources""
kubernetes.wasbs.sparkContainerilum-core default kubernetes cluster WASBS storage container name to store spark resources"ilum-files"
kubernetes.wasbs.dataContainerilum-core default kubernetes cluster WASBS storage container name to store ilum tables"ilum-tables"

hdfs kubernetes storage parameters​

NameDescriptionValue
kubernetes.hdfs.hadoopUsernameilum-core default kubernetes cluster HDFS storage hadoop username""
kubernetes.hdfs.configilum-core default kubernetes cluster HDFS storage dict of config files with name as key and base64 encoded content as value""
kubernetes.hdfs.sparkCatalogilum-core default kubernetes cluster HDFS storage catalog to store spark resources"ilum-files"
kubernetes.hdfs.dataCatalogilum-core default kubernetes cluster HDFS storage catalog to store ilum-tables"ilum-tables"
kubernetes.hdfs.keyTabilum-core default kubernetes cluster HDFS storage keytab file base64 encoded content""
kubernetes.hdfs.principalilum-core default kubernetes cluster HDFS storage principal name""
kubernetes.hdfs.krb5ilum-core default kubernetes cluster HDFS storage krb5 file base64 encoded content""
kubernetes.hdfs.trustStoreilum-core default kubernetes cluster HDFS storage trustStore file base64 encoded content""
kubernetes.hdfs.logDirectoryilum-core default kubernetes cluster HDFS storage directory path to store eventLog for history server""

Important! Make sure S3/GCS buckets or WASBS containers are already created and reachable! Important! Make sure HDFS logDirectory is absolute path of configured sparkCatalog with /ilum/logs suffix! Eg hdfs://name-node/user/username/spark-catalog/ilum/logs

ilum-core security parameters​

NameDescriptionValue
security.typeilum-core authentication type, available options: internal, ldap, oauth2internal
security.jwt.issuerUrlilum-core frontend URI used in the jwt iss claimhttps://ilum.cloud
security.jwt.timeToLiveilum-core jwt time to live in specified time units8h
security.jwt.publicKeyilum-core base64 encoded string containing the X.509 RSA 2048 bit public key""
security.jwt.privateKeyilum-core base64 encoded string containing the PKCS8 RSA 2048 bit private key""
security.authoritiesilum-core authorities mapping rules configuration. Used when authorization is ldap or oauth2. Allows to translate external auth provider groups and scopes to ilum roles
authorities:
  roles:
    prefix: ROLE_
    claim-name: groups
 scopes:
    prefix: SCOPE__
    claim-name: scp
security.authorities.roles.prefixilum-core authentication role prefixROLE_
security.authorities.roles.claimNameilum-core external authentication provider ID of the jwt claim which contains list of roles/groupsgroups
security.authorities.roles.mappingsilum-core role mapping definitions in form of a map external_role: ilum_role{}
security.authorities.scopes.prefixilum-core authentication scope prefixSCOPE_
security.authorities.scopes.claimNameilum-core external authentication provider ID of the jwt claim which contains list of scopesscp
security.authorities.scopes.mappingsilum-core scope mapping definitions in form of a map external_scope: ilum_scope{}

Internal config-map based authentication parameters​

NameDescriptionValue
security.internal.usersilum-core internal users configuration
users:
  - username: "admin"
    password: "admin"
    roles:
      - "ADMIN"
security.internal.users[].usernameilum-core user username""
security.internal.users[].passwordilum-core user plain password""
security.internal.users[].rolesilum-core user roles, available options: ADMIN, USER, VIEWER[]

LDAP based authentication parameters​

NameDescriptionValue
security.ldap.urlsilum-core LDAP URLs of the server[]
security.ldap.baseilum-core LDAP base suffix from which all operations should originate""
security.ldap.usernameilum-core LDAP login username of the server""
security.ldap.passwordilum-core LDAP login password of the server""
security.ldap.passwordEncoderilum-core LDAP password encoder. LDAP server authenticates users (bind operations) if empty, available options:
adaptive - password encoder that delegates to another encoder based upon a prefixed identifier
bcrypt
md5
sha256		""
security.ldap.userSearch.baseilum-core LDAP base DN from which the search for an user should be performed""
security.ldap.userSearch.filterilum-core LDAP pattern to be used for the user search. 0 is the usernameuid={0}
security.ldap.userSearch.passwordAttrilum-core LDAP ID of the attribute which contains the password of a useruserPassword
security.ldap.groupSearch.baseilum-core LDAP base DN from which the search for group membership should be performed""
security.ldap.groupSearch.filterilum-core LDAP pattern to be used for the group search. 0 is the user's DN(member={0})
security.ldap.groupSearch.roleAttrilum-core LDAP ID of the attribute which contains the role name for a groupcn

OAuth2 based authentication parameters​

NameDescriptionValue
security.oauth2.clientIdilum-core oauth2 Client ID""
security.oauth2.clientSecretilum-core oauth2 Client SECRET""
security.oauth2.issuerUriilum-core oauth2 URI that can either be an OpenID Connect discovery endpoint or an OAuth 2.0 Authorization Server Metadata endpoint defined by RFC 8414.""

ilum-core license parameters​

NameDescriptionValue
license.privateKeyilum license key""

ilum-core external spark submit parameters​

NameDescriptionValue
externalSparkSubmit.enabledilum-core external spark-submit flagfalse
externalSparkSubmit.imageilum-core external spark-submit base imagesilum/spark-launcher:spark-3.5.1
externalSparkSubmit.resourcesilum-core external spark-submit pod kubernetes resources
limits:
  memory: "500Mi"
requests:
  memory: "300Mi"

ilum-core spark history server parameters​

NameDescriptionValue
historyServer.enabledilum-core spark history server flagtrue
historyServer.imageilum-core spark history server imageilum/spark-launcher:spark-3.5.1
historyServer.addressilum-core spark history server addresshttp://ilum-history-server:9666
historyServer.pullPolicyilum-core spark history server image pull policyIfNotPresent
historyServer.imagePullSecretsilum-core spark history server image pull secrets[]
historyServer.parametersilum-core spark history server custom spark parameters
spark.history.fs.cleaner.enabled: true
spark.history.fs.cleaner.interval: 1d
spark.history.fs.cleaner.maxAge: 7d
historyServer.resourcesilum-core spark history server pod resources
limits:
  memory: "500Mi"
requests:
  memory: "300Mi"
historyServer.service.typeilum-core spark history server service typeClusterIP
historyServer.service.portilum-core spark history server service port9666
historyServer.service.nodePortilum-core spark history server service nodePort""
historyServer.service.clusterIPilum-core spark history server service clusterIP""
historyServer.service.loadBalancerIPilum-core spark history server service loadbalancerIP""
historyServer.service.annotationsilum-core history server service annotations{}
historyServer.ingress.enabledilum-core spark history server ingress flagfalse
historyServer.ingress.versionilum-core spark history server ingress version"v1"
historyServer.ingress.classNameilum-core spark history server ingress className""
historyServer.ingress.hostilum-core spark history server ingress host"host"
historyServer.ingress.pathilum-core spark history server ingress path"/(.*)"
historyServer.ingress.pathTypeilum-core spark history server ingress pathTypePrefix
historyServer.ingress.annotationsilum-core spark history server annotationsnginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/proxy-body-size: "600m"
nginx.org/client-max-body-size: "600m"
historyServer.statusProbe.enabledilum-core spark history server ilum-core healthcheck flagtrue
historyServer.statusProbe.imageilum-core spark history server ilum-core healthcheck imagecurlimages/curl:8.5.0
historyServer.nodeSelectorilum-core spark history server nodeSelector
historyServer.tolerationsilum-core spark history server tolerations[]
historyServer.affinityilum-core spark history server affinity

ilum-core job parameters​

NameDescriptionValue
job.retain.hoursilum-core spark jobs retention hours limit168
job.prometheus.enabledilum-core spark jobs prometheus enabled flagtrue
job.healthcheck.enabledilum-core spark jobs healthcheck enabled flagtrue
job.healthcheck.intervalilum-core spark jobs healthcheck interval in seconds300
job.healthcheck.toleranceilum-core spark jobs healthcheck response time tolerance in seconds120
job.memorysettings.executorsilum-core spark jobs executor count2
job.memorysettings.executorMemoryilum-core spark jobs executor memory allocation1g
job.memorysettings.driverMemoryilum-core spark jobs driver memory allocation1g
job.memorysettings.executorCoresilum-core spark jobs executor core count1
job.memorysettings.driverCoresilum-core spark jobs driver core count1
job.memorysettings.dynamicAllocationEnabledilum-core spark jobs dynamic allocation enabled flagfalse
job.memorysettings.minExecutorsilum-core spark jobs minimum number of executors0
job.memorysettings.initialExecutorsilum-core spark jobs initial number of executors0
job.memorysettings.maxExecutorsilum-core spark jobs maximum number of executors20

ilum-core default metastore parameters​

NameDescriptionValue
metastore.enabledilum-core default hive metastore enabled flagfalse
metastore.hive.addressilum-core default hive metastore addressthrift://ilum-hive-metastore:9083
metastore.hive.versionilum-core default hive metastore version3.1.3

ilum-core auto pausing settings​

NameDescriptionValue
job.autoPause.enabledilum-core group auto pausing flagtrue
job.autoPause.idleTimeilum-core auto pausing minimum group idle time in seconds3600
job.autoPause.periodilum-core time between checks of group idleness in seconds180

Generating RSA Private and Public Key​

In order to create 2048-bit RSA keys in an unencrypted Base64 PEM PKCS#8 format for authentication configuration, openssl was used.

Generate private key​

openssl genpkey -algorithm RSA \
    -pkeyopt rsa_keygen_bits:2048 \
    -pkeyopt rsa_keygen_pubexp:65537 | \
  openssl pkcs8 -topk8 -nocrypt -outform pem > private-key.p8

The contents of the private key should look like the following:

-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCsRnE83rm6BJya
nTyzVqX0SG+D4zBjkyWsOmGG+CoDdgQ6Z8AaocmnjP1SbRykQsQSMf6SeW+fdpH+
ccmzuHe7pZIa2o2Mg8xbk/UszJDaPztwoQbUt/2gHi/rZP8cIVkquzhnN/yxrMls
...
-----END PRIVATE KEY-----

In order to use private key as the setting security.jwt.privateKey, remove header and footer from the key.

Extract public key​

openssl pkey -pubout -inform pem -outform pem -in private-key.p8 -out public-key.spki

The contents of the public key should look like the following:

-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArEZxPN65ugScmp08s1al
9Ehvg+MwY5MlrDphhvgqA3YEOmfAGqHJp4z9Um0cpELEEjH+knlvn3aR/nHJs7h3
u6WSGtqNjIPMW5P1LMyQ2j87cKEG1Lf9oB4v62T/HCFZKrs4Zzf8sazJbMN3E/mJ
...
-----END PUBLIC KEY-----

In order to use public key as the setting security.jwt.publicKey, remove header and footer from the key.

Last updated on