diff --git a/docs/operator/CHANGELOG.md b/docs/operator/CHANGELOG.md index d931e2ad8..0da1cd42b 100644 --- a/docs/operator/CHANGELOG.md +++ b/docs/operator/CHANGELOG.md @@ -11,6 +11,15 @@ aliases: - /operator/changelog/index.html --- +## tip + +- [operator](https://docs.victoriametrics.com/operator/): properly apply `useStrictSecurity: true` to the `initContainers` for `VMAuth`, `VMAgent` and `VMAlertmanager`. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1134) for details. +- [vmauth](https://docs.victoriametrics.com/operator/resources/vmauth): Moved `spec.configSecret` to `spec.externalConfig.secretRef.name` and added `spec.externalConfig.localPath` to be able to provide custom configs via sidecar. +- [vmcluster](https://docs.victoriametrics.com/operator/resources/vmcluster): adds `requestsLoadBalancer` configuration to the `VMCluster.spec`. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1130) for details. +- [vmcluster](https://docs.victoriametrics.com/operator/resources/vmcluster): properly configure monitoring for `VMCluster` with enabled `backup`. +- [vmalertmanager](https://docs.victoriametrics.com/operator/resources/vmalertmanager): properly trigger reload when `ConfigMap` provided via `.spec.configMap` are changed. + + ## [v0.48.4](https://github.com/VictoriaMetrics/operator/releases/tag/v0.48.4) - 15 Oct 2024 - [api](https://docs.victoriametrics.com/operator/api): adds new fields `maxDiskUsagePerUrl` and`forceVMProto` to the `VMagent` `remoteWriteSpec` diff --git a/docs/operator/api.md b/docs/operator/api.md index 206db8363..01997ea17 100644 --- a/docs/operator/api.md +++ b/docs/operator/api.md @@ -78,6 +78,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMInsert](#vminsert) - [VMSelect](#vmselect) @@ -352,6 +353,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMInsert](#vminsert) - [VMSelect](#vmselect) @@ -426,6 +428,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMInsert](#vminsert) - [VMSelect](#vmselect) @@ -727,6 +730,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMInsert](#vminsert) - [VMSelect](#vmselect) @@ -770,6 +774,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMInsert](#vminsert) - [VMSelect](#vmselect) @@ -797,6 +802,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMInsert](#vminsert) - [VMSelect](#vmselect) @@ -931,6 +937,23 @@ _Appears in:_ | `vm_scrape_params` | VMScrapeParams defines VictoriaMetrics specific scrape parameters | _[VMScrapeParams](#vmscrapeparams)_ | false | +#### ExternalConfig + + + +ExternalConfig defines external source of configuration + + + +_Appears in:_ +- [VMAuthSpec](#vmauthspec) + +| Field | Description | Scheme | Required | +| --- | --- | --- | --- | +| `localPath` | LocalPath contains static path to a config, which is managed externally for cases
when using secrets is not applicable, e.g.: Vault sidecar. | _string_ | false | +| `secretRef` | SecretRef defines selector for externally managed secret which contains configuration | _[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#secretkeyselector-v1-core)_ | false | + + #### FileSDConfig @@ -1068,6 +1091,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMBackup](#vmbackup) - [VMInsert](#vminsert) @@ -1744,6 +1768,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMInsert](#vminsert) - [VMSelect](#vmselect) @@ -3003,6 +3028,82 @@ VMAuth is the Schema for the vmauths API | `spec` | | _[VMAuthSpec](#vmauthspec)_ | true | +#### VMAuthLoadBalancer + + + +VMAuthLoadBalancer configures vmauth as a load balancer +for the requests + + + +_Appears in:_ +- [VMClusterSpec](#vmclusterspec) + +| Field | Description | Scheme | Required | +| --- | --- | --- | --- | +| `disableInsertBalancing` | | _boolean_ | true | +| `disableSelectBalancing` | | _boolean_ | true | +| `enabled` | | _boolean_ | true | +| `spec` | | _[VMAuthLoadBalancerSpec](#vmauthloadbalancerspec)_ | true | + + +#### VMAuthLoadBalancerSpec + + + +VMAuthLoadBalancerSpec defines configuration spec for VMAuth used as load-balancer +for VMCluster component + + + +_Appears in:_ +- [VMAuthLoadBalancer](#vmauthloadbalancer) + +| Field | Description | Scheme | Required | +| --- | --- | --- | --- | +| `affinity` | Affinity If specified, the pod's scheduling constraints. | _[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#affinity-v1-core)_ | false | +| `configMaps` | ConfigMaps is a list of ConfigMaps in the same namespace as the Application
object, which shall be mounted into the Application container
at /etc/vm/configs/CONFIGMAP_NAME folder | _string array_ | false | +| `containers` | Containers property allows to inject additions sidecars or to patch existing containers.
It can be useful for proxies, backup, etc. | _[Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#container-v1-core) array_ | false | +| `disableSelfServiceScrape` | DisableSelfServiceScrape controls creation of VMServiceScrape by operator
for the application.
Has priority over `VM_DISABLESELFSERVICESCRAPECREATION` operator env variable | _boolean_ | false | +| `dnsConfig` | Specifies the DNS parameters of a pod.
Parameters specified here will be merged to the generated DNS
configuration based on DNSPolicy. | _[PodDNSConfig](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#poddnsconfig-v1-core)_ | false | +| `dnsPolicy` | DNSPolicy sets DNS policy for the pod | _[DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#dnspolicy-v1-core)_ | false | +| `extraArgs` | ExtraArgs that will be passed to the application container
for example remoteWrite.tmpDataPath: /tmp | _object (keys:string, values:string)_ | false | +| `extraEnvs` | ExtraEnvs that will be passed to the application container | _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#envvar-v1-core) array_ | false | +| `hostAliases` | HostAliases provides mapping for ip and hostname,
that would be propagated to pod,
cannot be used with HostNetwork. | _[HostAlias](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#hostalias-v1-core) array_ | false | +| `hostNetwork` | HostNetwork controls whether the pod may use the node network namespace | _boolean_ | false | +| `host_aliases` | HostAliasesUnderScore provides mapping for ip and hostname,
that would be propagated to pod,
cannot be used with HostNetwork.
Has Priority over hostAliases field | _[HostAlias](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#hostalias-v1-core) array_ | false | +| `image` | Image - docker image settings
if no specified operator uses default version from operator config | _[Image](#image)_ | false | +| `imagePullSecrets` | ImagePullSecrets An optional list of references to secrets in the same namespace
to use for pulling images from registries
see https://kubernetes.io/docs/concepts/containers/images/#referring-to-an-imagepullsecrets-on-a-pod | _[LocalObjectReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#localobjectreference-v1-core) array_ | false | +| `initContainers` | InitContainers allows adding initContainers to the pod definition.
Any errors during the execution of an initContainer will lead to a restart of the Pod.
More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ | _[Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#container-v1-core) array_ | false | +| `logFormat` | LogFormat for vmauth
default or json | _string_ | false | +| `logLevel` | LogLevel for vmauth container. | _string_ | false | +| `minReadySeconds` | MinReadySeconds defines a minim number os seconds to wait before starting update next pod
if previous in healthy state
Has no effect for VLogs and VMSingle | _integer_ | false | +| `nodeSelector` | NodeSelector Define which Nodes the Pods are scheduled on. | _object (keys:string, values:string)_ | false | +| `paused` | Paused If set to true all actions on the underlying managed objects are not
going to be performed, except for delete actions. | _boolean_ | false | +| `podDisruptionBudget` | PodDisruptionBudget created by operator | _[EmbeddedPodDisruptionBudgetSpec](#embeddedpoddisruptionbudgetspec)_ | false | +| `podMetadata` | Common params for scheduling
PodMetadata configures Labels and Annotations which are propagated to the vmauth lb pods. | _[EmbeddedObjectMetadata](#embeddedobjectmetadata)_ | true | +| `port` | Port listen address | _string_ | false | +| `priorityClassName` | PriorityClassName class assigned to the Pods | _string_ | false | +| `readinessGates` | ReadinessGates defines pod readiness gates | _[PodReadinessGate](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#podreadinessgate-v1-core) array_ | true | +| `replicaCount` | ReplicaCount is the expected size of the Application. | _integer_ | false | +| `resources` | Resources container resource request and limits, https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
if not defined default resources from operator config will be used | _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#resourcerequirements-v1-core)_ | false | +| `revisionHistoryLimitCount` | The number of old ReplicaSets to retain to allow rollback in deployment or
maximum number of revisions that will be maintained in the Deployment revision history.
Has no effect at StatefulSets
Defaults to 10. | _integer_ | false | +| `runtimeClassName` | RuntimeClassName - defines runtime class for kubernetes pod.
https://kubernetes.io/docs/concepts/containers/runtime-class/ | _string_ | false | +| `schedulerName` | SchedulerName - defines kubernetes scheduler name | _string_ | false | +| `secrets` | Secrets is a list of Secrets in the same namespace as the Application
object, which shall be mounted into the Application container
at /etc/vm/secrets/SECRET_NAME folder | _string array_ | false | +| `securityContext` | SecurityContext holds pod-level security attributes and common container settings.
This defaults to the default PodSecurityContext. | _[SecurityContext](#securitycontext)_ | false | +| `serviceScrapeSpec` | ServiceScrapeSpec that will be added to vmauthlb VMServiceScrape spec | _[VMServiceScrapeSpec](#vmservicescrapespec)_ | false | +| `serviceSpec` | AdditionalServiceSpec defines service override configuration for vmauth lb deployment
it'll be only applied to vmclusterlb- service | _[AdditionalServiceSpec](#additionalservicespec)_ | true | +| `terminationGracePeriodSeconds` | TerminationGracePeriodSeconds period for container graceful termination | _integer_ | false | +| `tolerations` | Tolerations If specified, the pod's tolerations. | _[Toleration](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#toleration-v1-core) array_ | false | +| `topologySpreadConstraints` | TopologySpreadConstraints embedded kubernetes pod configuration option,
controls how pods are spread across your cluster among failure-domains
such as regions, zones, nodes, and other user-defined topology domains
https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/ | _[TopologySpreadConstraint](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#topologyspreadconstraint-v1-core) array_ | false | +| `useDefaultResources` | UseDefaultResources controls resource settings
By default, operator sets built-in resource requirements | _boolean_ | false | +| `useStrictSecurity` | UseStrictSecurity enables strict security mode for component
it restricts disk writes access
uses non-root user out of the box
drops not needed security permissions | _boolean_ | false | +| `volumeMounts` | VolumeMounts allows configuration of additional VolumeMounts on the output Deployment/StatefulSet definition.
VolumeMounts specified will be appended to other VolumeMounts in the Application container | _[VolumeMount](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#volumemount-v1-core) array_ | false | +| `volumes` | Volumes allows configuration of additional volumes on the output Deployment/StatefulSet definition.
Volumes specified will be appended to other volumes that are generated.
/ +optional | _[Volume](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#volume-v1-core) array_ | true | + + #### VMAuthSpec @@ -3021,7 +3122,7 @@ _Appears in:_ | `configReloaderExtraArgs` | ConfigReloaderExtraArgs that will be passed to VMAuths config-reloader container
for example resyncInterval: "30s" | _object (keys:string, values:string)_ | false | | `configReloaderImageTag` | ConfigReloaderImageTag defines image:tag for config-reloader container | _string_ | false | | `configReloaderResources` | ConfigReloaderResources config-reloader container resource request and limits, https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
if not defined default resources from operator config will be used | _[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#resourcerequirements-v1-core)_ | false | -| `configSecret` | ConfigSecret is the name of a Kubernetes Secret in the same namespace as the
VMAuth object, which contains auth configuration for vmauth,
configuration must be inside secret key: config.yaml.
It must be created and managed manually.
If it's defined, configuration for vmauth becomes unmanaged and operator'll not create any related secrets/config-reloaders | _string_ | false | +| `configSecret` | ConfigSecret is the name of a Kubernetes Secret in the same namespace as the
VMAuth object, which contains auth configuration for vmauth,
configuration must be inside secret key: config.yaml.
It must be created and managed manually.
If it's defined, configuration for vmauth becomes unmanaged and operator'll not create any related secrets/config-reloaders
Deprecated, use externalConfig.secretRef instead | _string_ | true | | `containers` | Containers property allows to inject additions sidecars or to patch existing containers.
It can be useful for proxies, backup, etc. | _[Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#container-v1-core) array_ | false | | `default_url` | DefaultURLs backend url for non-matching paths filter
usually used for default backend with error message | _string array_ | true | | `disableSelfServiceScrape` | DisableSelfServiceScrape controls creation of VMServiceScrape by operator
for the application.
Has priority over `VM_DISABLESELFSERVICESCRAPECREATION` operator env variable | _boolean_ | false | @@ -3029,6 +3130,7 @@ _Appears in:_ | `dnsConfig` | Specifies the DNS parameters of a pod.
Parameters specified here will be merged to the generated DNS
configuration based on DNSPolicy. | _[PodDNSConfig](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#poddnsconfig-v1-core)_ | false | | `dnsPolicy` | DNSPolicy sets DNS policy for the pod | _[DNSPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#dnspolicy-v1-core)_ | false | | `drop_src_path_prefix_parts` | DropSrcPathPrefixParts is the number of `/`-delimited request path prefix parts to drop before proxying the request to backend.
See [here](https://docs.victoriametrics.com/vmauth#dropping-request-path-prefix) for more details. | _integer_ | false | +| `externalConfig` | ExternalConfig defines a source of external VMAuth configuration.
If it's defined, configuration for vmauth becomes unmanaged and operator'll not create any related secrets/config-reloaders | _[ExternalConfig](#externalconfig)_ | false | | `extraArgs` | ExtraArgs that will be passed to the application container
for example remoteWrite.tmpDataPath: /tmp | _object (keys:string, values:string)_ | false | | `extraEnvs` | ExtraEnvs that will be passed to the application container | _[EnvVar](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#envvar-v1-core) array_ | false | | `headers` | Headers represent additional http headers, that vmauth uses
in form of ["header_key: header_value"]
multiple values for header key:
["header_key: value1,value2"]
it's available since 1.68.0 version of vmauth | _string array_ | false | @@ -3157,6 +3259,7 @@ _Appears in:_ | `license` | License allows to configure license key to be used for enterprise features.
Using license key is supported starting from VictoriaMetrics v1.94.0.
See [here](https://docs.victoriametrics.com/enterprise) | _[License](#license)_ | false | | `paused` | Paused If set to true all actions on the underlying managed objects are not
going to be performed, except for delete actions. | _boolean_ | false | | `replicationFactor` | ReplicationFactor defines how many copies of data make among
distinct storage nodes | _integer_ | false | +| `requestsLoadBalancer` | RequestsLoadBalancer configures load-balancing for vminsert and vmselect requests
it helps to evenly spread load across pods
usually it's not possible with kubernetes TCP based service | _[VMAuthLoadBalancer](#vmauthloadbalancer)_ | true | | `retentionPeriod` | RetentionPeriod for the stored metrics
Note VictoriaMetrics has data/ and indexdb/ folders
metrics from data/ removed eventually as soon as partition leaves retention period
reverse index data at indexdb rotates once at the half of configured
[retention period](https://docs.victoriametrics.com/Single-server-VictoriaMetrics/#retention) | _string_ | true | | `serviceAccountName` | ServiceAccountName is the name of the ServiceAccount to use to run the
VMSelect, VMStorage and VMInsert Pods. | _string_ | false | | `useStrictSecurity` | UseStrictSecurity enables strict security mode for component
it restricts disk writes access
uses non-root user out of the box
drops not needed security permissions | _boolean_ | false | @@ -3704,6 +3807,7 @@ _Appears in:_ - [VMAgentSpec](#vmagentspec) - [VMAlertSpec](#vmalertspec) - [VMAlertmanagerSpec](#vmalertmanagerspec) +- [VMAuthLoadBalancerSpec](#vmauthloadbalancerspec) - [VMAuthSpec](#vmauthspec) - [VMInsert](#vminsert) - [VMSelect](#vmselect) diff --git a/docs/operator/resources/vmcluster.md b/docs/operator/resources/vmcluster.md index fe3c0bba6..09e0a1788 100644 --- a/docs/operator/resources/vmcluster.md +++ b/docs/operator/resources/vmcluster.md @@ -43,6 +43,44 @@ see [Extra arguments section](./#extra-arguments). Also, you can check out the [examples](#examples) section. +## Requests Load-Balancing + + Operator provides enhanced load-balancing mechanism for `vminsert` and `vmselect` clients. By default, operator uses built-in Kubernetes [service]() with `clusterIP` type for clients connection. It's good solution for short lived connections. But it acts poorly with long-lived TCP sessions and leads to the uneven resources utilisation for `vmselect` and `vminsert` components. + + Consider the following example: + +![CR](vmcluster_default_balancer.webp) + + In this case clients could establish multiple connections to the same `pod` via `service`. And client requests will be served only by subset of `pods`. + + Operator allows to tweak this behaviour with enabled `requestsLoadbalacing`: + +```yaml +apiVersion: operator.victoriametrics.com/v1beta1 +kind: VMCluster +metadata: + name: with-balanacer +spec: + retentionPeriod: "4" + replicationFactor: 1 + requestsLoadBalancer: + enabled: true + spec: + replicaCount: 2 +``` + + Operator will deploy `VMAuth` deployment with 2 replicas. And update vminsert and vmselect services to point to `vmauth`. + In addition, operator will create 3 additional services with the following pattern: + +- vminsertinternal-CLUSTER_NAME - needed for vmselect pod discovery +- vmselectinternal-CLUSTER_NAME - needed for vminsert pod discovery +- vmclusterlb-CLUSTER_NAME - needed for metrics collection and exposing `vmselect` and `vminsert` components via `VMAuth` balancer. + + Network scheme with load-balancing: + ![CR](vmcluster_with_balancer.webp) + + Operator allows to customise load-balancing configuration with `requestsLoadBalancer.Spec` settings. + ## High availability The cluster version provides a full set of high availability features - metrics replication, node failover, horizontal scaling. diff --git a/docs/operator/resources/vmcluster_default_balancer.webp b/docs/operator/resources/vmcluster_default_balancer.webp new file mode 100644 index 000000000..7b57688f0 Binary files /dev/null and b/docs/operator/resources/vmcluster_default_balancer.webp differ diff --git a/docs/operator/resources/vmcluster_with_balancer.webp b/docs/operator/resources/vmcluster_with_balancer.webp new file mode 100644 index 000000000..aa3b25311 Binary files /dev/null and b/docs/operator/resources/vmcluster_with_balancer.webp differ diff --git a/docs/operator/vars.md b/docs/operator/vars.md index 8a70c0703..0dc2488c0 100644 --- a/docs/operator/vars.md +++ b/docs/operator/vars.md @@ -136,4 +136,4 @@ aliases: | VM_PODWAITREADYINTERVALCHECK | 5s | false | Defines poll interval for pods ready check at statefulset rollout update | | VM_FORCERESYNCINTERVAL | 60s | false | configures force resync interval for VMAgent, VMAlert, VMAlertmanager and VMAuth. | | VM_ENABLESTRICTSECURITY | false | false | EnableStrictSecurity will add default `securityContext` to pods and containers created by operator Default PodSecurityContext include: 1. RunAsNonRoot: true 2. RunAsUser/RunAsGroup/FSGroup: 65534 '65534' refers to 'nobody' in all the used default images like alpine, busybox. If you're using customize image, please make sure '65534' is a valid uid in there or specify SecurityContext. 3. FSGroupChangePolicy: &onRootMismatch If KubeVersion>=1.20, use `FSGroupChangePolicy="onRootMismatch"` to skip the recursive permission change when the root of the volume already has the correct permissions 4. SeccompProfile: type: RuntimeDefault Use `RuntimeDefault` seccomp profile by default, which is defined by the container runtime, instead of using the Unconfined (seccomp disabled) mode. Default container SecurityContext include: 1. AllowPrivilegeEscalation: false 2. ReadOnlyRootFilesystem: true 3. Capabilities: drop: - all turn off `EnableStrictSecurity` by default, see https://github.com/VictoriaMetrics/operator/issues/749 for details | -[envconfig-sum]: f319004a92b62b1dad0c3e51323365dc \ No newline at end of file +[envconfig-sum]: f319004a92b62b1dad0c3e51323365dc