This white paper explores the technologies that provide the foundation for the Itential Platform’s High Availability (HA) capability and discusses multiple deployment scenarios.
Business-critical applications have always required continuous availability. As more organizations launch services online for consumption by global audiences, availability and scalability across distributed geographic regions become increasingly important considerations in systems design.
There are three principal reasons for the geographic distribution of databases across multiple data centers:
Under normal operating conditions, an Itential deployment will perform according to the performance and functional goals of the system. However, from time to time, certain inevitable failures or unintended actions can affect a system in adverse ways. Storage devices, network connectivity, power supplies, and other hardware components will fail.
These risks are often mitigated through use of redundant hardware components. Similarly, an Itential deployment provides configurable redundancy throughout both its software components and its data storage.
Before describing the Itential Platforms’s HA capability, a brief explanation of the platform’s architecture is required. The Itential Platform is developed using a combination of MEAN2 and MERN3 stack. Each of these is a set of Open-Source components that together provide an end-to-end framework for building dynamic web applications, starting from the top (code running in the browser) to the bottom (database). The stack is made up of:
In addition to the above, Itential uses Redis to manage Inter-Process Communication (IPC), Event Listening, and Client session tokens.
The following sections describe how each element of the Itential Platform architecture can be deployed to meet the customer’s High Availability (HA) and Disaster Recovery (DR) requirements, including server availability, client sessions management, and database replication and resilience.
High Availability (HA) refers to a site being up for as long as possible. This means there is enough infrastructure in the right locations to ensure there is no single point of failure that could take the site down.
Both failover and load balancing are means to achieve high availability:
There are numerous techniques that can be used to load balance client access requests across servers. When the load is low, one of the simple load balancing methods will suffice. But in times of high load, the more complex methods are used to ensure an even distribution of requests under network and service stress.
More complex/sophisticated algorithms are available, descriptions of which are outside the scope of this white paper.
The white paper assumes use of a load-balancer (either HAProxy5 or nginx6).
Note: The actual configuration of load-balancers is outside the scope of this document. It is enough to know that the Itential Platform supports multiple load balancing patterns.
To handle failover, Itential provides a health check API that can be used by load balancers to ensure traffic is not directed to a server that is currently unavailable. By using the [hostname]/status API, the load balancer routes requests only to healthy instances. When the load balancer determines that an instance is unhealthy, it stops routing requests to that instance and resumes once the instance has been restored to a healthy state.
Itential supports multiple patterns that support a wide range of customer deployment requirements, from simple single-server high availability to full blue/green multi-region disaster recovery configuration.
This section outlines the different patterns customers can deploy.
High Availability – The BasicsThe basic principle of HA systems specifies a design that aims to minimize the downtime of a system or service and keep them continuously available. The Itential Platform achieves this goal by using components that can be set up in high-availability clusters. The diagram below depicts a simplified high availability system architecture. In addition, the sections below on Redis and database replication explain in more details how HA is achieved at each layer.
The recommended minimum number of servers required to implement HA for Itential is 11 (as illustrated in the diagram below) — this being two Itential application servers, three Redis, and three MongoDB servers.
To avoid potential latency-related issues of data replication between clusters, it is recommended that each Itential Platform stack is deployed within the same data center (DC). However, if each data center is connected via very low latency links (e.g., sub 5m/s), then it is possible to deploy a stack that spans DCs. In that scenario, extensive testing should be performed to ensure latency issues do not arise.
Basic Active/StandbyAlso known as hot/warm or Active/Passive, active-standby consists of at least two nodes. However, as the name “active-standby” implies, not all nodes are going to be active. In the case of two nodes, for example, if the first node is already active, the second node must be passive or on standby.
The standby (a.k.a. failover) server acts as a backup that can take over as soon as the active (a.k.a. primary) server is disconnected or becomes unable to serve.
Highly Available Architecture (HA2)Also known as ‘hot/hot’, an active-active cluster is also typically made up of at least two nodes, but unlike active/standby, the nodes are actively running the same kind of service simultaneously. The main purpose of an active-active cluster is twofold: to provide component redundancy, which can tolerate at least one catastrophic failure, and to achieve load balancing. Load balancing distributes workloads across all nodes in order to prevent any single node from getting overloaded. Because there are more nodes available to serve, there will also be a marked improvement in throughput and response times.
Assigning of clients to the nodes in the cluster is not an arbitrary process. Rather, it is based on whatever load balancing algorithm is set on the load balancer. For example, in a “Round Robin” algorithm with two servers, the first client to connect is sent to the 1st server, the second client is sent to the 2nd server, the 3rd client back to the 1st server, the 4th client back to the 2nd server, and so on.
An Active/Standby Architecture (ASA) is an architecture that should be considered when Disaster Recovery is required. It is not required that they be geographically redundant, but they could be. The intent is to provide a standby environment that can be used in the event of a disaster that impacts the active stack. The standby stack should be preconfigured to quickly take over as the active stack.
The ideal ASA architecture will appear as two HA2 deployments, except for MongoDB. One or more of the MongoDB instances must be hosted in the standby location as a replica of the primary. Ideally, the MongoDB cluster will consist of at least 5 members. This will allow for a cluster of three mongos (query router components) in the worst-case disaster scenario.
This deployment requires two separate and complete Itential stacks. There is no constraint from a data center boundary, these stacks can exist in separate or the same data center. Control of the active environment is driven by the GLB load balancer. Components and configurations are not shared between the stacks: meaning, any changes made to one will have zero impact on the other.
This deployment is intended to reduce downtime as much as possible. A major advantage of this pattern is deployments and upgrades can be staged in advance during normal business hours against the inactive stack without impacting the active environment. Changes can be pushed, via automated CI/CD pipelines, to the inactive stack (green) without impacting the active deployment (blue). This allows ‘smoke tests’ to be performed in the production environment (devices, integrated ITSM systems etc.) in a limited and controlled manner, and when successful the flip from inactive to active (and vice versa) can be done via the GLB. In flight jobs on the old stack will continue to run until they are complete.
Developer ArchitectureA Developer Architecture is an Itential architecture where all of the components are installed as singletons and can not gracefully tolerate failures. This architecture is not recommended for production environments and testing environments that require strict uptimes and redundancies. This architecture is suitable for development environments only. The Itential Platform application performs many reads and writes against the database and is sensitive to great latencies. All components must be installed in the same data center.
The ideal ASA architecture will appear as two HA2 deployments, except for MongoDB. One or more of the MongoDB instances must be hosted in the standby location as a replica of the primary. Ideally, the MongoDB cluster will consist of at least 5 members. This will allow for a cluster of three mongos (query router components) in the worst-case disaster scenario.
The Itential Platform uses Redis for the following function:
Shared authentication token storage and expiration
Once a connection request to the Itential Platform has been successfully authenticated, either via an API call or a user connection from a browser, the session is allocated a token, which is then stored in Redis. Any subsequent call made within Itential will remain authenticated while the session token remains valid. The following sections discuss the configuration options available in an active/standby and active/active setup.
Active/StandbyWhen configured in Active/Standby mode, each instance of Redis is configured standalone, and tokens are not shared. When using a load-balancer that has been configured with ‘sticky’ connections, all subsequent requests will be sent to the same Itential Platform server, and the session tokens are valid.
Using Figure 6 as an example, assuming that the Itential Platform server in Data Center 1 fails, the load balancer will redirect traffic to the alternate Itential Platform server. As the session tokens stored in the Redis instance in DC1 are not replicated to DC2, the session token submitted with the request is invalid and the request is rejected. As a result, any session will need to be re-established via new login request.
Active/ActiveWhen running in active/active mode, the session tokens must be replicated to all Redis databases configured as a cluster. With this configuration, using the example below, when the ‘Itential 1’ server fails, the load balancer will direct traffic to the ‘Itential 2’ server. As the session tokens have been replicated to the Redis database, the session is deemed active, and the client application is not impacted by the failure.
To achieve an HA Topology, Redis Cluster functionality provides primary-secondary data replication across nodes. For monitoring and failover, Redis provides Redis Sentinel to automatically detect failures and bring the cluster back to a stable mode. Sentinel always checks the Primary and Secondary instances in the Redis cluster to determine whether they are working as expected. If Sentinel detects a failure in the Primary node in each cluster, Sentinel will start a failover process. As a result, Sentinel will pick a Secondary instance and promote it to become the Primary instance. Ultimately, the other remaining Secondary instances will be automatically reconfigured to use the new Primary instance.
Given Itentials enhanced HA capabilities, active/active is, when possible, the recommended configuration option for Redis.
As stated above, the database used by the Itential Platform is MongoDB. Out of the box, MongoDB provides extensive HA and data replication functionality, as well as application drivers7.
Using the node.js specific driver allows Itential to leverage all of the HA and replication capabilities that MongoDB provides. The primary functionality is the ability to monitor the state of the MongoDB deployment and ensure that the application is always connected to the Primary server. The sections below describe Replica Sets and member elections in more detail. Additionally, more details on MongoDB drivers are available in the following blog post.8
Replica SetsMongoDB maintains multiple copies of data, called replica sets, using native replication. Users should deploy replica sets to help prevent database downtime. Replica sets are self-healing as failover and recovery is fully automated, so it is not necessary to manually intervene to restore a system in the event of a failure.
Replica sets also enhance operational flexibility by providing a way to perform system maintenance (i.e., upgrading hardware and software) while preserving service continuity. This is an important capability as these operations can account for as much as one third of all downtime in traditional systems.
A replica set consists of multiple database replicas. At any given time, one member acts as the primary replica set member, and the other members act as secondary replica set members. If the primary member suffers an outage (e.g., a power failure, hardware fault, or network partition), one of the secondary members is automatically elected to primary, typically within several seconds, and the client connections automatically failover to that new primary member.
The number of replicas in a MongoDB replica set is configurable, with a larger number of replica members providing increased data durability and protection against database downtime (e.g., in case of multiple machine failures, rack failures, data center failures, or network partitions). Up to 7 voting members can be configured per replica set, providing operational flexibility and wide data distribution across multiple geographic sites.
You can learn more about the members of a replica set from the documentation.
Replica Set ElectionsIn the event of a primary replica set member failing, the election process is controlled by sophisticated algorithms based on an extended implementation of the Raft consensus protocol. Not only does this allow fast failover to maximize service availability, but the algorithms also ensure only the most suitable secondary members are evaluated for election to primary and reduce the risk of unnecessary failovers (“false positives”). Before a secondary replica set member is promoted, the election algorithms evaluate a range of parameters, including:
MongoDB provides a rich set of features to help users deploy highly available and scalable systems. In designing for high availability, administrators must evaluate read and write operations in the context of different failure scenarios. The performance and availability SLAs (Service Level Agreements) of a system play a significant role in determining:
Administrators can configure the behavior of MongoDB’s replica sets to enable data center awareness. Configuration can be based on a number of different dimensions, including awareness of geographical regions in deployments to multiple data centers, or racks, networks, and power circuits in a single data center.
With MongoDB, administrators can:
Read and write behaviors can be configured per operation and per collection. Collectively, they enable users to precisely control and scale database operations across regions, based on specific application requirements.
MongoDB allows users to specify write availability in the system, using an option called write concern. Each operation can specify the appropriate write concern, ranging from unacknowledged to an acknowledgement that writes have been committed to:
It is also possible to configure the write concern so that writes are only acknowledged once specific policies have been fulfilled, such as writing to at least two replica set members in one data center and at least one replica in a second data center.
As detailed in this white paper, the use of robust load-balancing techniques, clustered Redis deployments, and geographically distributed MongoDB replica sets allows an organization to design an HA infrastructure that will allow rapid response to events, minimize the number of disruptions, and nearly eliminate the resulting periods of downtime.
See how Itential connects AI reasoning to governed execution across your entire infrastructure.
