Overview of Enterprise Clusters
Clusters refer to systems composed of multiple master nodes that appear as a single entity externally, providing a single access point (domain name or IP address). This setup acts like a large computer.
Issues and Solutions
In internet applications, as site requirements for hardware performance, response speed, service availability, and data reliability increase, a single server cannot meet the needs for load balancing and high availability.
- Using expensive small or large servers
- Building a service cluster using relatively inexpensive general-purpose servers
One commonly used clustering technology in enterprises is LVS (Linux Virtual Server).
Types of Clusters Based on Target
- Load Balancing Cluster
- High Availability Cluster
- High Performance Computing Cluster
Load Balancing Cluster (Load Balance Cluster)
The goal is to enhance the responsiveness of application systems, distributing more access requests and reducing latency to achieve high concurrency and load (LB) performance.
High Availability Cluster (High Availability Cluster)
The aim is to improve the reliability of application systems, minimizing downtime, ensuring continuous service, and achieving high availability (HA) fault tolerance.
High Performance Computing Cluster (High Performance Computer Cluster)
The objective is to enhance the CPU computing speed, expand hardware resources, and improve analytical capabilities, achieving equivalent high-performance computing (HPC) capabilities as large or supercomputers.
Architecture of Load Balancing Clusters
Structure of Load Balancing
- First Layer: Load Scheduler (Load Balancer or Director): The unique entry point for accessing the entire cluster system, using a common VIP address for all servers. Typically configured with primary and backup schedulers for hot backup.
- Second Layer: Server Pool: The application services provided by the cluster, carried by the server pool. Each node has its own independent RIP address (real IP), handling client requests dispatched by the scheduler. When a node temporarily fails, the scheduler's failover mechanism isolates it, re-including it in the server pool after the issue is resolved.
- Third Layer: Shared Storage: Provides stable, consistent file access services for all nodes in the server pool, ensuring the cluster's uniformity. Shared storage can use NAS devices or dedicated servers providing NFS shared services.
Analysis of Load Balancing Cluster Working Modes
Load balancing clusters are currently the most widely used type of cluster in enterprises.
- Address Conversion
- IP Tunneling
- Direct Routing
NAT Mode (Address Conversion)
Network Address Translation, abbreviated as NAT mode. Similar to a private network structure behind a firewall, the load balancer serves as the gateway for all server nodes, acting both as the client's access entry and the exit for responses from each node.
TUN Mode —— IP Tunneling
Open network structure. The load balancer acts solely as the client's access entry, with each node responding directly to clients via their respective public IP addresses without passing through the load balancer.
DR Mode —— Direct Routing
Semi-open network structure. Similar to TUN mode, but nodes are not dispersed across different locations; instead, they are located on the same physical network segment as the load balancer.
About LVS Virtual Servers
Linux Virtual Server is a load balancing project developed for the Linux kernel. LVS essentially represents virtualized applications based on IP addresses, offering an efficient solution for load balancing based on IP addresses and content requests.
LVS is now part of the Linux kernel, default compiled as the ip_vs module, which can be automatically invoked if needed. In CentOS 7, the following operations can manually load the ip_vs module and check the version information of the current ip_vs module.
modprobe ip_vs # Manually load the ip_vs module cat /proc/net/ip_vs # Check the version information of the current ip_vs module
LVS Load Scheduling Algorithms
- Round Robin
- Weighted Round Robin
- Least Connections
- Weighted Least Connections
Managing LVS Clusters with ipvsadm
ipvsadm is a management tool used on the load balancer for managing LVS clusters. It calls the ip_vs module to add or remove server nodes and view the running status of the cluster. In CentOS 7, you need to manually install the ipvsadm.x86_64 0:1.27-7.el7 package.
Common Terminology
CIP: Client IP, the IP address of the client. VIP: Virtual IP, the IP address that the load balancer uses to provide external access, generally implemented for high availability through Virtual IP. RIP: RealServer IP, the IP address of the backend real server. DIP: Director IP, the IP address for communication between the load balancer and backend servers. CMAC: Client's MAC address, accurately should be the MAC address of the router connected to LVS. VMAC: The MAC address corresponding to the VIP of the load balancer LVS. DMAC: The MAC address corresponding to the DIP of the load balancer LVS. RMAC: The MAC address corresponding to the RIP of the backend real server.
NAT Mode LVS Load Balancing Cluster Deployment
Experiment Environment Preparation
Load Balancer: Internal gateway ens33: 192.168.10.40, External gateway ens37: 12.0.0.1
Web Node Server 1: 192.168.10.20
Web Node Server 2: 192.168.10.30
NFS Server: 192.168.10.10
Client: 12.0.0.10
Deployment of Shared Storage (NFS Server: 192.168.10.10)
systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 yum install nfs-utils rpcbind -y systemctl start rpcbind.service systemctl start nfs.service systemctl enable rpcbind.service systemctl enable nfs.service mkdir /opt/fzr /opt/zzj chmod 777 /opt/fzr /opt/zzj echo 'this is fzr web!' > /opt/fzr/index.html echo 'this is zzj web!' > /opt/zzj/index.html vim /etc/exports /opt/fzr 192.168.10.0/24(rw,sync) /opt/zzj 192.168.10.0/24(rw,sync) --Publish shared directory--- exportfs -rv
Configuration of Node Servers (192.168.10.20, 192.168.10.30)
systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0 yum install httpd -y systemctl start httpd.service systemctl enable httpd.service yum install nfs-utils rpcbind -y showmount -e 192.168.10.10 systemctl start rpcbind.service systemctl start nfs.service systemctl enable rpcbind.service systemctl enable nfs.service --192.168.10.20--- mount.nfs 192.168.10.10:/opt/fzr /var/www/html Another node server mount as follows: --192.168.10.30--- mount.nfs 192.168.10.10:/opt/zzj /var/www/html
Configuration of Load Balancer (Internal Gateway ens33: 192.168.10.10, External Gateway ens37: 12.0.0.1)
systemctl stop firewalld.service systemctl disable firewalld.service setenforce 0
Configuration of SNAT Forwarding Rules
vim /etc/sysctl.conf net.ipv4.ip_forward = 1 # Enable IP routing Or echo '1' > /proc/sys/net/ipv4/ip_forward sysctl -p iptables -t nat -F iptables -F iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o ens36 -j SNAT --to-source 12.0.0.10
Loading LVS Kernel Modules
modprobe ip_vs # Load ip_vs module cat /proc/net/ip_vs # View ip_vs version information ## Load all modules for i in $(ls /usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo $i; /sbin/modinfo -F filename $i >/dev/null 2>&1 && /sbin/modprobe $i;done
Installation of ipvsadm Management Tool
yum -y install ipvsadm --Before starting the service, save the load distribution strategy--- ipvsadm-save > /etc/sysconfig/ipvsadm or ipvsadm --save > /etc/sysconfig/ipvsadm systemctl start ipvsadm.service
Configuration of Load Distribution Strategy (NAT mode requires configuration only on the server, no special configuration on node servers)
ipvsadm -C # Clear existing policies ipvsadm -A -t 12.0.0.10:80 -s rr ipvsadm -a -t 12.0.0.10:80 -r 192.168.10.20:80 -m [-w 1] # Add real IP ipvsadm -a -t 12.0.0.10:80 -r 192.168.10.30:80 -m [-w 1] # Add real IP ipvsadm # Enable policy ipvsadm -ln # View node status, Masq indicates NAT mode ipvsadm-save > /etc/sysconfig/ipvsadm # Save policy ipvsadm -d -t 12.0.0.1:80 -r 192.168.10.20:80 -m [-w 1] # Delete one node server from the cluster ipvsadm -D -t 12.0.0.1:80 # Delete the entire virtual server systemctl stop ipvsadm # Stop service (clear policies) systemctl start ipvsadm # Start service (rebuild rules) ipvsadm-restore < /etc/sysconfig/ipvsadm # Restore LVS policy
Testing Effectiveness
Set the gateway in the network settings to the gateway server.
On a client machine with IP 12.0.0.10, open a browser and access http://12.0.0.10/, continuously refresh the browser to test the load balancing effect. Refresh intervals should be long.
Instance Operation: NAT Mode LVS Load Balancing Cluster Deployment
Environment Preparation
- Gateway Server Network Card Configuration (add new network card in virtual machine settings), internal gateway ens33: 192.168.10.40, external gateway ens36: 12.0.0.1
- Change the gateways of Web node servers and NFS server to the internal gateway of the load balancer ens33: 192.168.10.40
- Web Node Server 1: 192.168.10.20
Web Node Server 2: 192.168.10.30
NFS Server: 192.168.10.10 - Client IP and gateway settings
- Disable firewalls and SELinux on all devices
Deployment of Shared Storage (NFS Server: 192.168.2.100)
- Check if rpcbind and nfs-utils are installed and set up shared directories
- Start NFS service, publish NFS shared directory, and view
Configuration of Node Servers (192.168.10.20, 192.168.10.30)
- Configure Node Server 1 (192.168.10.20)
- Configure Node Server 2 (192.168.10.30)
Configuration of Load Balancer (internal gateway ens33: 192.168.10.40, external gateway ens37: 12.0.0.10)
- Configure SNAT forwarding rules
- Load LVS kernel modules
- Install ipvsadm management tool and start service
- Configure load distribution strategy (NAT mode requires configuration only on the server, no special configuration on node servers)