公司共有两个业务，网上图书馆和一个电商网站。
现要求运维设计一个安全架构，本着高可用、廉价的原则。
具体情况如下：
网上图书馆是基于jsp开发；
电商系统是基于php开发；
两个网站都是使用的mysql数据库；
要求你提供一个高可用且廉价的架构方案，部署分发业务到外网；
请画出拓扑图及通过虚拟机搭建测试环境。
答
1. 服务器的配置以及信息
| 系统 | 配置 | ip | 模块名 |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.5/192.168.179.5 | lb01 |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.6/192.168.179.6 | lb02 |
| CentOS release 6 64位 | 1核1G 50G 硬盘 | 172.16.1.7/192.168.179.7 | web01 |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.8/192.168.179.8 | web02 |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.9/192.168.179.9 | web03 |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.10/192.168.179.10 | web04 |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.31/192.168.179.31 | NFS |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.41/192.168.179.41 | Rsync |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.51/192.168.179.51 | 数据库 |
| CentOS release 6.9 64位 | 1核1G 50G 硬盘 | 172.16.1.51/192.168.179.51 | ansible |
1. lb模块（2个项目共用）
1) lb模块采用的负载均衡机制， 运用了vrrp（路由冗余协议）， 保证一台宕机 不影响业务的运转， 基于并发量的考虑我们这边使用了（keepalived+nginx）这种架构， 当后面并发上来，可以考虑换成（Keepalived + LVS ）架构， 或者商业硬件，比如F5,A10之类的。
2) 我们这里做了互为主备配置
| lb的物理ip | 主 | 备 | 虚拟ip |
| 172.16.1.5/192.168.179.5（lb01） | 网上图书馆 | 电商系统 | 192.168.179.3 |
| 172.16.1.6/192.168.179.6（lb02） | 电商系统 | 网上图书馆 | 192.168.179.4 |
2. 网上图书馆业务处理模块
基于安全考虑nginx 反向代理tomcat， 把tomcat放在后面， 不直接对外开放， 一些需要用到的图片资源,文件资源， 用共享文件服务器（nfs）,然后数据直接同步到备份服务器。数据直接写入主mysql，从从mysql里面读数据。我们这里直接就一台mysql， 没有做读写分离
| 主机名 | 安装的软件 | ip |
| web01 | tomcat+ jdk + nginx | 172.16.1.7/192.168.179.7 |
| web02 | tomcat+ jdk + nginx | 172.16.1.8/192.168.179.8 |
| nfs01 | nfs-utils rpcbind rsync sersync | 172.16.1.31/192.168.179.31 |
| backup | rsync | 172.16.1.41/192.168.179.41 |
| db | mysql | 172.16.1.51/192.168.179.51 |
3. 电商系统业务处理模块
一些需要用到的图片资源,文件资源， 用共享文件服务器（nfs）， 然后数据直接同步到备份服务器。 数据直接写入主mysql，从从mysql里面读数据。我们这里直接就一台mysql，没有做读写分离
| 主机名 | 安装的软件 | ip |
| web03 | nginx+ php | 172.16.1.7/192.168.179.7 |
| web04 | nginx+ php | 172.16.1.8/192.168.179.8 |
| nfs01 | nfs-utils rpcbind rsync sersync | 172.16.1.31/192.168.179.31 |
| backup | rsync | 172.16.1.41/192.168.179.41 |
| db | mysql | 172.16.1.51/192.168.179.51 |
2. 安装的软件信息
| 软件的名称 | 版本 | 备注 |
| mysql | 5.1.73-8 | 64位 |
| nginx | 1.10.2-1 | 64位 |
| keepalived | 1.2.13-5 | 64位 |
| nfs-utils | 1.2.3-78 | 64位 |
| rpcbind | 0.2.0-16 | 64位 |
| php | 5.3.3 | 64位 |
| tomcat | 7.0.90-1 | 64位 |
| jdk | 1.8.0_191 | 64位 |
3. 配置ansible服务器
1） 配置需要目录
[root@m01 ~]# mkdir -p /server/scripts
[root@m01 ~]# mkdir -p /server/tools
[root@m01 ~]# mkdir -p /application
2） 查看是否缺少什么软件，
rpm -qa|grep openssh #缺少openssh-clients，所以没有ssh-copy-id
rpm -qa|grep ansible #没安装
rpm -qa|grep vim #没安装
3） 安装软件
[root@m01 ~]#yum install -y openssh* ansible vim
[root@m01 ~]#mkdir -pv /etc/ansible/roles/{mysql,keepalived,nginx,jsp,rsync,nfs }/{files,tasks,templates, }
4） 创建密钥分发脚本
[root@m01 ~]# cd /server/scripts/
[root@m01 scripts]# vim fenfa.sh
#!/bin/bash
ipz=(5 6 7 8 9 10 31 41 51 61)
######################
#删除KEY文件 #
######################
rm -f /root/.ssh/id_dsa*
######################
#创建密钥对 #
######################
ssh-keygen -t dsa -f /root/.ssh/id_dsa -N “”
######################
#分发密钥对 #
######################
for ip in ${ipz[*]} #${ipz[*]}显示ipz数组
do
sshpass -p123456 ssh-copy-id -i /root/.ssh/id_dsa.pub “-o StrictHostKeyChecking=no 172.16.1.$ip”
done
5） 创建检查是否可以进行基于密钥远程管理
[root@m01 scripts]# vim fenfa_check.sh
#!/bin/bash
ipz=(5 6 7 8 9 10 31 41 51 61)
######################
#请输入参数 #
######################
if [ $# -ne 1 ];then
echo “pls input one agrs”
exit 1
fi
######################
#测试 #
######################
for ip in ${ipz[*]}
do
echo ===== info 172.16.1.$ip=====
ssh 172.16.1.$ip $1 ##$1 表示第一个参数
echo “”
done
测试
[root@m01 scripts]# sh fenfa_check.sh hostname
===== info 172.16.1.5=====
lb01
===== info 172.16.1.6=====
lb02
===== info 172.16.1.7=====
web01
===== info 172.16.1.8=====
web02
===== info 172.16.1.9=====
web03
===== info 172.16.1.10=====
web04
===== info 172.16.1.31=====
nfs01
===== info 172.16.1.41=====
backup
===== info 172.16.1.51=====
MySQL
===== info 172.16.1.61=====
m01
6） 其他客户端安装libselinux-python
[root@m01 scripts]# sh fenfa_check.sh “yum install -y libselinux-python”
[root@m01 scripts]# sh fenfa_check.sh “rpm -qa|grep libselinux-python”
===== info 172.16.1.5=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.6=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.7=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.8=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.9=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.10=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.31=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.41=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.51=====
libselinux-python-2.0.94-7.el6.x86_64
===== info 172.16.1.61=====
libselinux-python-2.0.94-7.el6.x86_64
7）编写ansible的hosts文件
[root@m01 ~]# vim /etc/ansible/hosts
[keepalived]
172.16.1.6
172.16.1.5
[php]
172.16.1.7
172.16.1.8
[jdk]
172.16.1.9
172.16.1.10
[nfs]
172.16.1.31
[rsync]
172.16.1.41
[mysql]
172.16.1.51
4. 安装mysql
1） mysql剧本
– hosts: mysql
tasks:
– name: step01:mkdir server/tools
file:
name: /server/tools
state: directory
– name: step02:mkdir server/scripts
file:
name: /server/scripts
state: directory
– name: step03:install mysql
yum: name=mysql-devel,mysql-server state=installed
– name: step04:install mysql-python
yum: name=MySQL-python state=installed
– name: step05:create mysql data
file:
path: /data/mysql
state: directory
owner: mysql
group: mysql
– name: step06:change mysql data
lineinfile:
dest: /etc/my.cnf
regexp: ‘^datadir=’
line: ‘datadir=/data/mysql’
– name: step07:statr mysqld
service: name=mysqld state=started enabled=yes
– name: step08:change password
mysql_user:
user: ‘root’
login_password: ”
password: ‘123456a’
update_password: ‘always’
– name: step09:create dianshang db
mysql_db:
login_user: ‘root’
login_password: ‘123456a’
name: ‘dianshang’
state: ‘present’
– name: step10:create tushu db
mysql_db:
login_user: ‘root’
login_password: ‘123456a’
name: ‘tushu’
state: ‘present’
– name: step11:copy mysqlshow.sh
copy:
src: /etc/ansible/roles/mysql/files/mysqlshow.sh
dest: /server/scripts
– name: step12:sh mysqlshow.sh
shell: sh /server/scripts/mysqlshow.sh
2） 创建mysqlshow.sh
#/bin/bash
user_dsl=”grant all on dianshang.* to dianshang@’172.16.1.0/255.255.255.0′ identified by ‘sung123′”
user_ds=”grant all on dianshang.* to dianshang@’localhost’ identified by ‘sung123′”
user_tsl=”grant all on tushu.* to tushu@’172.16.1.0/255.255.255.0′ identified by ‘sung123′”
user_ts=”grant all on tushu.* to tushu@’localhost’ identified by ‘sung123′”
mysql -uroot -p123456a -e “$user_ds”
mysql -uroot -p123456a -e “$user_ts”
mysql -uroot -p123456a -e “$user_dsl”
mysql -uroot -p123456a -e “$user_tsl”
5. 安装NFS
1） NFS剧本
– hosts: nfs
tasks:
– name: step01:install nfs
yum:
name: nfs-utils,rpcbind
state: installed
– name: step02:mkdir data
file:
path: /data
recurse: yes
owner: nfsnobody
group: nfsnobody
state: directory
– name: step03:mkdir data/jdk
file:
path: /data/jdk
recurse: yes
owner: nfsnobody
group: nfsnobody
state: directory
– name: step04:mkdir data/php
file:
path: /data/php
recurse: yes
owner: nfsnobody
group: nfsnobody
state: directory
– name: step05:echo exports
shell: echo “/data 172.16.1.0/24(rw,sync)” > /etc/exports
– name: step06:statr rpcbind
service:
name: rpcbind
state: restarted
enabled: yes
– name: step07:statr nfs
service:
name: nfs
state: restarted
enabled: yes
– hosts: php
tasks:
– name: step08:install nfs
yum:
name: nfs-utils,rpcbind
state: installed
– name: step09:mkdir data/php
file:
path: /data/php
recurse: yes
state: directory
– hosts: jdk
tasks:
– name: step10:install nfs
yum:
name: nfs-utils,rpcbind
state: installed
– name: step11:mkdir data/jdk
file:
path: /data/jdk
recurse: yes
state: directory
– hosts: jdk,php
tasks:
– name: step12:statr rpcbind
service:
name: rpcbind
state: restarted
enabled: yes
– name: step13:statr nfs
service:
name: nfs
state: restarted
enabled: yes
6. 安装sersync
1） sersync安装剧本
– hosts: rsync
tasks:
– name: step01:install rsync
yum:
name: rsync
state: installed
– name: step02:copy rsync.conf
copy:
src: /etc/ansible/roles/rsync/templates/rsyncd.conf
dest: /etc/
– name: step03:create user rsync
user:
name: rsync
createhome: no
shell: /sbin/nologin
state: present
– name: step04:mkdir /backup
file:
path: /backup
recurse: yes
owner: rsync
group: rsync
state: directory
– name: step05:copy rsync.password
copy:
src: /etc/ansible/roles/rsync/templates/rsync.password
dest: /etc/
mode: 600
– name: step06:start rsync
shell: rsync –daemon creates=/var/run/rsyncd.pid
– hosts: nfs,mysql
tasks:
– name: step07:mkdir server/tools
file:
name: /server/tools
state: directory
– name: step08:mkdir server/scripts
file:
name: /server/scripts
state: directory
– name: step09:install rsync unzip
yum:
name: rsync,unzip
state: installed
– name: step10:copy rsync.password
copy:
src: /etc/ansible/roles/rsync/templates/rsync_client.password
dest: /etc/rsync.password
mode: 600
– name: step11:copy sersync.zip
copy:
src: /etc/ansible/roles/rsync/files/sersync_installdir_64bit.zip
dest: /server/tools
– name: step12:copy sersync.sh
copy:
src: /etc/ansible/roles/rsync/files/mzsersync.sh
dest: /server/scripts/
– name: step13:sh sersync.sh
shell: sh /server/scripts/mzsersync.sh
– name: step14:copy sersync.conf
template:
src: /etc/ansible/roles/rsync/templates/sersync.conf
dest: /usr/local/sersync/conf/confxml.xml
– name: step15:copy start sersync.sh
copy:
src: /etc/ansible/roles/rsync/files/start.sh
dest: /server/scripts/start.sh
– name: step09:statr sersync
shell: sh /server/scripts/start.sh start
2） sersync解压脚本
#!/bin/bash
if [ ! -f /usr/local/sersync ];
then
cd /server/tools/
unzip sersync_installdir_64bit.zip
cd sersync_installdir_64bit
mv sersync /usr/local/
fi
~
3） rsync.conf
uid = rsync
gid = rsync
use chroot = no
max connections = 200
timeout = 300
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
[backup]
comment = “backup dir by oldboy”
path = /backup
4） serync.conf
<?xml version=”1.0″ encoding=”ISO-8859-1″?>
<head version=”2.5″>
<host hostip=”localhost” port=”8008″></host>
<debug start=”false”/>
<fileSystem xfs=”false”/>
<filter start=”false”>
<exclude expression=”(.*)\.svn”></exclude>
<exclude expression=”(.*)\.gz”></exclude>
<exclude expression=”^info/*”></exclude>
<exclude expression=”^static/*”></exclude>
</filter>
<inotify>
<delete start=”true”/>
<createFolder start=”true”/>
<createFile start=”false”/>
<closeWrite start=”true”/>
<moveFrom start=”true”/>
<moveTo start=”true”/>
<attrib start=”false”/>
<modify start=”false”/>
</inotify>
<sersync>
<localpath watch=”/data”>
<remote ip=”172.16.1.41″ name=”backup”/>
<!–<remote ip=”192.168.8.39″ name=”tongbu”/>–>
<!–<remote ip=”192.168.8.40″ name=”tongbu”/>–>
</localpath>
<rsync>
<commonParams params=”-az”/>
<auth start=”true” users=”rsync_backup” passwordfile=”/etc/rsync.password”/>
<userDefinedPort start=”false” port=”874″/><!– port=874 –>
<timeout start=”false” time=”100″/><!– timeout=100 –>
<ssh start=”false”/>
</rsync>
<failLog path=”/tmp/rsync_fail_log.sh” timeToExecute=”60″/><!–default every 60mins execute once–>
<crontab start=”false” schedule=”600″><!–600mins–>
<crontabfilter start=”false”>
<exclude expression=”*.php”></exclude>
<exclude expression=”info/*”></exclude>
</crontabfilter>
</crontab>
<plugin start=”false” name=”command”/>
</sersync>
<plugin name=”command”>
<param prefix=”/bin/sh” suffix=”” ignoreError=”true”/> <!–prefix /opt/tongbu/mmm.sh suffix–>
<filter start=”false”>
<include expression=”(.*)\.php”/>
<include expression=”(.*)\.sh”/>
</filter>
</plugin>
<plugin name=”socket”>
<localpath watch=”/opt/tongbu”>
<deshost ip=”192.168.138.20″ port=”8009″/>
</localpath>
</plugin>
<plugin name=”refreshCDN”>
<localpath watch=”/data0/htdocs/cms.xoyo.com/site/”>
<cdninfo domainname=”ccms.chinacache.com” port=”80″ username=”xxxx” passwd=”xxxx”/>
<sendurl base=”http://pic.xoyo.com/cms”/>
<regexurl regex=”false” match=”cms.xoyo.com/site([/a-zA-Z0-9]*).xoyo.com/images”/>
</localpath>
</plugin>
</head>
5） serync启动脚本
#! /bin/bash
#
#sersyncd
#
. /etc/rc.d/init.d/functions
case “$1” in
start)
cd /usr/local/sersync/bin
chmod +x sersync
./sersync -dro /usr/local/sersync/conf/confxml.xml
if [ $? -eq 0 ]
then
echo -e “Staring sersyncd [ OK ]”
exit 0
fi
;;
stop)
kill ‘ps aux | grep sersync | grep -v grep | awk ‘{print $2}”
if [ $? -eq 0 ]
then
echo -e “Stopping sersyncd [ OK ]”
exit 0
fi
;;
status)
ps aux | grep sersync | grep -v grep
;;
esac
7. 安装图书馆模块和电商模块
1）图书馆模块
– hosts: jdk
tasks:
– name: step01:install java
yum:
name: java
state: installed
– name: step02:install tomcat
yum:
name: tomcat,tomcat-webapps,tomcat-admin-webapps
state: installed
– name: step04:mkdir drp/
file:
name: /usr/share/tomcat/webapps/drp
state: directory
– hosts: 172.16.1.9
tasks:
– name: step01:copy test.html
copy:
src: /etc/ansible/roles/jsp/templates/index.html
dest: /usr/share/tomcat/webapps/drp/index.html
– hosts: 172.16.1.10
tasks:
– name: step01:copy test.html
copy:
src: /etc/ansible/roles/jsp/templates/index2.html
dest: /usr/share/tomcat/webapps/drp/index.html
– hosts: jdk
tasks:
– name: step01:start tomcat
service: name=tomcat state=restarted enabled=yes
2）电商模块
– hosts: php
tasks:
– name: spet01:install epel
yum:
name: epel-release,php
state: installed
– name: spet02:install nginx
yum:
name: php,nginx,php-fpm
state: installed
– hosts: 172.16.1.7
tasks:
– name: step01:copy test.html
copy:
src: /etc/ansible/roles/nginx/templates/index.html
dest: /usr/share/nginx/html/sung.html
– hosts: 172.16.1.8
tasks:
– name: step01:copy test.html
copy:
src: /etc/ansible/roles/nginx/templates/index2.html
dest: /usr/share/nginx/html/sung.html
– hosts: php
tasks:
– name: step01:start ngixn
service: name=nginx state=restarted enabled=yes
8. 安装keepalived
1） Keepalived剧本
– hosts: keepalived
tasks:
– name: step01:install keepalived nginx
yum:
name: keepalived,nginx
state: installed
– name: step02:copy nginx.conf(PHP)
copy:
src: /etc/ansible/roles/keepalived/templates/PHP.conf
dest: /etc/nginx/conf.d/PHP.conf
– name: step03:copy nginx.conf(JSP)
copy:
src: /etc/ansible/roles/keepalived/templates/JSP.conf
dest: /etc/nginx/conf.d/JSP.conf
– name: step04:rm keepalived.conf
shell: rm -f /etc/keepalived/keepalived.conf
– name: step06:copy cjeck_web.sh
copy:
src: /etc/ansible/roles/keepalived/files/check_web.sh
dest: /server/scripts/check_web.sh
mode: 0755
– hosts: 172.16.1.5
tasks:
– name: step04:copy (lb01)
copy:
src: /etc/ansible/roles/keepalived/templates/lb01.conf
dest: /etc/keepalived/keepalived.conf
– hosts: 172.16.1.6
tasks:
– name: step05:copy (lb02)
copy:
src: /etc/ansible/roles/keepalived/templates/lb02.conf
dest: /etc/keepalived/keepalived.conf
– hosts: keepalived
tasks:
– name: step06:statr keepalived
service: name=keepalived state=restarted enabled=yes
– name: step07:statr nginx
service: name=nginx state=restarted enabled=yes
2） lb0102.conf
Lb02
! Configuration File for keepalived
global_defs {
router_id lb02
}
vrrp_script check_web {
#定义一个监控脚本，脚本必须有执行权限
script “/server/scripts/check_web.sh”
#指定脚本间隔时间
interval 2
#脚本执行完成，让优先级值和权重值进行运算，从而实现主备切换
weight 2
}
vrrp_instance gorup01 {
state MASTER
interface eth0
virtual_router_id 51
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.179.3/24 dev eth0 label eth0:1
}
track_script {
check_web
}
}
vrrp_instance gorup02 {
state BACKUP
interface eth0
virtual_router_id 52
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.179.4/24 dev eth0 label eth0:2
}
track_script {
check_web
}
}
Lb01
! Configuration File for keepalived
global_defs {
router_id lb01
}
vrrp_script check_web {
#定义一个监控脚本，脚本必须有执行权限
script “/server/scripts/check_web.sh”
#指定脚本间隔时间
interval 2
#脚本执行完成，让优先级值和权重值进行运算，从而实现主备切换
weight 2
}
vrrp_instance gorup01 {
state BACKUP
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.179.3/24 dev eth0 label eth0:1
}
track_script {
check_web
}
}
vrrp_instance gorup02 {
state MASTER
interface eth0
virtual_router_id 52
priority 150
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.179.4/24 dev eth0 label eth0:2
}
track_script {
check_web
}
}
3）PHP.conf
upstream PHP{
#集群有几台服务器即可配置几台，weight表示权重，权重越大被访问到的几率越大
#这里添加的是上面启动好的两台Nginx服务器
server 192.168.179.8:80 weight=1;
server 192.168.179.7:80 weight=1;
}
#nginx基本配置
server{
listen 80; #端口号
server_name 192.168.179.4; #服务名
location /{
#将访问请求转向至服务器集群,mycluster和上面upstream mycluster 对应
proxy_pass http://PHP;
# 真实的客户端IP
proxy_set_header X-Real-IP $remote_addr;
# 请求头中Host信息
proxy_set_header Host $host;
# 代理路由信息，此处取IP有安全隐患
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# 真实的用户访问协议
proxy_set_header X-Forwarded-Proto $scheme;
}
}
4）JSP.conf
upstream JSP {
#集群有几台服务器即可配置几台，weight表示权重，权重越大被访问到的几率越大
#这里添加的是上面启动好的两台Tomcat服务器
server 192.168.179.9:8080 weight=1;
server 192.168.179.10:8080 weight=1;
}
#nginx基本配置
server{
listen 8088; #端口号
server_name 192.168.179.3; #服务名
location /{
#将访问请求转向至服务器集群,mycluster和上面upstream mycluster 对应
proxy_pass http:// JSP;
# 真实的客户端IP
proxy_set_header X-Real-IP $remote_addr;
# 请求头中Host信息
proxy_set_header Host $host;
# 代理路由信息，此处取IP有安全隐患
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# 真实的用户访问协议
proxy_set_header X-Forwarded-Proto $scheme;
}
}
5） check_web
#!/bin/bash
web_info=$(ps -ef|grep [n]ginx|wc -l)
if [ $web_info -lt 2 ]
then
/etc/init.d/keepalived stop
fi
6） 测试
1） 测试网页
2） 测试存储和备份
3） 关闭lb01
因为博客排版问题，剧本格式显示不对，附带剧本压缩包