ucloud云上部署GreenPlum4.3.12集群
一、机器环境、版本如下：
操作系统：CentOS6.5
数据库版本：
greenplum-db-4.3.12.0-rhel5-x86_64
greenplum-cc-web-3.2.0-LINUX-x86_64
节点信息：
主机名 Ip地址 说明
mdw 172.28.1.11 主库master
smdw 172.28.1.12 备库standby master
sdw1 172.28.1.13 segment库节点一
sdw2 172.28.1.14 segment库节点二二、系统初始化设置(四台机器都修改)
1、修改系统参数/etc/sysctl.conf
kernel.shmmax = 500000000
kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 250 512000 100 2048
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65535
net.core.netdev_max_backlog = 10000
vm.overcommit_memory = 22、文件句柄数修改/etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
* soft core unlimited3、关闭selinux、iptables
修改/etc/selinux/config中SELINUX=disabled
关闭iptables
service iptables stop
chkconfig iptables off4、调整磁盘IO调度
Linux磁盘I/O调度器对磁盘的访问支持不同的策略，默认的为cfq，GreenPlum建议设置为deadline
查看磁盘的I/O调度策略，看到默认的为[cfq]
查看系统分区情况：
[root@dw01 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 20G 3.3G 16G 18% /
tmpfs 12G 0 12G 0% /dev/shm
/dev/vdb 493G 311M 467G 1% /data查看/分区的I/O调度策略
[root@dw01 ~]# cat /sys/block/vda/queue/scheduler
noop anticipatory [deadline] cfq
查看/data分区的I/O调度策略
[root@dw01 ~]# cat /sys/block/vdb/queue/scheduler
noop anticipatory [deadline] cfq
查看当前系统内核
[root@dw01 ~]# uname -r
2.6.32-696.18.7.1.el6.ucloud.x86_64修改系统引导文件，在/boot/grub/menu.lst 文件里面关于kernel这一行的末尾添加elevator=deadline
/boot/grub/menu.lst
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You do not have a /boot partition. This means that
# all kernel and initrd paths are relative to /, eg.
# root (hd0,0)
# kernel /boot/vmlinuz-version ro root=/dev/vda1
# initrd /boot/initrd-[generic-]version.img
#boot=/dev/vda
default=0
timeout=1
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.32-696.18.7.1.el6.ucloud.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-696.18.7.1.el6.ucloud.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M-2G:64M,2G-4G:128M,4G-:192M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM quiet console=tty1 console=ttyS0,115200n8 elevator=deadline
initrd /boot/initramfs-2.6.32-696.18.7.1.el6.ucloud.x86_64.img
title CentOS (2.6.32-696.18.7.el6.x86_64.debug)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-696.18.7.el6.x86_64.debug ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M-2G:64M,2G-4G:128M,4G-:192M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM quiet console=tty1 console=ttyS0,115200n8
initrd /boot/initramfs-2.6.32-696.18.7.el6.x86_64.debug.img
title CentOS (2.6.32-431.11.29.el6.ucloud.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-431.11.29.el6.ucloud.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M-2G:64M,2G-4G:128M,4G-:192M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM quiet console=tty1 console=ttyS0,115200n8
initrd /boot/initramfs-2.6.32-431.11.29.el6.ucloud.x86_64.img
title CentOS (2.6.32-431.11.25.el6.ucloud.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-431.11.25.el6.ucloud.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M-2G:64M,2G-4G:128M,4G-:192M KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM quiet console=tty1 console=ttyS0,115200n8
initrd /boot/initramfs-2.6.32-431.11.25.el6.ucloud.x86_64.img查看硬盘I/O预读扇区值：默认为256
[root@dw01 ~]# blockdev --getra /dev/vda1
256
[root@dw01 ~]# blockdev --getra /dev/vdb
256修改为65536
blockdev --setra 65536 /dev/vda1
blockdev --setra 65536 /dev/vdb须将其写入开机配置文件/etc/rc.d/rc.local 否则重启就会失效。
/etc/rc.d/rc.local
echo "blockdev --setra 65536 /dev/vda1" >> /etc/rc.d/rc.local
echo "blockdev --setra 65536 /dev/vdb" >> /etc/rc.d/rc.local 5、修改hostname
[root@dw01 ~]# vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=dw01
NOZEROCONF=yes6、修改hosts
/etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.28.1.11 dw01 mdw
172.28.1.12 dw02 smdw
172.28.1.13 dw03 sdw1
172.28.1.14 dw04 sdw2以上设置四台机器都需设置，重启生效；
7、创建配置文件
a、创建包含Greenplum部署的所有主机名，内容如下
[gpadmin@dw01 gpconfig]$ cat host_file
mdw
smdw
sdw1
sdw2b、创建包含备库standby mastersegment库节点一节点二的主机名，内容如下
[gpadmin@dw01 gpconfig]$ cat dw234
smdw
sdw1
sdw2c、创建包含segment库节点一节点二的主机名，内容如下
[gpadmin@dw01 gpconfig]$ cat seg_hosts
sdw1
sdw2三、下载GreenPlum安装包
根据操作系统版本下载：
https://network.pivotal.io/products/pivotal-gpdb#/releases/4540/file_groups/560这里选择
Greenplum Database 4.3.12.0 for RedHat Entrerprise Linux 5, 6 and 7
122 MB4.3.12.0注意 登陆Pivotal Network账号以后，才能下载
md5sum *zip
ee90c7a35c706404840be38ba1be557b greenplum-cc-web-3.2.0-LINUX-x86_64.zip
edaa67d561653fbf81431c968e5f297f greenplum-db-4.3.12.0-rhel5-x86_64.zip四、解压安装
unzip greenplum-db-4.3.12.0-rhel5-x86_64.zip
./greenplum-db-4.3.12.0-rhel5-x86_64.bin
根据系统提示输入yes和回车，默认会按照在/usr/local/greenplum-db-4.3.12.0下，并创建软链接greenplum-db ---/greenplum-db-4.3.12.0
在/home/gpadmin/.bash_profile文件中添加
source /usr/local/greenplum-db/greenplum_path.sh
export GPHOME=/usr/local/greenplum-db
export MASTER_DATA_DIRECTORY=/data/gpmaster/gpseg-1五、免key登陆
四台机器之间互相做免密码登陆，参考/Linux/ssh-key.html
source /usr/local/greenplum-db/greenplum_path.sh
gpssh-exkeys -f /usr/local/greenplum-db/gpconfig/host_file #打通所有服务器六、创建配置文件gpinitsystem_config
在master机器上创建
mkdir -p /usr/local/greenplum-db/gpconfig
创建/usr/local/greenplum-db/gpconfig/gpinitsystem_config文件内容如下：
ARRAY_NAME="EMC Greenplum DW"
SEG_PREFIX=gpseg
PORT_BASE=40000
declare -a DATA_DIRECTORY=(/data/gpdata/gpdatap1 /data/gpdata/gpdatap1)
MASTER_HOSTNAME=dw01
MASTER_DIRECTORY=/data/gpmaster
MASTER_PORT=5432
TRUSTED_SHELL=ssh
CHECK_POINT_SEGMENTS=8
ENCODING=UNICODE
MIRROR_PORT_BASE=50000
REPLICATION_PORT_BASE=41000
MIRROR_REPLICATION_PORT_BASE=51000
declare -a MIRROR_DATA_DIRECTORY=(/data/gpdata/gpdatam1 /data/gpdata/gpdatam1)
MACHINE_LIST_FILE=/usr/local/greenplum-db/gpconfig/seg_hostsgpinitsystem_config关键配置项参数说明：
DATA_DIRECTORY：决定了每个Segment上部署多少个Primary Instance，一个目录就是一个Instance。这里建议把所有的Instance都指向一个目录，例如部署两个实例，则为：/gpdata/primary /gpdata/primary。原因是我们在使用中会创建新的文件系统，这样便于和文件系统区分。
MASTER_HOSTNAME：Master的主机名。
MASTER_DIRECTORY：Master文件目录。
MASTER_PORT：该端口用于访问Greenplum数据库。
MIRROR_DATA_DIRECTORY：（如果需要部署Mirror），决定了每个Segment上部署多少个Mirror Instance。类似于DATA_DIRECTORY，例如：/gpdata/mirror /gpdata/mirror。
DATABASE_NAME：创建一个数据库，输入数据库名称。也可以之后手工创建。六、分发、创建用户、目录、权限
安装包分发：master上操作
cd /usr/local
scp -r greenplum-db-4.3.12.0/ dw02:/usr/local/
scp -r greenplum-db-4.3.12.0/ dw03:/usr/local/
scp -r greenplum-db-4.3.12.0/ dw04:/usr/local/
gpssh -f /usr/local/greenplum-db/gpconfig/host_file
Note: command history unsupported on this machine ...
=>
依次执行以下命令
useradd -g gpadmin gpadmin
useradd -g gpmon gpmon
echo 123456 | passwd gpadmin --stdin
echo 123456 | passwd gpmon --stdin
mkdir -p /data/gpdata/gpdatap1
mkdir -p /data/gpdata/gpdatam1
mkdir -p /data/gpmaster
chown -R gpadmin.gpadmin /data/gpdata/
chown -R gpadmin.gpadmin /data/gpmaster/
chown -R gpadmin.gpadmin /usr/local/greenplum-db-4.3.12.0/七、系统检查
source /usr/local/greenplum-db/greenplum_path.sh
gpcheck -f /usr/local/greenplum-db/gpconfig/host_file -m mdw -s smdw
20180406:14:39:12:026168 gpcheck:dw01:root-[INFO]:-dedupe hostnames
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-Detected platform: Generic Linux Cluster
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-generate data on servers
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-copy data files from servers
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-delete remote tmp files
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-Using gpcheck config file: /usr/local/greenplum-db/./etc/gpcheck.cnf
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-GPCHECK_NORMAL
20180406:14:39:13:026168 gpcheck:dw01:root-[INFO]:-gpcheck completing...如果系统检查有Error提示，需根据提示修改系统参数，然后重新检查
检查网络性能
gpcheckperf -f host_file -r N -d /tmp/ > checknet.out
cat checknet.out
-------------------
-- NETPERF TEST
-------------------
====================
== RESULT
====================
Netperf bisection bandwidth test
mdw -> smdw = 287.060000
sdw1 -> sdw2 = 279.900000
smdw -> mdw = 299.840000
sdw2 -> sdw1 = 302.450000
Summary:
sum = 1169.25 MB/sec
min = 279.90 MB/sec
max = 302.45 MB/sec
avg = 292.31 MB/sec
median = 299.84 MB/sec八、初始化数据库
seg_hosts是segment服务器列表，一行存一个hostname，smdw是standby master的hostname名字，在gpadmin账号下运行：
su - gpadmin
cd /usr/local/greenplum-db/gpconfig/
gpinitsystem -c /usr/local/greenplum-db/gpconfig/gpinitsystem_config -h seg_hosts -s smdw如有报错参考官方文档：http://gpdb.docs.pivotal.io/43120/install_guide/init_gpdb.html
安装完以后登陆
[gpadmin@dw01]$ psql -d postgres
psql (8.2.15)
Type "help" for help.
postgres=# help
You are using psql, the command-line interface to PostgreSQL.
Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
postgres=#查看各机器进程
gpssh -f /usr/local/greenplum-db/gpconfig/host_file
Note: command history unsupported on this machine ...
=> netstat -nltp | grep postgres
[sdw2] (Not all processes could be identified, non-owned process info
[sdw2] will not be shown, you would have to be root to see it all.)
[sdw2] tcp 0 0 0.0.0.0:40000 0.0.0.0:* LISTEN 10601/postgres
[sdw2] tcp 0 0 0.0.0.0:40001 0.0.0.0:* LISTEN 10602/postgres
[sdw2] tcp 0 0 172.28.64.190:41000 0.0.0.0:* LISTEN 10636/postgres
[sdw2] tcp 0 0 172.28.64.190:41001 0.0.0.0:* LISTEN 10641/postgres
[sdw2] tcp 0 0 0.0.0.0:50000 0.0.0.0:* LISTEN 10603/postgres
[sdw2] tcp 0 0 0.0.0.0:50001 0.0.0.0:* LISTEN 10600/postgres
[sdw2] tcp 0 0 172.28.64.190:51000 0.0.0.0:* LISTEN 10621/postgres
[sdw2] tcp 0 0 172.28.64.190:51001 0.0.0.0:* LISTEN 10620/postgres
[sdw2] tcp 0 0 :::40000 :::* LISTEN 10601/postgres
[sdw2] tcp 0 0 :::40001 :::* LISTEN 10602/postgres
[sdw2] tcp 0 0 :::50000 :::* LISTEN 10603/postgres
[sdw2] tcp 0 0 :::50001 :::* LISTEN 10600/postgres
[smdw] (Not all processes could be identified, non-owned process info
[smdw] will not be shown, you would have to be root to see it all.)
[smdw] tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 8501/postgres
[smdw] tcp 0 0 :::5432 :::* LISTEN 8501/postgres
[ mdw] (Not all processes could be identified, non-owned process info
[ mdw] will not be shown, you would have to be root to see it all.)
[ mdw] tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 17489/postgres
[ mdw] tcp 0 0 :::5285 :::* LISTEN 17496/postgres
[ mdw] tcp 0 0 :::5432 :::* LISTEN 17489/postgres
[sdw1] (Not all processes could be identified, non-owned process info
[sdw1] will not be shown, you would have to be root to see it all.)
[sdw1] tcp 0 0 0.0.0.0:40000 0.0.0.0:* LISTEN 10662/postgres
[sdw1] tcp 0 0 0.0.0.0:40001 0.0.0.0:* LISTEN 10660/postgres
[sdw1] tcp 0 0 172.28.56.68:41000 0.0.0.0:* LISTEN 10685/postgres
[sdw1] tcp 0 0 172.28.56.68:41001 0.0.0.0:* LISTEN 10684/postgres
[sdw1] tcp 0 0 0.0.0.0:50000 0.0.0.0:* LISTEN 10661/postgres
[sdw1] tcp 0 0 0.0.0.0:50001 0.0.0.0:* LISTEN 10663/postgres
[sdw1] tcp 0 0 172.28.56.68:51000 0.0.0.0:* LISTEN 10693/postgres
[sdw1] tcp 0 0 172.28.56.68:51001 0.0.0.0:* LISTEN 10694/postgres
[sdw1] tcp 0 0 :::40000 :::* LISTEN 10662/postgres
[sdw1] tcp 0 0 :::40001 :::* LISTEN 10660/postgres
[sdw1] tcp 0 0 :::50000 :::* LISTEN 10661/postgres
[sdw1] tcp 0 0 :::50001 :::* LISTEN 10663/postgres
=> 九、安装PerformanceMonitor数据收集Agent
source /usr/local/greenplum-db/greenplum_path.sh
gpperfmon_install --enable --password 123456 --port 5432
gpstop -r #重启GP生效查看进程
ps -ef |grep gpmmon
gpadmin 17498 17489 0 Apr04 ? 00:01:04 /usr/local/greenplum-db-4.3.12.0/bin/gpmmon -D /data/gpmaster/gpseg-1/gpperfmon/conf/gpperfmon.conf -p 5432查看端口
netstat -nltp | grep gp
tcp 0 0 0.0.0.0:28080 0.0.0.0:* LISTEN 11984/gpmonws
tcp 0 0 :::8888 :::* LISTEN 17611/gpsmon 查看监控数据是否写入数据库
psql -d gpperfmon -c 'select * from system_now;'
ctime | hostname | mem_total | mem_used | mem_actual_used | mem_actual_free | swap_total | swap_used | swap_page_in | swap_page_out | c
pu_user | cpu_sys | cpu_idle | load0 | load1 | load2 | quantum | disk_ro_rate | disk_wo_rate | disk_rb_rate | disk_wb_rate | net_rp_rate | net_wp_rate | n
et_rb_rate | net_wb_rate
---------------------+----------+-------------+------------+-----------------+-----------------+------------+-----------+--------------+---------------+--
--------+---------+----------+-------+-------+-------+---------+--------------+--------------+--------------+--------------+-------------+-------------+--
-----------+-------------
2018-04-06 14:59:15 | dw01 | 25132879872 | 1774948352 | 391598080 | 24741281792 | 536866816 | 0 | 0 | 0 |
0.22 | 0.34 | 99.44 | 0.02 | 0.04 | 0 | 15 | 0 | 3 | 0 | 47850 | 40 | 43 |
9267 | 21458
2018-04-06 14:59:15 | dw02 | 8188444672 | 963731456 | 157970432 | 8030474240 | 536866816 | 0 | 0 | 0 |
0.13 | 0.32 | 99.55 | 0.05 | 0.03 | 0 | 15 | 0 | 2 | 0 | 9788 | 3 | 3 |
331 | 572
2018-04-06 14:59:15 | dw03 | 8188444672 | 2338099200 | 239792128 | 7948652544 | 536866816 | 0 | 0 | 0 |
0.28 | 0.42 | 99.27 | 0 | 0 | 0 | 15 | 0 | 9 | 0 | 178355 | 169 | 66 |
174387 | 154813
2018-04-06 14:59:15 | dw04 | 8188444672 | 2338926592 | 242188288 | 7946256384 | 536866816 | 0 | 0 | 0 |
0.28 | 0.61 | 99.07 | 0 | 0 | 0 | 15 | 0 | 8 | 0 | 175088 | 165 | 65 |
167569 | 162326
(4 rows)至此GreenPlum4.3.12集群安装完毕·
Tag标签:「GreenPlum ucloud 部署」更新时间:「2022-08-01 09:01:26」阅读次数:「844」