iSCSI/OCFS2 Maintenance Plan
Goal: Upgrade the iSCSI drives with bigger drives.
Event: https://blazingraptor.com/iscsi-maintenance-plan.html
Slack Channel: #iscsi-maintenance-plan-for-private-customer
Related Tickets: PRIVATE
Plan Ownership: @bcorson
The Problem: web03.PRIVATE.com & web04.PRIVATE.com are running out of space. You can see the issue here
[root@web03 ~]# df -h Filesystem Size Used Avail Use% Mounted on (truncated output) /dev/sdd1 2.6T 2.4T 151G 95% /var/www/vhosts
Unlike a lot of our other setups, that use the NATAPP SAN, this setup has it's own dedicated SAN. This is a product that we no longer offer. You can see the SAN on iscsi01.PRIVATE.com & iscsi02.PRIVATE.com
[Pacemaker: online | DRBD: Primary Connected]
[root@iscsi01.PRIVATE.com ~]
[20:19:23]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
(truncated output)
| |-vg_shared-pool00-tpool 253:3 0 2.6T 0 lvm
Notice how both show the size being 2.6 TB? And notice it's 95% full. Once it gets to 100% full, things will start crashing.
We need to upgrade the drives in the backing servers (iscsi01.PRIVATE.com / iscsi02.PRIVATE.com) and resize the SAN on iscsi01.PRIVATE.com / iscsi02.PRIVATE.com.
Notes:
6-8hr depends on how long the DRDB data sync takes during the iSCSI Upgrade.
Longer event windows means fewer nights of maintenance.
Fewer hours of Degraded High-Availability Services (No Automatic Failovers)
Summary:
Date 1 - We are making a backup of the website vhost data just in case anything goes wrong
Date 2: - We are replacing and resizing the drives.
Signoff Tasks - Required before plan can be executed
Customer:
[ ] Plan Approval: Plan - 6-8hr Maintenance Event Windows (2 Nights)
[ ] Approved time slot: 11pm
[ ] Approved date slots: 11-24-2024
#linux-sysadmins:
[ ] Resize the SAN:
#dcops:
[ ] Replace the drives
Who is doing what?
@DCOPS - Will be on hand to swap drives at the appropriate time slots during the Day 2 iSCSI Drive Upgrades. They will be available for 1hr starting at Midnight of Day 2 and 1 hr at 4AM. They will be swapping drivees only, no data copy, no raid setup. ESGs will handle the entirety of the configuration of the new drives and copying of the DRDB data . Each server runs a 4 x 1.92TB SW RAID10 upgrading to 4 x 3.8 TB. The SW RAID is md100 and existing drives are: sda sdb sdc sdd
@Linux-Sysadmins - The detailed instructions are below for the disk capacity upgrade of the HA iSCSI cluster and follow-up webserver (OCFS2 Cluster) maintenance related to this event. We have to upgrade the HA iSCSI cluster before we can upgrade the OCFS2 cluster. Due to the time constraints with syncing DRBD data on iscsi01/02, we may need to split the OCFS2 cluster upgrade into a separate maintenance event.
NOTE: We don't use IPMI fencing on these so don't worry about that when you see it in the pcs status
NOTE: We don't use zbindfs on these.
References:
- Old Event 2022 (from last time, we are basing this plan on that old one): PRIVATE
- Waypoint: PRIVATE
- Waypoint: PRIVATE
- Waypoint: PRIVATE
- Planning Slack Channel: #old-iscsi-maintenance-plan-for-private-customer
- Old Planning Slack Channel 2022: #old-iscsi-maintenance-plan-for-private-customer
Event Maintenance Plan
Day 1 - OCFS2 Data Backup
Required for Day 2 Readiness - Run on Web03 ONLY
A. We must run an initial rsync backup and schedule a follow-up "catchup" rsync for several hours prior to "Day 2b- HA-iSCSI Cluster Drive Upgrades"
B. Execute Initial Backup
Before you begin, make sure we have space for the backup. If you see below, We're using 2.4 TB of disk space on the /var/www/vhosts. That means we need at least 2.4 TB of free disk space on /drive3. We were supposed to have upgraded drive3 before we started this maintenance.
[root@web03 ~]# df -h | grep -E '(Filesystem|drive3|vhosts)' Filesystem Size Used Avail Use% Mounted on /dev/sdc1 1.8T 1.2T 595G 66% /drive3 /dev/sdd1 2.6T 2.4T 151G 95% /var/www/vhosts
Also, if you look in drive3, you'll already see a really old backup from 2022, this can be deleted
[root@web03 ~]# stat /drive3/OCFS2_Backup/vhosts;ls /drive3/OCFS2_Backup/vhosts File: ‘/drive3/OCFS2_Backup/vhosts’ Size: 4096 Blocks: 8 IO Block: 4096 directory Device: 821h/2081d Inode: 57671682 Links: 14 Access: (2755/drwxr-sr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-11-13 05:53:32.495466126 -0500 Modify: 2022-12-05 04:50:34.283013778 -0500 Change: 2022-12-05 19:55:44.993560492 -0500 Birth: - new www.ipsumlorem.com www.parturientmontes.com test www.consectetur.com www.sitametconsectetur.com test.bak.txt www.nullamidolor.com www.ridiculusmus.com tools www.euleo.com www.metusac.com www.seddoeum.com www.adipiscingelit.org www.ametconsectetur.com
Go ahead and make a new backup
time rsync -aH --info=progress2 --no-inc-recursive --log-file=/root/rsync-vhosts.txt /var/www/vhosts/ /drive3/OCFS2_Backup/vhosts/
[ ] Initial Backup Started on web03
[ ] Initial Backup Completed on web03
Finished Day 1 - OCFS2 Data Backup
Day 2a - Catchup OCFS2 Data Backup
A. Execute Catchup Backup - several hours prior to "Day 2b- HA-iSCSI Cluster Drive Upgrades"
time rsync -aH --info=progress2 --no-inc-recursive --log-file=/root/rsync-vhosts-catchup.txt /var/www/vhosts/ /drive3/OCFS2_Backup/vhosts/
[ ] Initial Backup Started on web03
[ ] Initial Backup Completed on web03
Day 2b - HA-iSCSI Cluster Drive Upgrades
Step 1. Put The iSCSI HOST into standby for maintenance
Note: An Important Distinction - Please Upgrade iscsi01 First
The plan as written expects to upgrade the drives on iscsi01 FIRST. To avoid problems down the plan, make sure we start from the state that iscsi01 is secondary. Start with a failover if necessary to make iscsi01 the secondary node (if it is not already). This first section will guide you through failover if it is needed.
On iscsi01.PRIVATE.com only (secondary)
Very Important
DO NOT PROCEED unless DRDB STATUS shows "UpToDate" for BOTH SERVERS
A. The command prompt will identify as primary or secondary OR you can check the DRDB status by running this on both nodes:
hostname; drbdadm status
Example Output shows iscsi01 as the secondary node:
iscsi01.PRIVATE.com
shared role:Secondary
disk:UpToDate
peer role:Primary
replication:Established peer-disk:UpToDate
- [if iscsi01 is SECONDARY] jump down to 1.B below
- [else if iscsi01 is PRIMARY] we need to fail it over so it becomes secondary (promoting iscsi02 to primary). If you are not sure how to do that see https://waypoint.PRIVATE.com/link as we use pcs cluster standby $NODE and pcs cluster unstandby $NODE
DO NOT PROCEED unless is iscsi01.PRIVATE.com secondary
B. Put the host in standby
pcs cluster standby iscsi01.PRIVATE.com
C. Make sure no services are running on the secondary server
pcs status
- We should make sure that there are no failed resources
- We want to make sure things are clean before making changes
D. Additionally you should see the command line prompt will now show the following to reflect it is in standby
[Pacemaker: standby | DRBD: disconnected]
E. Confirm/Disable DRBD so it doesn't try to start on boot (should already be but good to check)
systemctl disable drbd
F. Shutdown the server
poweroff
G. Let #DCOPS know they can proceed with replacing the drives on iscsi01 with new ones.
They should not be partitioned, or added into the SW RAID, and they should NOT copy any data from the old drives. ESGs will handle partitioning and raiding the new drives.
Step 2. Once DCOPS is done and iscsi01 is back online.
Continuing still on iscsi01.PRIVATE.com only (secondary)
A. Perform a server reboot on iscsi01 prior to running through the next steps. This is to check that the wrong drives have not been wiped/altered.
B. Check the new drives to verify the names. Should be: sda, sdb, sdc, sdd based on the other server but you never know so it's always good to double-check the names. We also need to double-check that the drives are unpartitioned.
lsblk --output NAME,SIZE,TYPE,FSTYPE,LABEL|column -t
Note: If lsblk says they are partitioned you will need to run wipefs on the relevant 3.8TB drives before proceeding.
lwipefs --all /dev/sdX
Reboot to make sure everything is still good after the wipe
reboot
C. Repeat this for each of the drives that were replaced.
Replace $DRIVE with /dev/sda
Replace $LABEL with a unique identifier ($HOSTNAME:100 works and is what is used on the old drives.)
parted $DRIVE mklabel gpt parted $DRIVE mkpart $LABEL 2048s "80%"
THE COMMAND BELOW WILL GENERATE YOUR PARTED/MDADM COMMANDS IN THE CORRECT ORDER (It will only print commands not execute them)
For DISKS if your drives are not sequential ie. {a..d}, you can specify them in a CSV list instead: eg. {e,g,s,t}
LABEL=$HOSTNAME:100
DISKS=$(echo /dev/sd{a..d})
unset PARTS; for DRIVE in $DISKS; do echo; echo "#Partition $DRIVE"; echo parted $DRIVE mklabel gpt; echo parted $DRIVE mkpart $LABEL 2048s \"80%\"; PARTS="$PARTS ${DRIVE}1";
done; echo; echo "#Create the new MD100 SW RAID"; echo "mdadm --create /dev/md100 --bitmap=internal --metadata=1.2 --level 10 --raid-disks 4 $PARTS"; echo
Below is an example mdadm command in case the one generated above somehow fails. The key difference is this command versus the others is that the specification of partition 1 on each drive.
mdadm --create /dev/md100 --bitmap=internal --metadata=1.2 --level 10 --raid-disks 4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
D. Create the new DRBD volume
drbdadm create-md shared/0
E. Take iscsi01 out of standby
pcs cluster unstandby iscsi01
F. (OPTIONAL) If the DRBD resync is very slow. You can speed up the DRBD sync by running the following:
drbdadm disk-options --c-plan-ahead=0 --resync-rate=1000M shared drbdadm net-options --max-epoch-size=8000 --max-buffers=8000 shared
G. At this point we just have to wait until DRBD has finished syncing. Watch it with either of these commands:
watch -n 5 cat /proc/drbd
OR
watch -n 5 'drbdadm status'
H. Once DRBD has finished syncing up move to Step 3.A.
Step 3. DRBD finished syncing and ready to failover.
A. Let the customer know we are ready to failover in the ticket
B. Update DCOPS of the same once we are close to the maintenance time.
C. Once we are ready to failover from iscsi02.PRIVATE.com to iscsi01.PRIVATE.com move on to #4
Step 4. Make iscsi02.PRIVATE.com secondary and shut down
(On iscs02.PRIVATE.com only)
A. Verify DRBD is in sync before doing anything else. Watch it with either of these commands:
watch -n 5 cat /proc/drbd
OR
watch -n 5 'drbdadm status'
If DRBD is showing UpToDate then proceed with part B below
B. Put the host in standby
pcs cluster standby iscsi02.PRIVATE.com
Wait for it to failover to iscsi01 and make sure no services are running on the now secondary server iscsi02
pcs status
C. Additionally you should see the command line prompt will now show the following to reflect it is in standby
[Pacemaker: standby | DRBD: disconnected]
D. Disable DRBD so it doesn't try to start on boot (should already be but good to check)
systemctl disable drbd
E. Shutdown the server
poweroff
F. Let DCOPS know they can proceed with replacing the drives on iscsi02 with new ones. They should not be partitioned or added into the software RAID.
Step 5. Restart iSCSI Connections - If they fail to reconnect after the iSCSI host failover. Otherwise skip this section.
Should not need to stop anything for that to work. Iscsi will replay any paused commands once the fail over happens. It is OK to have it there to show what to do if there are issues, but should not be needed.
(web03.PRIVATE.com and web04.PRIVATE.com only)
A. Most likely we just need to do the following.
We will resize the OCFS2 volume and clean up the derelict hosts later in this plan here: Day 2 - MFR:OCFS2 Cluster Upgrade."
iscsiadm -m node iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts --logout iscsiadm -m node iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts --login
In the event the logout/login method above doesn't work - you may also need to do the following otherwise your done with this section.
B. Let OCC know that we are doing maintenance on these
web03.PRIVATE.com and web04.PRIVATE.com
C. On BOTH web03 and web04 do the following
pm2 stop sockets systemctl stop httpd
Check for any other processes using the OCFS2 mount
lsof |grep -i vhosts
* If there are additional processes then you will need to kill those process IDs
Otherwise proceed with the following
umount /var/www/vhosts systemctl stop ocfs2 systemctl restart iscsi
D. Verify the iSCSI device is available
iscsiadm -m session
It should show:
tcp: [1] 192.168.50.100:3260,1 iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts (non-flash)
If it does not, then you will need to log in to iSCSI
iscsiadm -m discovery -t sendtargets -p 192.168.50.100
It should show:
192.168.50.100:3260,1 iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts
Then login to the iSCSI device:
iscsiadm -m node -p 192.168.50.100 -T iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts --login
E. Restart OCFS2, remount the SAN, and start Apache
(On BOTH web03 and web04)
systemctl start ocfs2 mount /var/www/vhosts systemctl start httpd systemctl start pm2
F. Verify PRIVATE.com loads
Step 6 - When DCOPS is done & the server iscsi02 is back online
Continue with iscsi02.PRIVATE.com only (secondary)
A. Check the new drives to verify the names. Should be: sda, sdb, sdc, sdd based on the other server but you never know, so it's always good to double-check the names. We also need to double-check that the drives are unpartitioned.
lsblk --output NAME,SIZE,TYPE,FSTYPE,LABEL|column -t
We will resize the OCFS2 volume and clean up the derelict hosts later in this plan here: Day 2 - MFR:OCFS2 Cluster Upgrade."
B. Repeat this for each of the drives that were replaced.
Replace $DRIVE with /dev/sda
Replace $LABEL with a unique identifier ($HOSTNAME:100 works and is what is used on the old drives.)
parted $DRIVE mklabel gpt parted $DRIVE mkpart $LABEL 2048s "80%"
THE COMMAND BELOW WILL GENERATE YOUR PARTED/MDADM COMMANDS IN THE CORRECT ORDER (It will only print commands not execute them)
For DISKS if your drives are not sequential ie. {a..d}, you can specify them in a CSV list instead: eg. {e,g,s,t}
LABEL=$HOSTNAME:100
DISKS=$(echo /dev/sd{a..d})
unset PARTS; for DRIVE in $DISKS; do echo; echo "#Partition $DRIVE"; echo parted $DRIVE mklabel gpt; echo parted $DRIVE mkpart $LABEL 2048s \"80%\"; PARTS="$PARTS ${DRIVE}1";
done; echo; echo "#Create the new MD100 SW RAID"; echo "mdadm --create /dev/md100 --bitmap=internal --metadata=1.2 --level 10 --raid-disks 4 $PARTS"; echo
Below is an example mdadm command in case the one generated above somehow fails. The key difference is this command versus the others is that the specification of partition 1 on each drive.
mdadm --create /dev/md100 --bitmap=internal --metadata=1.2 --level 10 --raid-disks 4 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
C. Create the new DRBD volume
drbdadm create-md shared/0
D. Take iscsi02 out of standby
pcs cluster unstandby iscsi02
E. Wait until DRBD has finished syncing. Monitor with:
watch -n 5 cat /proc/drbd
OR
watch -n 5 'drbdadm status'
F. (Optional) If the DRBD resync is very slow. You can speed up the DRBD sync by running the following:
drbdadm disk-options --c-plan-ahead=0 --resync-rate=1000M shared drbdadm net-options --max-epoch-size=8000 --max-buffers=8000 shared
G. Move to Step 7.A
Step 7 - Once DRBD has finished syncing up again. On the Primary server only (iscsi01.PRIVATE.com):
A. Verify the disk space drives prior to resizing
lsblk --output NAME,SIZE,TYPE,FSTYPE,LABEL|column -t
B. Resize DRBD
pvresize /dev/drbd0
C. Resize the LVM filesystem leaving 20% free for snapshot space
lvextend -l +80%FREE vg_shared/pool00
D. Wait until DRBD has finished syncing (Yes again). Watch it with either of these commands:
watch -n 5 cat /proc/drbd
OR
watch -n 5 'drbdadm status'
E. Resize LUN if needed
1) Grab targets
tgm list
Example
[Pacemaker: online | DRBD: Primary Connected] [root@iscsi01.PRIVATE.com ~] [01:33:09]# tgm list iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts
2) Verify size of LUN
tgm iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts luns list
Example
[Pacemaker: online | DRBD: Primary Connected] [root@iscsi01.PRIVATE.com ~] [01:33:20]# tgm iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts luns list 0 /dev/vg_shared/vhosts_lun0 2.53 TB
3) If the LUN size does not match, resize
4) First get the size
tgm summary
Example
[root@iscsi01.PRIVATE.com ~] [09:11:04]# tgm summary Connected sessions: 2 Total storage: 2.59 TB * Used: 2.53 TB (97.94%) * Allocated: 2.53 TB (97.00%) Discovery auth: * username: PRIVATE * password: MASKED iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts * Size: 2.53 TB * ACLs: 4 - username: pha7peel9fie - password: MASKED
5) Set the size
tgm iqn.1997-08.com.PRIVATE.iscsi01-02:vhosts luns resize ???t
Example
[Pacemaker: online | DRBD: Primary Connected] [root@iscsi01.PRIVATE.com ~] tgm iqn.2014-01.es.enteng:training:example2 luns resize 2.53t Resized LUN 0 to 70g
F. If you need to speed that up again. You can speed up the DRBD sync by running the following:
drbdadm disk-options --c-plan-ahead=0 --resync-rate=500M shared drbdadm net-options --max-epoch-size=8000 --max-buffers=8000 shared
G. Make sure that the SAN is larger. You shouldn't see 95% disk usage anymore.
Both web03.PRIVATE.com & web04.PRIVATE.com should show less than 95% usage.
[root@web03 ~]# df -h Filesystem Size Used Avail Use% Mounted on (truncated output) /dev/sdd1 2.6T 2.4T 151G 95% /var/www/vhosts
port upgrade moratorium (the stuff below was missing from the event/plan. I'm adding it as part of a port moratorium) (work in progress):
Step 8. Restart connections
A. Do the "Step 5. Restart iSCSI Connections" section
Step 9 - Stop the connections
A. Stop Apache (on both nodes) (if updatedb is running you may want to stop that too)
systemctl stop httpd killall updatedb
B. Kill all processes (on both nodes) (the PIDs will be different than what's below)
[root@web03 ~]# lsof |grep -i vhosts
perl 5149 root 3rW REG 8,49 27973 106086986 /var/www/vhosts/www.PRIVATE.com/cgi-bin/realZingers/functions/cronJobs/mailSystem/fetch_a_match.pl
master_cr 7794 root 255r REG 8,49 306 1356681 /var/www/vhosts/www.PRIVATE.com/cgi-bin/realZingers/functions/cronJobs/master_cron
[root@web03 ~]# umount /var/www/vhosts
umount: /var/www/vhosts: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
[root@web03 ~]# kill -9 5149 7794
[root@web03 ~]# umount /var/www/vhosts
[root@web03 ~]#
This part is tricky, I had to at one point restart step 9 all over again and reboot both servers. I also used a multi-line command/loop to more easily disconnect connections. Without doing this, I was unable to stop ocfs2 and/or unmount..
for i in `lsof |grep -i vhosts | awk '{print $2}'`
do
kill -9 $i
done
umount /var/www/vhosts
C. Stop ocfs2 (on both nodes)
systemctl stop ocfs2
Step 10 - Recreate the mounts
You should only need to do this on one of the web nodes, let's go with web03.
A. Expand the disk
[root@web03 ~]# /root/bin/expand-partition.sh /dev/sdd Error: The backup GPT table is not at the end of the disk, as it should be. This might mean that another operating system believes the disk is smaller. Fix, by moving the backup to the end (and removing the old backup)? Warning: Not all of the space available to /dev/sdd appears to be used, you can fix the GPT to use all of the space (an extra 3137339392 blocks) or continue with the current setting?
B. You'll still see the wrong size
root@web03 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT (truncated output) sdd 8:48 0 2.6T 0 disk |-sdd1 8:49 0 1.3T 0 part /var/www/vhosts [root@web03 ~]# gdisk -l /dev/sdd Number Start (sector) End (sector) Size Code Name 1 2048 2726297566 1.3 TiB 0700 Mic
C. Use gdisk to delete the disk (we are not writing changes to disk yet, so don't be scared) (we are removing the old disk mount and making a new bigger mount. We aren't actually destroying any data, that data is on the SAN on a different server)
[root@web03 ~]# gdisk /dev/sdd GPT fdisk (gdisk) version 0.8.10 Partition table scan: MBR: protective BSD: not present APM: not present GPT: present Found valid GPT with protective MBR; using GPT. Command (? for help): p Disk /dev/sdd: 8589934592 sectors, 4.0 TiB Logical sector size: 512 bytes Disk identifier (GUID): 578F387D-AAA0-4DB0-AAF3-AFC06169B129 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 8589934558 Partitions will be aligned on 2048-sector boundaries Total free space is 3137341406 sectors (1.5 TiB) Number Start (sector) End (sector) Size Code Name 1 2048 5452595166 2.5 TiB 0700 Microsoft basic data Command (? for help): d Using 1
You should see it gone (we have not written changes to disk yet, so don't be scared)
Command (? for help): p Disk /dev/sdd: 8589934592 sectors, 4.0 TiB Logical sector size: 512 bytes Disk identifier (GUID): 578F387D-AAA0-4DB0-AAF3-AFC06169B129 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 8589934558 Partitions will be aligned on 2048-sector boundaries Total free space is 8589934525 sectors (4.0 TiB) Number Start (sector) End (sector) Size Code Name
D. Recreate the disk (we have not written changes to disk yet, so don't be scared)
Command (? for help): n
Partition number (1-128, default 1):
First sector (34-8589934558, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-8589934558, default = 8589934558) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 0700
Changed type of partition to 'Microsoft basic data'
Command (? for help): p
Disk /dev/sdd: 8589934592 sectors, 4.0 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): 578F387D-AAA0-4DB0-AAF3-AFC06169B129
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 8589934558
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 8589934558 4.0 TiB 0700 Microsoft basic data
You should see it gone (we have not written changes to disk yet, so don't be scared)
Below is an example where it shows the wrong disk size
Command (? for help): p Disk /dev/sdd: 8589934592 sectors, 4.0 TiB Logical sector size: 512 bytes Disk identifier (GUID): 578F387D-AAA0-4DB0-AAF3-AFC06169B129 Partition table holds up to 128 entries First usable sector is 34, last usable sector is 8589934558 Partitions will be aligned on 2048-sector boundaries Total free space is 2014 sectors (1007.0 KiB) Number Start (sector) End (sector) Size Code Name 1 2048 8589934558 2.5 TiB 0700 Microsoft basic data
Also, you may need to reboot both web nodes and may need to redo step 9 to make sure nothing's using the SAN and everything is disconnected. Otherwise, that can cause issues for resizing, deleting and recreating the disk. The main issue being that you won't see the larger size despite the SAN offering a bigger size. I ran into that very issue.
If you truly get stuck, please refer to ticket PRIVATE and see what we did to get this working.
E. Write the change to disk (be careful of this, now we are making changes)
Command (? for help): w Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING PARTITIONS!! Do you want to proceed? (Y/N): y OK; writing new GUID partition table (GPT) to /dev/sdd. Warning: The kernel is still using the old partition table. The new table will be used at the next reboot. The operation has completed successfully.
F. We need to partprobe to pick up the changes
[root@web03 ~]# partprobe [root@web03 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdd 8:48 0 4T 0 disk |-sdd1 8:49 0 4T 0 part /var/www/vhosts sdb 8:16 0 477G 0 disk |-sdb4 8:20 0 468G 0 part |-md1 9:1 0 467.8G 0 raid1 / |-sdb2 8:18 0 8G 0 part [SWAP] |-sdb3 8:19 0 1G 0 part |-md0 9:0 0 1022M 0 raid1 /boot |-sdb1 8:17 0 1M 0 part sdc 8:32 0 5.5T 0 disk |-sdc1 8:33 0 5.5T 0 part /drive3 sda 8:0 0 477G 0 disk |-sda4 8:4 0 468G 0 part |-md1 9:1 0 467.8G 0 raid1 / |-sda2 8:2 0 8G 0 part [SWAP] |-sda3 8:3 0 1G 0 part |-md0 9:0 0 1022M 0 raid1 /boot |-sda1 8:1 0 1M 0 part
G. We need to resize so the vhost shows the changes too
[root@web03 ~]# tunefs.ocfs2 -S /dev/sdd1 [root@web03 ~]# systemctl start ocfs2 [root@web03 ~]# df -hT /dev/sdd1 Filesystem Type Size Used Avail Use% Mounted on /dev/sdd1 ocfs2 4.0T 2.5T 1.6T 61% /var/www/vhosts
Both nodes should show the correct size. I ran partprobe just to be sure on the other node too.
[root@web04 ~]# partprobe [root@web04 ~]# systemctl start ocfs2 [root@web04 ~]# df -hT /dev/sdd1 Filesystem Type Size Used Avail Use% Mounted on /dev/sdd1 ocfs2 4.0T 2.5T 1.6T 61% /var/www/vhosts
H. Restart services on both nodes
[root@web03 ~]# systemctl start httpd [root@web03 ~]# pm2 start sockets [PM2] Spawning PM2 daemon with pm2_home=/root/.pm2 [PM2] PM2 Successfully daemonized [PM2][ERROR] Script not found: /root/sockets
[root@web04 ~]# systemctl start httpd [root@web04 ~]# pm2 start sockets [PM2] Spawning PM2 daemon with pm2_home=/root/.pm2 [PM2] PM2 Successfully daemonized [PM2][ERROR] Script not found: /root/sockets