Ubuntu to CentOS rsync errors with chgrp Operation not permitted (1)

When running rsync to backup from a ZFS volume on my primary Ubuntu server to a backup server running Centos I was getting non-fatal errors like this:

rsync -avr –delete –progress /DISK2/vmshare/ /mnt/share4/vmshare/

The key flag in that commandline is -a
That switch enables archive mode which, among other things, preserves the permissions on files and directories.
I say that the errors were non-fatal because the files and directories would copy over and I could access them via SMB or locally from multiple machines without issue.
Clearly then the issue occurs post-write, during the modify permissions step.
This was strange because I am definitely the owner of the data.

Since the error message mentions chgrp that’s a good place to start.
I logged into the source server (Ubuntu) and executed (bold added for emphasis):

luis@osiris:~$ id
uid=1000(luis) gid=1000(luis) groups=1000(luis),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),107(lpadmin),124(sambashare)

[luis@chronos2 ~]$ id
uid=1000(luis) gid=500(luis) groups=500(luis),10(wheel) context=unconfined_u:unc1023

My User ID (UID) matches fine but the Group ID (GID) does not match.

The solution is simply to get them to match up. Since most of my Servers are running Ubuntu it made sense to change the GID on the destination CentOS machine:
First let’s elevate the shell:

[luis@chronos2 ~]$ sudo -s

Then we can change the GID:

[root@chronos2 luis]# groupmod -g 1000 luis

Now if we can check to see if that’s taken effect:

[root@chronos2 luis]# id luis
uid=1000(luis) gid=1000(luis) groups=1000(luis),10(wheel)

Perfect.
But now I notice that the errors still appear. I realise that this is because the files and folders that exist on the destination are still owned by the wrong/old GID of 500.
So I need to change that on the destination server:

[root@chronos2 luis]# chown -R luis:luis /DISK2_BACKUP/

Finally there are no more chgrp errors when I run my rsync backup job.

Note that if you had been running rsync as root then you’d probably have not seen the issue because the problem is not with what permissions get set (you don’t want to change those otherwise you wouldn’t be using the -a switch in rsync in the first place); but rather the issue is with what account has the mojo to set those permissions.
In the user-level scenario above, the user executing the rsync (with GID 1000) does not have permissions to set the group permissions on the destination because the owner of the destination data is GID 500.
In the root scenario, root is typically UID =0 and GID = 0. This is the case on both Ubuntu and CentOS and so you would not hit the issue.