UnixPedia : HPUX / LINUX / SOLARIS: What is the maximum number of groups (GIDs) a user can belong to when using NFS with AUTH_UNIX / AUTH_SYS on RHEL

Saturday, November 10, 2018

What is the maximum number of groups (GIDs) a user can belong to when using NFS with AUTH_UNIX / AUTH_SYS on RHEL


What is the maximum number of groups (GIDs) a user can belong to when using NFS with AUTH_UNIX / AUTH_SYS on RHEL


Environment

  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • NFS

Issue

  • GIDs of users in more than 16 groups are not recognized properly on NFS.
  • I can not change ownership of subdirectory of nfs filesystem, following is the error messages:
Raw
$ chown USER:GROUP /nfs-mount-point/file
chown: changing ownership of `/nfs-mount-point/file': Operation not permitted
  • User getting "Permission Denied" error while creating a file on NFS share.
Raw
# su - testuser1
$ touch example
touch: cannot touch `example`: Permission denied
  • What limits are in place for group settings on NFS ?

Resolution

  • If the NFS environment requires a user to belong to more than 16 groups then use RPCSEC_GSS (e.g. with Kerberos) instead of AUTH_UNIX / AUTH_SYS. How to configure NFSv4 with kerberos authentication?
  • On RHEL 6 and newer the NFS-server can be instructed to discard the groups given by the NFS-client. The --manage-gids option for rpc.mountd (see man rpc.mountd) needs to be set on the NFS-server in /etc/sysconfig/nfs. The flag tells the server to ignore the 16 groups sent by the client and resolve group membership locally (NFS-server side). This effectively bypasses the limit imposed by the RPC data structure and requires that the NFS-server see either the same or a superset of the groups available to the NFS-client. Note that AUTH_SYSwith --manage-gids is less secure than switching to RPCSEC_GSS. If RPCSEC_GSS is an option for your environment, it is a better solution.
  • NOTE: If you intend to do file locking over NFS, there maybe a limitation in NFSv3's file locking protocol (NLM) where it is unable to use RPCSEC_GSS. In that situation one should use NFSv4 with RPCSEC_GSS (e.g. with Kerberos).

Root Cause

  • NFS uses the RPC protocol to authenticate users.
  • The RPC protocol's AUTH_UNIX / AUTH_SYS Credentials structure limits the number of groups to 16, as specified in RFC 5531.

         struct authsys_parms {
        unsigned int stamp;
        string machinename<255>;
        unsigned int uid;
        unsigned int gid;
        unsigned int gids<16>;
     }
  • NFS uses the AUTH_SYS protocol by default.

Diagnostic Steps

  1. Capture a tcpdump.
  2. Inspect the the RPC request (e.g. SETATTR Call).

    Credentials
    Flavor: AUTH_UNIX (1)
    Length: 92
    Stamp: STAMP
    Machine Name: NAME
        length: 7
        contents: NAME
        fill bytes: opaque data
    UID: 901
    GID: 901
    Auxiliary GIDs (16) [901, 951, 952, 953, 954, 955, 956, 957, 958, 959, 960, 961, 962, 963, 964, 965]
        GID: 901
        GID: 951
        GID: 952
        GID: 953
        GID: 954
        GID: 955
        GID: 956
        GID: 957
        GID: 958
        GID: 959
        GID: 960
        GID: 961
        GID: 962
        GID: 963
        GID: 964
        GID: 965
Verifier
    Flavor: AUTH_NULL (0)
    Length: 0
  3. For the SETATTR example, note that the GID the file is being set to, 982, does not appear in the credentials above.

    Network File System, SETATTR Call FH:0x2713c62b
[Program Version: 3]
[V3 Procedure: SETATTR (2)]
object
    length: 32
    [hash (CRC-32): 0x2713c62b]
    decode type as: unknown
    filehandle: 000000000000fe1b0009bdae6802ffffffffffffffffffff...
new_attributes
    mode: no value
        set_it: no value (0)
    uid: value follows
        set_it: value follows (1)
        uid: 901
    gid: value follows
        set_it: value follows (1)
        gid: 982
    size: no value
        set_it: no value (0)
    atime: don't change
        set_it: don't change (0)
    mtime: don't change
        set_it: don't change (0)
guard: no value
    check: no value (0)
Based on the second resolution provided i.e. to discard the groups given by the NFS client.
Here are the steps to reproduce it.
  • Created a user testuser1 with same uid and gid on NFS server and client.
  • Created 20 groups with same gid on NFS server and client
  • Made user testuser1 members of these 20 groups
  • Created 20 directories and gave them respective ownerships
On NFS server :

[root@server-test test1]# id -a testuser1
uid=20362(testuser1) gid=20362(testuser1) groups=20362(testuser1),20363(group1),20364(group2),20365(group3),20366(group4),20367(group5),20368(group6),20369(group7),20370(group8),20371(group9),20372(group10),20373(group11),20374(group12),20375(group13),20376(group14),20377(group15),20378(group16),20379(group17),20380(group18),20381(group19),20382(group20)
On client :

server-test:/test1 /testnfs nfs rw,relatime,vers=3,rsize=16384,wsize=16384,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.65.210.132,mountvers=3,mountport=43813,mountproto=udp,local_lock=none,addr=10.65.210.132 0 0

[root@client-test testnfs]# usermod -G group1,group2,group3,group4,group5,group6,group7,group8,group9,group10,group11,group12,group13,group14,group15,group16,group17,group18,group19,group20 testuser1

[root@client-test testnfs]# id -a testuser1
uid=20362(testuser1) gid=20362(testuser1) groups=20362(testuser1),20363(group1),20364(group2),20365(group3),20366(group4),20367(group5),20368(group6),20369(group7),20370(group8),20371(group9),20372(group10),20373(group11),20374(group12),20375(group13),20376(group14),20377(group15),20378(group16),20379(group17),20380(group18),20381(group19),20382(group20)
  • Setup on server and client is as follows:

drwxrwxr-x. 2 testuser1 group1  4.0K Aug 19 15:15 1
drwxr-xr-x. 2 testuser1 group2  4.0K Aug 19 15:15 2
drwxr-xr-x. 2 testuser1 group3  4.0K Aug 19 15:15 3
drwxr-xr-x. 2 testuser1 group4  4.0K Aug 19 15:15 4
drwxr-xr-x. 2 testuser1 group5  4.0K Aug 19 15:15 5
drwxr-xr-x. 2 testuser1 group6  4.0K Aug 19 15:15 6
drwxr-xr-x. 2 testuser1 group7  4.0K Aug 19 15:15 7
drwxr-xr-x. 2 testuser1 group8  4.0K Aug 19 15:16 8
drwxr-xr-x. 2 testuser1 group9  4.0K Aug 19 15:16 9
drwxr-xr-x. 2 testuser1 group10 4.0K Aug 19 15:16 10
drwxr-xr-x. 2 testuser1 group11 4.0K Aug 19 15:16 11
drwxr-xr-x. 2 testuser1 group12 4.0K Aug 19 15:16 12
drwxr-xr-x. 2 testuser1 group13 4.0K Aug 19 15:16 13
drwxr-xr-x. 2 testuser1 group14 4.0K Aug 19 15:16 14
drwxr-xr-x. 2 testuser1 group15 4.0K Aug 19 15:16 15
drwxr-xr-x. 2 testuser1 group16 4.0K Aug 19 15:16 16
drwxr-xr-x. 2 testuser1 group17 4.0K Aug 19 15:16 17
drwxr-xr-x. 2 testuser1 group18 4.0K Aug 19 15:16 18
drwxr-xr-x. 2 testuser1 group19 4.0K Aug 19 15:16 19
drwxr-xr-x. 2 testuser1 group20 4.0K Aug 19 15:16 20
  • Tried changing ownership of one of the directory.

[root@client-test /]# su - testuser1
$ cd /testnfs
testnfs]$ chown testsuer1:group20 1
chown: changing ownership of `1': Operation not permitted <--------------Error reported
  • Whereas locally on the NFS server, this operation works without any problem :

[root@server-test test1]# su - testuser1                      [  OK  ]
[testuser1@server-test ~]$ cd /test1
[testuser1@server-test test1]$ chown testuser1:group20 1
[testuser1@server-test test1]$ ls -ld 1
drwxrwxr-x. 2 testuser1 group20 4096 Aug 19 15:15 1
  • Tcpdump captured during the same time also shows "Permission Error".
    Inspect the the RPC request (e.g. SETATTR Call). For the SETATTR example, note that the GID the dir is being set to, 20382, does not appear in the credentials below.

Credentials
Flavor: AUTH_UNIX (1)
Length: 100
Stamp: 0x004f0cff
Machine Name: client-test
length: 14
contents: client-test
fill bytes: opaque data
UID: 20362
GID: 20362
Auxiliary GIDs (16) [20362, 20363, 20364, 20365, 20366, 20367, 20368, 20369, 20370, 20371, 20372, 20373, 20374, 20375, 20376, 20377]
GID: 20362
GID: 20363
GID: 20364
GID: 20365
GID: 20366
GID: 20367
GID: 20368
GID: 20369
GID: 20370
GID: 20371
GID: 20372
GID: 20373
GID: 20374
GID: 20375
GID: 20376
GID: 20377
Verifier
Flavor: AUTH_NULL (0)
Length: 0

Network File System, SETATTR Call FH:0x1622cb12
[Program Version: 3]
[V3 Procedure: SETATTR (2)]
object
length: 28
[hash (CRC-32): 0x1622cb12]
decode type as: unknown
filehandle: 01000601c8166eb71d244e24a1af18679aeb921c02000200...
new_attributes
mode: no value
set_it: no value (0)
uid: value follows
set_it: value follows (1)
uid: 20362
gid: value follows
set_it: value follows (1)
gid: 20382
size: no value
set_it: no value (0)
atime: don't change
set_it: don't change (0)
mtime: don't change
set_it: don't change (0)
guard: no value
check: no value (0)
  • Then on NFS server, I made following entry in /etc/sysconfig/nfs and restarted NFS service:

RPCMOUNTDOPTS="--manage-gids"
  • Unmount NFS share from client and re-mounted again.
  • Lastly, I tried to change the ownership once again as follows and this time it worked without any problems :

# mount -t nfs -o vers=3 10.65.210.132:/test1 /testnfs
# su - testuser1
$ cd /testnfs
$ chown testuser1:group18 1
$ ls -ld 1
drwxrwxr-x. 2 testsuer1 group18 4096 Aug 19 15:15 1
Thus we can conclude, setting rpc.mountd --manage-gids solves NFS limitation of 16 groups.
Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)