I recently realized thanks to a colleague @MMeuree, that the ID_TOKEN that’s supposed to contain the group membership as shown below:
does not list more than 4 groups (here I grabbed the token using another flow). So, if the user belongs to more than 4 groups, you’re going to see hasgroups: true as part of the token instead of the actual groups. This behavior is by design no matter what you specified in the App manifest with regards to the groupMembershipClaims attribute. So, the alternative is simply to query the Graph API.