The reason that you don’t see all your contacts may be that some contacts are shared with you, but you aren’t assigned to them. If the contact only shows up under Contacts shared with me list, then that contact was shared with you and not assigned to you. This does not put them in your contacts list, but it does allow you to get a notification for any changes made to that contact.