Matter distribution around clusters is highly anisotropic because clusters are the nodes of the cosmic web. The shape of the clusters and the number of filaments to which they are connected, that is, their connectivity, is thought to reflect their level of anisotropic matter distribution and must in principle be related to their physical properties. We investigate the effect of the dynamical state and the formation history on both the morphology and local connectivity of about 2400 groups and clusters of galaxies from the large hydrodynamical simulation IllustrisTNG at z = 0. We find that the mass of groups and clusters mainly affects the geometry of the matter distribution: Massive halos are significantly more elliptical and are more strongly connected to the cosmic web than low-mass halos. Beyond the mass-driven effect, ellipticity and connectivity are correlated and are imprints of the growth rate of groups and clusters. Both anisotropy measures appear to trace different dynamical states, such that unrelaxed groups and clusters are more elliptical and more connected than relaxed ones. This relation between matter anisotropies and dynamical state is the sign of different accretion histories. Relaxed groups and clusters have mostly been formed a long time ago and are slowly accreting matter at the present time. They are highly spherical and weakly connected to their environment, mostly because they had enough time to relax and thus lost the connection with their preferential directions of accretion and merging. In contrast, late-formed unrelaxed objects are highly anisotropic with strong connectivities and ellipticities. These groups and clusters are in their formation phase and must be strongly affected by the infalling of materials from filaments.