The social relationship between an individual and their residential environment is shaped by a range of housing market rules and regulations, residential choice and by constraints. This paper elaborates on that relationship by focusing on the distance between an individual’s (and his/her household) social position and the social position of the neighbourhoods of residence. Through the analysis of large-scale longitudinal register data for each resident in the four largest cities of the Netherlands, we studied the relations mentioned, as well as the residential moves triggered by such relations, as well as the outcome effects on individual-neighbourhood relations in the destination neighbourhoods. We found that the larger the social distance (positive or negative) between an individual and the median social position of their residential neighbourhood, the higher the odds that the individual would move from that neighbourhood. Those individuals that moved tended to select destination neighbourhoods that reduced their social distance. Our findings offer new input for debates and policies relating to de-segregation and social mixing.