Intolerance toward immigrants has recently reached noticeable highs in Switzerland. Referring to the conflict theory, the perception of a specific group as a threat tends to lead to intolerance toward that group. The expectation of a negative relationship between threat and tolerance is nevertheless not assumed to be universally valid for all members of society. In particular, the contact theory assumes that intercultural encounters should promote positive attitudes toward culturally different individuals and groups. Using a new and unique data set, our Heckman selection models reveal that Swiss who view rising immigration to mean a loss of economic privileges and an erosion of Swiss cultural values are less tolerant toward immigrants. Moreover, our results indicate that contact with immigrants may moderate this effect. However, not all group settings are able to reduce the perceived threats in a similar way, and not all sorts of social contact are able to foster tolerance toward immigrants.