Construction material plays an increasingly important role in the environmental impacts of buildings. In order to investigate impacts of materials on a building level, we present a bottom‐up building stock model that uses three‐dimensional and geo‐referenced building data to determine volumetric information of material stocks in Swiss residential buildings. We used a probabilistic modeling approach to calculate future material flows for the individual buildings. We investigated six scenarios with different assumptions concerning per‐capita floor area, building stock turnover, and construction material. The Swiss building stock will undergo important structural changes by 2035. While this will lead to a reduced number in new constructions, material flows will increase. Total material inflow decreases by almost half while outflows double. In 2055, the total amount of material in‐ and outflows are almost equal, which represents an important opportunity to close construction material cycles. Total environmental impacts due to production and disposal of construction material remain relatively stable over time. The cumulated impact is slightly reduced for the wood‐based scenario. The scenario with more insulation material leads to slightly higher material‐related emissions. An increase in per‐capita floor area or material turnover will lead to a considerable increase in impacts. The new modeling approach overcomes the limitations of previous bottom‐up building models and allows for investigating building material flows and stocks in space and time. This supports the development of tailored strategies to reduce the material footprint and environmental impacts of buildings and settlements.