The present study aimed to identify the trends of changes in physiological indicators and subjective restorativeness in response to audio-visual interactions in the environment. Four scenarios types were presented using four different modalities (video-sound, image-sound, sound-only, and video-only; each modality was evaluated by independent groups of subjects). The physiological responses and subjective restoration of subjects were measured to assess the interactions between the audio-visual modalities. These data were also analysed to determine the physiological and subjective differences between dynamic and static visual presentations. We found that with visual modalities, the heart rate (HR), heart rate variability (HRV) calculated using the standard deviation of the NN intervals (SDNN HRV), high-frequency band in the HRV power spectrum (HF HRV), alpha reactivity on electroencephalography, and skin temperature (ST) decreased; however, the beta reactivity on EEG and skin conductance level (SCL) increased. With auditory modalities, the SDNN HRV, HF-HRV, ST, and respiration depth decreased; however, the respiration rate and SCL increased. Use of static images and sound to reproduce the natural environment evoked more physiological comfort and subjective restorativeness. These findings could provide physiological insights for the theory of the restorative environment.