The tool is made available for research use through the Open Science Framework.
In conclusion, Mind-Video is a groundbreaking AI tool that has the potential to revolutionize the field of neuroscience by enabling the reconstruction of high-quality videos from brain activity captured through fMRI data. By employing a two-module pipeline that bridges the gap between image and video brain decoding, Mind-Video is able to generate videos that demonstrate high semantic accuracy, including motions and scene dynamics. The tool's flexible and adaptable pipeline, which consists of an fMRI encoder and an augmented stable diffusion model trained separately and finetuned together, enables the encoder to learn brain features through multiple stages. The attention analysis of the transformers decoding fMRI data reveals the dominance of the visual cortex in processing visual spatiotemporal information and the hierarchical nature of the encoder's layers in extracting structural and abstract visual features. Overall, Mind-Video is a significant advancement in the field of neuroscience and has the potential to enable new research and applications in the field of brain-computer interfaces and the study of human cognition.
