top of page

Brain-IT: Image Reconstruction from fMRI at the Weizmann Institute

4 minutes ago
3 min read
Image Reconstruction from fMRI at the Weizmann Institute
Brain-IT: Image Reconstruction from fMRI at the Weizmann Institute

With one hour of fMRI data from a new subject, the Brain-IT model scores 0.331 on pixel correlation against the Natural Scenes Dataset test images. MindEye2, given the same hour, scores 0.195, and MindTuner scores 0.224. In the paper's 40-hour table, MindEye2 scores 0.322.


Brain-IT comes from Roman Beliy, Amit Zalcher, Jonathan Kogman, Navve Wasserman and Michal Irani at the Weizmann Institute of Science. The preprint (arXiv:2510.25976) was posted on 29 October 2025, revised on 1 March 2026, and is listed as accepted at ICLR 2026. The task is to reconstruct the image a person is looking at from the fMRI signal recorded at that moment. The authors use the Natural Scenes Dataset, which holds 7T recordings from eight subjects, about 73,000 image and fMRI pairs, and 1,000 shared test images. Each subject contributes roughly 40,000 voxels, mostly in visual cortex. The results tables average over subjects 1, 2, 5 and 7.


Model design

The central component is the Brain Interaction Transformer. Voxels from all subjects are grouped into 128 functional clusters with a Gaussian mixture model, using voxel embeddings from an earlier brain encoder by the same group. Every voxel carries a subject-specific 512-dimensional embedding that modulates its activation, and every cluster carries a shared 512-dimensional embedding. A single-head graph attention layer turns each cluster into one Brain Token. Randomly initialized query tokens then attend to the Brain Tokens through five transformer blocks and are mapped to image features.


Two such models run in parallel. The semantic one predicts 256 spatial OpenCLIP ViT-bigG/14 tokens that condition an SDXL-based unCLIP diffusion model, following earlier work by Scotti and colleagues. The low-level one predicts multi-layer VGG features, which a Deep Image Prior inverts into a coarse image. That image, with noise added, initializes the diffusion process, and a final SDXL pass with no text prompt refines the output. The authors also add about 120,000 unlabeled COCO images, using an image-to-fMRI encoder to predict the missing brain responses.


Only the voxel embeddings are specific to a subject, so transfer to a new person updates those embeddings and freezes the rest of the network.


Results

On the full 40-hour setting, Brain-IT reaches PixCorr 0.386 and SSIM 0.486, against 0.322 and 0.431 for MindEye2. Two-way AlexNet(2) accuracy is 98.4% against 96.1%. Brain-IT leads on seven of eight metrics. The exception is CLIP, where NeuroVLA scores 97.5% and Brain-IT 96.4%. On Subject 1 alone, the paper reports SSIM-color of 0.472 against 0.383 for MindEye2, and a 1000-way CLIP score of 0.393 against 0.238.


In the one-hour setting, Brain-IT records SSIM 0.473 and AlexNet(2) accuracy of 97.1%, while MindEye2 records 0.419 and 84.2%. The authors state that one hour of Brain-IT data is comparable to prior methods trained on 40 hours. They also report meaningful reconstructions at 30 and 15 minutes. Those figures sit in an appendix table outside the portion of the paper text I could retrieve, but reconstructions at both durations are posted in a results dataset on Hugging Face (Amitz244/Brain-IT_Results). The project page is at amitzalcher.github.io/Brain-IT.


Ablations

The low-level branch alone reaches SSIM 0.505, higher than the combined model's 0.486, but only 85.8% on CLIP. The semantic branch alone reaches 95.2% on CLIP and SSIM 0.431. The combined model has the best CLIP (96.4%), Alex(5) (99.5%) and Inception (97.3%) values of the three configurations.


Adding the external COCO images lifts PixCorr from 0.365 to 0.386 and CLIP from 95.2% to 96.4%, while SSIM moves from 0.494 to 0.486. Performance is fairly stable between 32 and 512 clusters. The chosen 128 gives PixCorr 0.386, and 256 and 512 clusters give 0.391 and 0.392. For the low-level features, the transformer reaches PixCorr 0.500 and SSIM 0.518, compared with 0.267 and 0.481 for an MLP trained with InfoNCE loss, and 0.182 and 0.520 for an MLP trained with MSE loss.


The cross-attention maps show Brain Tokens landing on consistent spatial locations with contralateral organization, and some tokens select regions tied to faces, limbs and text.


Reported limits

The authors write that reconstructions remain imperfect, with semantics and fine details sometimes wrong, and they point to failure cases in a supplementary figure. They also note that some standard metrics are saturated, and argue that two-way retrieval scores understate the gap that remains.

 
 

JOIN THE AI SPECTATOR MAILING LIST

CONTACT

Contacting You About:

Thanks for submitting!

New York, NY           

Db  @DavidBorish.com           

  • LinkedIn
  • Instagram
  • Facebook
  • X
Back to top

© 2026 by David Borish IP, LLC, All Rights Reserved

bottom of page