Было у меня такое. в основном с рендером на большие разрешения.
Вот что говорит об этом Владо:
Basically, you are running out of RAM. One way to reduce the memory usage for the render elements is to write the image to a .vrimg file and then use the vrimg2exr converter to extract the channels.
(*) Make sure you are using the V-Ray VFB;
(*) In the Common tab of the Render scene dialog set the resolution to something like 320x240;
(*) In the V-Ray VFB settings, turn
OFF the
"Get resolution from Max" option for the VFB, and specify the actual resolution that you need;
(*) Turn
off the
"use memory frame buffer" option;
(*) Turn
on the
"write raw image file" option and specify a .vrimg file in the field below it.
(*) Turn
on the
"generate preview" option.
(*) After the rendering is complete, use the vrimg2exr tool to extract the render elements as .exr files. You can find the vrimg2exr tool in the Start menu > Programs > Chaos Group > V-Ray for 3dsmax > Tools. There is also a GUI for it posted in the off-topic section of the forum, I think.
Alternatively, you can simply render on a 64-bit machine with 64-bit 3dsmax and V-Ray.
Best regards,
Vlado
вот еще информация к размышлению
http://3dcenter.ru/forum/index.php?showtopic=80451