Sora AI Review: Will AI Replace Videographers For Good?

Have you ever wanted to create high-quality videos from nothing but words? In February 2024, OpenAI unveiled Sora, an AI system capable of creating photorealistic videos from text prompts that can be up to 20 seconds long. Since December 2024, the tool has been accessible to…

Ecologists find computer vision models’ blind spots in retrieving wildlife images

Try taking a picture of each of North America’s roughly 11,000 tree species, and you’ll have a mere fraction of the millions of photos within nature image datasets. These massive collections of snapshots — ranging from butterflies to humpback whales — are a great research tool for ecologists because they provide evidence of organisms’ unique behaviors, rare conditions, migration patterns, and responses to pollution and other forms of climate change.

While comprehensive, nature image datasets aren’t yet as useful as they could be. It’s time-consuming to search these databases and retrieve the images most relevant to your hypothesis. You’d be better off with an automated research assistant — or perhaps artificial intelligence systems called multimodal vision language models (VLMs). They’re trained on both text and images, making it easier for them to pinpoint finer details, like the specific trees in the background of a photo.

But just how well can VLMs assist nature researchers with image retrieval? A team from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), University College London, iNaturalist, and elsewhere designed a performance test to find out. Each VLM’s task: locate and reorganize the most relevant results within the team’s “INQUIRE” dataset, composed of 5 million wildlife pictures and 250 search prompts from ecologists and other biodiversity experts. 

Looking for that special frog

In these evaluations, the researchers found that larger, more advanced VLMs, which are trained on far more data, can sometimes get researchers the results they want to see. The models performed reasonably well on straightforward queries about visual content, like identifying debris on a reef, but struggled significantly with queries requiring expert knowledge, like identifying specific biological conditions or behaviors. For example, VLMs somewhat easily uncovered examples of jellyfish on the beach, but struggled with more technical prompts like “axanthism in a green frog,” a condition that limits their ability to make their skin yellow.

Their findings indicate that the models need much more domain-specific training data to process difficult queries. MIT PhD student Edward Vendrow, a CSAIL affiliate who co-led work on the dataset in a new paper, believes that by familiarizing with more informative data, the VLMs could one day be great research assistants. “We want to build retrieval systems that find the exact results scientists seek when monitoring biodiversity and analyzing climate change,” says Vendrow. “Multimodal models don’t quite understand more complex scientific language yet, but we believe that INQUIRE will be an important benchmark for tracking how they improve in comprehending scientific terminology and ultimately helping researchers automatically find the exact images they need.”

The team’s experiments illustrated that larger models tended to be more effective for both simpler and more intricate searches due to their expansive training data. They first used the INQUIRE dataset to test if VLMs could narrow a pool of 5 million images to the top 100 most-relevant results (also known as “ranking”). For straightforward search queries like “a reef with manmade structures and debris,” relatively large models like “SigLIP” found matching images, while smaller-sized CLIP models struggled. According to Vendrow, larger VLMs are “only starting to be useful” at ranking tougher queries.

Vendrow and his colleagues also evaluated how well multimodal models could re-rank those 100 results, reorganizing which images were most pertinent to a search. In these tests, even huge LLMs trained on more curated data, like GPT-4o, struggled: Its precision score was only 59.6 percent, the highest score achieved by any model.

The researchers presented these results at the Conference on Neural Information Processing Systems (NeurIPS) earlier this month.

Inquiring for INQUIRE

The INQUIRE dataset includes search queries based on discussions with ecologists, biologists, oceanographers, and other experts about the types of images they’d look for, including animals’ unique physical conditions and behaviors. A team of annotators then spent 180 hours searching the iNaturalist dataset with these prompts, carefully combing through roughly 200,000 results to label 33,000 matches that fit the prompts.

For instance, the annotators used queries like “a hermit crab using plastic waste as its shell” and “a California condor tagged with a green ‘26’” to identify the subsets of the larger image dataset that depict these specific, rare events.

Then, the researchers used the same search queries to see how well VLMs could retrieve iNaturalist images. The annotators’ labels revealed when the models struggled to understand scientists’ keywords, as their results included images previously tagged as irrelevant to the search. For example, VLMs’ results for “redwood trees with fire scars” sometimes included images of trees without any markings.

“This is careful curation of data, with a focus on capturing real examples of scientific inquiries across research areas in ecology and environmental science,” says Sara Beery, the Homer A. Burnell Career Development Assistant Professor at MIT, CSAIL principal investigator, and co-senior author of the work. “It’s proved vital to expanding our understanding of the current capabilities of VLMs in these potentially impactful scientific settings. It has also outlined gaps in current research that we can now work to address, particularly for complex compositional queries, technical terminology, and the fine-grained, subtle differences that delineate categories of interest for our collaborators.”

“Our findings imply that some vision models are already precise enough to aid wildlife scientists with retrieving some images, but many tasks are still too difficult for even the largest, best-performing models,” says Vendrow. “Although INQUIRE is focused on ecology and biodiversity monitoring, the wide variety of its queries means that VLMs that perform well on INQUIRE are likely to excel at analyzing large image collections in other observation-intensive fields.”

Inquiring minds want to see

Taking their project further, the researchers are working with iNaturalist to develop a query system to better help scientists and other curious minds find the images they actually want to see. Their working demo allows users to filter searches by species, enabling quicker discovery of relevant results like, say, the diverse eye colors of cats. Vendrow and co-lead author Omiros Pantazis, who recently received his PhD from University College London, also aim to improve the re-ranking system by augmenting current models to provide better results.

University of Pittsburgh Associate Professor Justin Kitzes highlights INQUIRE’s ability to uncover secondary data. “Biodiversity datasets are rapidly becoming too large for any individual scientist to review,” says Kitzes, who wasn’t involved in the research. “This paper draws attention to a difficult and unsolved problem, which is how to effectively search through such data with questions that go beyond simply ‘who is here’ to ask instead about individual characteristics, behavior, and species interactions. Being able to efficiently and accurately uncover these more complex phenomena in biodiversity image data will be critical to fundamental science and real-world impacts in ecology and conservation.”

Vendrow, Pantazis, and Beery wrote the paper with iNaturalist software engineer Alexander Shepard, University College London professors Gabriel Brostow and Kate Jones, University of Edinburgh associate professor and co-senior author Oisin Mac Aodha, and University of Massachusetts at Amherst Assistant Professor Grant Van Horn, who served as co-senior author. Their work was supported, in part, by the Generative AI Laboratory at the University of Edinburgh, the U.S. National Science Foundation/Natural Sciences and Engineering Research Council of Canada Global Center on AI and Biodiversity Change, a Royal Society Research Grant, and the Biome Health Project funded by the World Wildlife Fund United Kingdom.

Monetizing Research for AI Training: The Risks and Best Practices

As the demand for generative AI grows, so does the hunger for high-quality data to train these systems. Scholarly publishers have started to monetize their research content to provide training data for large language models (LLMs). While this development is creating a new revenue stream for…

The Little Triangle in the Tooltip

Today, I want to focus on what I’ll call the little triangle in the tooltip. It receives minimal attention but it amazes you by how many ways there are to make them. Let’s start with the simplest and make our way up to the not-so-simple.

The Little…

Tiny, wireless antennas use light to monitor cellular communication

Monitoring electrical signals in biological systems helps scientists understand how cells communicate, which can aid in the diagnosis and treatment of conditions like arrhythmia and Alzheimer’s.

But devices that record electrical signals in cell cultures and other liquid environments often use wires to connect each electrode on the device to its respective amplifier. Because only so many wires can be connected to the device, this restricts the number of recording sites, limiting the information that can be collected from cells.

MIT researchers have now developed a biosensing technique that eliminates the need for wires. Instead, tiny, wireless antennas use light to detect minute electrical signals.

Small electrical changes in the surrounding liquid environment alter how the antennas scatter the light. Using an array of tiny antennas, each of which is one-hundredth the width of a human hair, the researchers could measure electrical signals exchanged between cells, with extreme spatial resolution.

The devices, which are durable enough to continuously record signals for more than 10 hours, could help biologists understand how cells communicate in response to changes in their environment. In the long run, such scientific insights could pave the way for advancements in diagnosis, spur the development of targeted treatments, and enable more precision in the evaluation of new therapies.

“Being able to record the electrical activity of cells with high throughput and high resolution remains a real problem. We need to try some innovative ideas and alternate approaches,” says Benoît Desbiolles, a former postdoc in the MIT Media Lab and lead author of a paper on the devices.

He is joined on the paper by Jad Hanna, a visiting student in the Media Lab; former visiting student Raphael Ausilio; former postdoc Marta J. I. Airaghi Leccardi; Yang Yu, a scientist at Raith America, Inc.; and senior author Deblina Sarkar, the AT&T Career Development Assistant Professor in the Media Lab and MIT Center for Neurobiological Engineering and head of the Nano-Cybernetic Biotrek Lab. The research appears today in Science Advances.

“Bioelectricity is fundamental to the functioning of cells and different life processes. However, recording such electrical signals precisely has been challenging,” says Sarkar. “The organic electro-scattering antennas (OCEANs) we developed enable recording of electrical signals wirelessly with micrometer spatial resolution from thousands of recording sites simultaneously. This can create unprecedented opportunities for understanding fundamental biology and altered signaling in diseased states as well as for screening the effect of different therapeutics to enable novel treatments.”

Biosensing with light

The researchers set out to design a biosensing device that didn’t need wires or amplifiers. Such a device would be easier to use for biologists who may not be familiar with electronic instruments.

“We wondered if we could make a device that converts the electrical signals to light and then use an optical microscope, the kind that is available in every biology lab, to probe these signals,” Desbiolles says.

Initially, they used a special polymer called PEDOT:PSS to design nanoscale transducers that incorporated tiny pieces of gold filament. Gold nanoparticles were supposed to scatter the light — a process that would be induced and modulated by the polymer. But the results weren’t matching up with their theoretical model.

The researchers tried removing the gold and, surprisingly, the results matched the model much more closely.

“It turns out we weren’t measuring signals from the gold, but from the polymer itself. This was a very surprising but exciting result. We built on that finding to develop organic electro-scattering antennas,” he says.

The organic electro-scattering antennas, or OCEANs, are composed of PEDOT:PSS. This polymer attracts or repulses positive ions from the surrounding liquid environment when there is electrical activity nearby. This modifies its chemical configuration and electronic structure, altering an optical property known as its refractive index, which changes how it scatters light.

When researchers shine light onto the antenna, the intensity of the light changes in proportion to the electrical signal present in the liquid.

Six-by-six array of tiny lights that glow brighter as voltage goes from 0 to -0.8.
The brightness of light scattered by the tiny antennas the researchers developed, called OCEANs, changes in response to changing electrical signals in the liquid environment that surrounds them, as shown here. By capturing and measuring the light with an optical microscope, researchers could decode the intricate signals used for cellular communication.

Credit: Courtesy of the researchers

With thousands or even millions of tiny antennas in an array, each only 1 micrometer wide, the researchers can capture the scattered light with an optical microscope and measure electrical signals from cells with high resolution. Because each antenna is an independent sensor, the researchers do not need to pool the contribution of multiple antennas to monitor electrical signals, which is why OCEANs can detect signals with micrometer resolution.

Intended for in vitro studies, OCEAN arrays are designed to have cells cultured directly on top of them and put under an optical microscope for analysis.

“Growing” antennas on a chip

Key to the devices is the precision with which the researchers can fabricate arrays in the MIT.nano facilities.

They start with a glass substrate and deposit layers of conductive then insulating material on top, each of which is optically transparent. Then they use a focused ion beam to cut hundreds of nanoscale holes into the top layers of the device. This special type of focused ion beam enables high-throughput nanofabrication.

“This instrument is basically like a pen where you can etch anything with a 10-nanometer resolution,” he says.

They submerge the chip in a solution that contains the precursor building blocks for the polymer. By applying an electric current to the solution, that precursor material is attracted into the tiny holes on the chip, and mushroom-shaped antennas “grow” from the bottom up.

The entire fabrication process is relatively fast, and the researchers could use this technique to make a chip with millions of antennas.

“This technique could be easily adapted so it is fully scalable. The limiting factor is how many antennas we can image at the same time,” he says.

The researchers optimized the dimensions of the antennas and adjusted parameters, which enabled them to achieve high enough sensitivity to monitor signals with voltages as low as 2.5 millivolts in simulated experiments. Signals sent by neurons for communication are usually around 100 millivolts.

“Because we took the time to really dig in and understand the theoretical model behind this process, we can maximize the sensitivity of the antennas,” he says.

OCEANs also responded to changing signals in only a few milliseconds, enabling them to record electrical signals with fast kinetics. Moving forward, the researchers want to test the devices with real cell cultures. They also want to reshape the antennas so they can penetrate cell membranes, enabling more precise signal detection.

In addition, they want to study how OCEANs could be integrated into nanophotonic devices, which manipulate light at the nanoscale for next-generation sensors and optical devices.

This research is funded, in part, by the U.S. National Institutes of Health and the Swiss National Science Foundation. Research reported in this press release was supported by the National Heart, Lung, and Blood Institute (NHLBI) of the National Institutes of Health and does not necessarily represent the official views of the NIH.

Global MIT At-Risk Fellows Program expands to invite Palestinian scholars

When the Global MIT At-Risk Fellows (GMAF) initiative launched in February 2024 as a pilot program for Ukrainian researchers, its architects expressed hope that GMAF would eventually expand to include visiting scholars from other troubled areas of the globe. That time arrived this fall, when MIT launched GMAF-Palestine, a two-year pilot that will select up to five fellows each year currently either in Palestine or recently displaced to continue their work during a semester at MIT.

Designed to enhance the educational and research experiences of international faculty and researchers displaced by humanitarian crises, GMAF brings international scholars to MIT for semester-long study and research meant to benefit their regions of origin while simultaneously enriching the MIT community.

Referring to the ongoing war and humanitarian crisis in Gaza, GMAF-Palestine Director and MIT Professor Kamal Youcef-Toumi says that “investing in scientists is an important way to address this significant conflict going on in our world.” Youcef-Toumi says it’s hoped that this program “will give some space for getting to know the real people involved and a deeper understanding of the practical implications for people living through the conflict.”

Professor Duane Boning, vice provost for international activities, considers the GMAF program to be a practical way for MIT to contribute to solving the world’s most challenging problems. “Our vision is for the fellows to come to MIT for a hands-on, experiential joint learning and research experience that develops the tools necessary to support the redevelopment of their regions,” says Boning.

“Opening and sustaining connections among scholars around the world is an essential part of our work at MIT,” says MIT President Sally Kornbluth. “New collaborations so often spark new understanding and new ideas; that’s precisely what we aim to foster with this kind of program.”  

Crediting Program Manager Dorothy Hanna with much of the legwork that got the fellowship off the ground, Youcef-Toumi says fellows for the program’s inaugural year will be chosen from early- and mid-career scientists via an open application and nominations from the MIT community. Following submission of applications and interviews in January, five scholars will be selected to begin their fellowships at MIT in September 2025.

Eligible applicants must have held academic or research appointments at a Palestinian university within the past five years; hold a PhD or equivalent degree in a field represented at MIT; have been born in Gaza, the West Bank, East Jerusalem, or refugee camps; have a reasonable expectation of receiving a U.S. visa, and be working in a research area represented at MIT. MIT will cover all fellowship expenses, including travel, accommodations, visas, health insurance, instructional materials, and living stipends.

To build strong relationships during their time at MIT, GMAF-Palestine will pair fellows with faculty mentors and keep them connected with other campus communities, including the Ibn Khaldun Fellowship for Saudi Arabian Women, an over 10-year-old program that Youcef-Toumi’s team also oversees. 

“MIT has a special environment and mindset that I think will be very useful. It’s a competitive environment, but also very supportive,” says Youcef-Toumi, a member of the Department of Mechanical Engineering faculty, director of the Mechatronics Research Laboratory, and co-director of the Center for Complex Engineering Systems. “In many other places, if a person is in math, they stay in math. If they are in architecture, they stay in architecture and they are not dealing with other departments or other colleges. In our case, because students’ work is often so interdisciplinary, a student in mechanical engineering can have an advisor in computer science or aerospace, and basically everything is open. There are no walls.”

Youcef-Toumi says he hopes MIT’s collegial environment among diverse departments and colleagues is a value fellows will retain and bring back to their own universities and communities.

“We are all here for scholarship. All of the people who come to MIT … they are coming for knowledge. The technical part is one thing, but there are other things here that are not available in many environments — you know, the sense of community, the values, and the excellence in academics,” Youcef-Toumi says. “These are things we will continue to emphasize, and hopefully these visiting scientists can absorb and benefit from some of that. And we will also learn from them, from their seminars and discussions with them.”

Referencing another new fellowship program launched by MIT, Kalaniyot for Israeli scholars, led by MIT professors Or Hen and Ernest Fraenkel, Youcef-Toumi says, “Getting to know the Kalaniyot team better has been great, and I’m sure we will be helping each other. To have people from that region be on campus and interacting with different people … hopefully that will add a more positive effect and unity to the campus. This is one of the things that we hope these programs will do.”

As with any first endeavor, GMAF-Palestine’s first round of fellowships and the experiences of the fellows, and the observations of the GMAF team, will inform future iterations of the program. In addition to Youcef-Toumi, leadership for the program is provided by a faculty committee representing the breadth of scholarship at MIT. The vision of the faculty committee is to establish a sustainable program connecting the Palestinian community and MIT.

“Longer term,” Youcef-Toumi says, “we hope to show the MIT community this is a really impactful program that is worth sustaining with continued fundraising and philanthropy. We plan to stay in touch with the fellows and collect feedback from them over the first five years on how their time at MIT has impacted them as researchers and educators. Hopefully, this will include ongoing collaborations with their MIT mentors or others they meet along the way at MIT.”

MIT-Kalaniyot launches programs for visiting Israeli scholars

Over the past 14 months, as the impact of the ongoing Israel-Gaza war has rippled across the globe, a faculty-led initiative has emerged to support MIT students and staff by creating a community that transcends ethnicity, religion, and political views. Named for a flower that blooms along the Israel-Gaza border, MIT-Kalaniyot began hosting weekly community lunches that typically now draw about 100 participants. These gatherings have gained the interest of other universities seeking to help students not only cope with but thrive through troubled times, with some moving to replicate MIT’s model on their own campuses.

Now, scholars at Israel’s nine state-recognized universities will be able to compete for MIT-Kalaniyot fellowships designed to allow Israel’s top researchers to come to MIT for collaboration and training, advancing research while contributing to a better understanding of their country.

The MIT-Kalaniyot Postdoctoral Fellows Program will support scholars who have recently graduated from Israeli PhD programs to continue their postdoctoral training at MIT. Meanwhile, the new MIT-Kalaniyot Sabbatical Scholars Program will provide faculty and researchers holding sabbatical-eligible appointments at Israeli research institutions with fellowships for two academic terms at MIT.

Announcement of the fellowships through the association of Israeli university presidents spawned an enthusiastic response. 

“We’ve received many emails, from questions about the program to messages of gratitude. People have told us that, during a time of so much negativity, seeing such a top-tier academic program emerge feels like a breath of fresh air,” says Or Hen, the Class of 1956 Associate Professor of Physics and associate director of the Laboratory for Nuclear Science, who co-founded MIT-Kalaniyot with Ernest Fraenkel, the Grover M. Hermann Professor in Health Sciences and Technology.

Hen adds that the response from potential program donors has been positive, as well.

“People have been genuinely excited to learn about forward-thinking efforts and how they can simultaneously support both MIT and Israeli science,” he says. “We feel truly privileged to be part of this meaningful work.”

MIT-Kalaniyot is “a faculty-led initiative that emerged organically as we came to terms with some of the challenges that MIT was facing trying to keep focusing on its mission during a very difficult period for the U.S., and obviously for Israelis and Palestinians,” Fraenkel says.

As the MIT-Kalaniyot Program gained momentum, he adds, “we started talking about positive things faculty can do to help MIT fulfill its mission and then help the world, and we recognized many of the challenges could actually be helped by bringing more brilliant scholars from Israel to MIT to do great research and to humanize the face of Israelis so that people who interact with them can see them, not as some foreign entity, but as the talented person working down the hallway.”

“MIT has a long tradition of connecting scholarly communities around the world,” says MIT President Sally Kornbluth. “Programs like this demonstrate the value of bringing people and cultures together, in pursuit of new ideas and understanding.”    

Open to applicants in the humanities, architecture, management, engineering, and science, both fellowship programs aim to embrace Israel’s diverse demographics by encouraging applications from all communities and minority groups throughout Israel.

Fraenkel notes that because Israeli universities reflect the diversity of the country, he expects scholars who identify as Israeli Arabs, Palestinian citizens of Israel, and others could be among the top candidates applying and ultimately selected for MIT-Kalaniyot fellowships. 

MIT is also expanding its Global MIT At-Risk Fellows Program (GMAF), which began last year with recruitment of scholars from Ukraine, to bring Palestinian scholars to campus next fall. Fraenkel and Hen noted their close relationship with GMAF-Palestine director Kamal Youcef-Toumi, a professor in MIT’s Department of Mechanical Engineering.  

“While the programs are independent of each other, we value collaboration at MIT and are hoping to find positive ways that we can interact with each other,” Fraenkel says.

Also growing up alongside MIT-Kalaniyot’s fellowship programs will be new Kalaniyot chapters at universities such as the University of Pennsylvania and Dartmouth College, where programs have already begun, and others where activity is starting up. MIT’s inspiration for these efforts, Hen and Fraenkel say, is a key aspect of the Kalaniyot story.

“We formed a new model of faculty-led communities,” Hen says. “As faculty, our roles typically center on teaching, mentoring, and research. After October 7 happened, we saw what was happening around campus and across the nation and realized that our roles had to expand. We had to go beyond the classroom and the lab to build deeper connections within the community that transcends traditional academic structures. This faculty-led approach has become the essence of MIT-Kalaniyot, and is now inspiring similar efforts across the nation.”

Once the programs are at scale, MIT plans to bring four MIT-Kalaniyot Postdoctoral Fellows to campus annually (for three years each), as well as four MIT-Kalaniyot Sabbatical Scholars, for a total of 16 visiting Israeli scholars at any one time.

“We also hope that when they go back, they will be able to maintain their research ties with MIT, so we plan to give seed grants to encourage collaboration after someone leaves,” Fraenkel says. “I know for a lot of our postdocs, their time at MIT is really critical for making networks, regardless of where they come from or where they go. Obviously, it’s harder when you’re across the ocean in a very challenging region, and so I think for both programs it would be great to be able to maintain those intellectual ties and collaborate beyond the term of their fellowships.”

A common thread between the new Kalaniyot programs and GMAF-Palestine, Hen says, is to rise beyond differences that have been voiced post-Oct. 7 and refocus on the Institute’s core research mission.

“We’re bringing in the best scholars from the region — Jews, Israelis, Arabs, Palestinians — and normalizing interactions with them and among them through collaborative research,” Hen says. “Our mission is clear: to focus on academic excellence by bringing outstanding talent to MIT and reinforcing that we are here to advance research in service of humanity.”