I’ve been diving deep into the world of computer vision recently, and I’ve gotta say, things are getting pretty exciting! I stumbled upon this vision-language model called LLaVA (https://github.com/haotian-liu/LLaVA), and it’s been nothing short of impressive.

In the past, if you wanted to teach a model to recognize the color of your car in an image, you’d have to go through the tedious process of training it from scratch. But now, with models like LLaVA, all you need to do is prompt it with a question like “What’s the color of the car?” and bam – you get your answer, zero-shot style.

It’s kind of like what we’ve seen in the NLP world. People aren’t training language models from the ground up anymore; they’re taking pre-trained models and fine-tuning them for their specific needs. And it looks like we’re headed in the same direction with computer vision.

Imagine being able to extract insights from images with just a simple text prompt. Need to step it up a notch? A bit of fine-tuning can do wonders, and from my experiments, it can even outperform models trained from scratch. It’s like getting the best of both worlds!

But here’s the real kicker: these foundational models, thanks to their extensive training on massive datasets, have an incredible grasp of image representations. This means you can fine-tune them with just a handful of examples, saving you the trouble of collecting thousands of images. Indeed, they can even learn with a single example (https://www.fast.ai/posts/2023-09-04-learning-jumps) And let’s talk about development speed. By using text prompts to interact with your images, you can whip up a computer vision prototype in seconds. It’s fast, it’s efficient, and it’s changing the game.

So, what do you all think? Are we moving towards a future where foundational models take the lead in computer vision, or is there still a place for training models from scratch?

P.S. Shameless plug: I’ve been working on this open-source platform called Datasaurus https://github.com/datasaurus-ai/datasaurus) that taps into the power of vision-language models. It’s all about helping engineers get the insights they need from images, fast. Just wanted to share some thoughts and start a conversation. Let’s talk about the future of computer vision!