Teaching a computer to recognize a kitchen

Every apartment listing comes with dozens of photos: the kitchen, the bedrooms, sometimes a garden or a floor plan.

These photos arrive with no information about what they show. Meanwhile, features like search and recommendations would benefit from knowing whether a photo represents a kitchen, floor plan, or garden.

To tag photos with this information, we need a classifier: a machine learning model that takes photos as input and adds semantic labels like KITCHEN and GARDEN.

Three ways to build a classifier

There are three realistic ways to build one.

The first option is to prompt a large language model, the same kind of technology behind ChatGPT or Claude. You send it a photo and a question, and it answers. It works, but the costs quickly add up: roughly $1,500 for a million photos.

The second is to train your own classifier on top of a freely available AI vision model. This is dramatically cheaper to run: the same million photos come out to about 37 cents in compute. But it has a weakness that we ran into.

The third option is to go one step further and fine-tune that base model: continue its training with our own data, in our case 40,000 apartment photos annotated by hand. Fine-tuning allows the vision model to specialize in classifying real estate photos.

Simple has a price tag

Our first classifier had a stubborn problem: low confidence. Confidence means how certain the model is about each category; a label is only assigned when the model is certain enough. Nearly one photo in ten received no labels due to low confidence.

We discovered some reasons behind this.

The base model learned about the world from captioned internet images, so it is good at recognizing objects: sofas, ovens, cars, dogs. But some of our categories aren’t about objects at all. A garden is mostly background. A hallway is mostly architecture. The model wasn’t paying attention to what mattered.

There were edge cases: a bedroom showing through a doorway should not be tagged while a dining set sitting in the living room should. Without additional training, the base model had no way of making the distinction.

The $30 fix

After training the model further on our own annotated photos, the under-labeling problem essentially disappeared. Categories like GARDEN, which the original model struggled with, improved dramatically. The entire training experiment cost about $30 in computing.

When is this worth doing?

Building your own model isn’t always the right call. If you only classify a few hundred photos a day, it’s cheap enough that it really doesn’t matter how you do it. If prompting covers your needs, go for it. It is also easier to maintain. But when volumes are in the millions, your own custom model gets easier to justify.

Even if you build your own model, you might not need to fine-tune. But if low confidence becomes a problem, like it did for us, then you probably should consider it.

Where we are now

Photo classification is now part of the production infrastructure at Alma. Because every photo is tagged automatically, the labels are available at scale for Alma real estate products to build on.

For the full technical story, including the models and the numbers, see our article in Towards Data Science: https://towardsdatascience.com/why-we-fine-tuned-siglip-and-why-thats-not-always-the-right-call/

The authors of this Alma Tech Blog are Miikka Silfverberg, Senior Data Scientist, and Max Silfverberg, Data Scientist, AI Solutions Lead. They both work in Alma Marketplaces segment focusing on digital marketplaces, especially in the areas of housing and mobility. The segment’s best-known services include Etuovi, Nettiauto, Objektvision and Suomenlaki.com. The segment operates in Finland and Sweden.

Share article