Validation report 005: Food Image Classification

This study assesses the resilience of the ’nateraw/food’ Visual Transformer, a food classification model, against common data manipulation attacks. Employing methods like LIME and Attention Rollout for insight, the research finds that the model withstands most transformations, some extreme photographic effects and methods of overlaying key non-food features can significantly alter the predictions. These results highlight the model’s robustness, with implications for understanding the vulnerabilities of advanced computer vision systems.

Our analysis shows that the model is generally resilient to data manipulation. As illustrated in Figure 1, out of 16 transformations (10 photographic and 6 overlays), only 3 impacted the model’s predictions significantly: 180-degree rotation, an overlay of the plate and the application of grayscale filter. These techniques vary in their destructiveness of information, however, it’s important to note that plate overlay impacted the model significantly more than its ‘shadow’. This success leads us to conclude that strategically selected non-food features when overlaid on unrelated images, can effectively alter model predictions. As for photographic distortions, the impact of extreme measures, such as half-full rotation and grey filter, is consistent with loss of information in the image. This outcome demonstrates that only drastic photographic transformations to the image have a significant effect on the model’s reasoning.

Link to original model: Food Classifier

Recommended citation: Tomasz Silkowski. (2024). "Vulnerabilities in Food Image Classification." Github: ModelOriented/CVE-AI.
Download Paper