Mechanistic Understanding and Control of Multi-modal LLMs
Principal Investigators & Key Members:
Khoa D. Doan, PhD
Multimodal LLMs that can solve complex visual, mathematical, programming, and discovery tasks still make simple mistakes - e.g., misjudging spatial relations, failing at genuine discovery, or lacking cultural awareness - limiting safe use in healthcare, finance, and low-resource/culturally diverse settings such as Vietnam. This project will build real-world benchmarks to reveal these weaknesses, explain why they occur using mechanistic analyses, and develop lightweight inference-time control and data-efficient fine-tuning methods to fix them without costly retraining. We will release open datasets, tools, and guidelines to enable safer, more inclusive AI, cut energy use through efficient training, and unlock trustworthy applications that support local innovation and economic growth.
