Most of the real-world datasets are highly imbalanced and learning from those imbalanced data is challenging. Imbalanced data often leads to biased models that perform poorly on minority class examples. Addressing the class imbalance through weight space characterization is an exploratory research work addressing the unsolved problem both in academia and industry. Conducted an extensive ablation study using various evaluation approaches to highlight the inherent limitations of those conventional metrics. To advance in weight space learning in the applied context, we have identified numerous potential use cases. These involve adjusting the weight values of a trained model for downstream tasks, utilizing a large language model (LLM).