Leveraging Self-attention For Input-dependent Soft Prompting In Llms | Awesome LLM Papers Contribute to Awesome LLM Papers

Awesome LLM Papers

A curated collection of research papers on Large Language Models (LLMs), transformers, prompting, fine-tuning, and multi-modal systems. Maintained by Sean Moran.

Contact Sean Moran about this survey or website. Made with Jekyll and Hyde.

Leveraging Self-attention For Input-dependent Soft Prompting In Llms

Ananth Muppidi, Abhilash Nandy, Sambaran Bandyopadhyay . No Venue 2025

[Paper] [Paper]   Search on Google Scholar   Search on Semantic Scholar
Fine Tuning Model Architecture Prompting

The performance of large language models in domain-specific tasks necessitates fine-tuning, which is computationally expensive and technically challenging. This paper focuses on parameter-efficient fine-tuning using soft prompting, a promising approach that adapts pre-trained models to downstream tasks by learning a small set of parameters. We propose a novel Input Dependent Soft Prompting technique with a self-Attention Mechanism (ID-SPAM) that generates soft prompts based on the input tokens and attends different tokens with varying importance. Our method is simple and efficient, keeping the number of trainable parameters small. We show the merits of the proposed approach compared to state-of-the-art techniques on various tasks and show the improved zero shot domain transfer capability.

https://huggingface.co/discussions/paper/68464b273ec10bdd8ab4da89

Similar Work