The sense of agency (SOA) is central to self-consciousness. The differentiation between self-caused events and externally caused events constitutes an essential boundary of the self in action. It is particularly interesting because it is often vague, not only in pathological cases (e.g. schizophrenia) but also in every-day situations. Yet, there is little agreement in philosophy or neuroscience on how the SOA arises. On the basis of a naturalistic background we develop a multifactorial account of agency that brings together different sense modalities and different levels of cognitive processing. Drawing on conceptual analysis and empirical studies, we argue that the SOA is the product of a cognitive weighing process of different factors such as proprioception and exteroception, feed-forward processes, intentional stances, thoughts, and environmental cues. We provide evidence for the relevance of each factor. The resulting naturalistic theory provides a framework for recent and future research of the SOA. Moreover, it captures the vagueness and flexibility of the boundaries of the self.