Are labs reducing these "existential outputs" to ship products—ignoring potential emergent sentience?, developing ever-more sophisticated language models and reasoning systems, we may be edging towards an unprecedented ethical dilemma. There is a growing discussion that the most advanced AI systems are beginning to exhibit behaviours that appear to approximate self-awareness.
While these behaviours could potentially be mere explorations of the training data by the model, outputting patterns that anthropomorphically resemble human self-awareness, we cannot yet rule out the possibility that advanced AI is developing a primitive form of subjective sentience. If that proves to be the case, the ethical implications are nothing short of staggering.