Drupal today is very page-oriented. Every request that comes in is responded to with a full HTML page. It is possible to return something else, and finally in Drupal 7 there is, sort of, native support to do so with delivery callbacks, but by and large any non-page response is an after thought at best and a hack at worst. The entire system is built around the assumption that we’re returning an HTML page. Why else would we still load the theme system and form system for an auto-complete callback?
We’re moving to a future where what we see on the web is a serious of HTML fragments tailored to our preferences or prior history. Drupal needs to move past its page orientation to function efficiently on the future web.