Is this a crazy idea? i.e. we really should not be doing this
Not a crazy idea per say, but it might be difficult. Godot wasn't really designed to be used as a standalone rendering library, so there may be difficulties in getting it working as one. As @Megalomaniak said, you can use it without the editor and as a platform, however it does not really expose a rendering API or anything.
Are you aware of any similar efforts?
The question of using Godot as a library comes up every so often, but I don't know if there is already any movement towards modifying Godot so it can act as a library. I would suggest searching something like "Godot as a rendering library" in a search engine, if you have not already, and see if that brings up any results of similar efforts.
Assuming everything goes well, are build changes like this welcome upstream in Godot?
While I cannot say for sure, and I am by no means a core Godot developer, I think it would be unlikely that such changes would be merged into Godot. That's not to say it is impossible or unwelcome, I just think it may not be merged because of the additional complexity it could bring and because Godot is intended to be used as a game engine, not a rendering library, I'm not sure the cost of the complexity would be deemed to be worth it. That said, that is just what I think would happen, not necessarily what would. If you want to know more definitively, I would ask on the devel
channel on the IRC.
Also, some interesting, low-level rendering APIs, if you want to go down that route:
Both are mainly just for making graphics programming easier and cross platform, but they might be interesting to look at. I've been meaning to try a project with either, but I haven't had the time and motivation to get one going just yet.