so I figured out that when deleting all the files you got from a plugin you have to go into the ".godot" and then "imported" files (in your file explorer) and delete all imports of the deleted things that got imported to fully get rid of the plugin and save tons of space
I just wanted to share this as I did not know also for godot 3 the files may be different but it is probably still similar