Including submodules without entire modules with PyInstaller

I'm turning some of my work into EXEs to be distributed amongst my colleagues, and before doing so I'd prefer to slim down the EXE that PyInstaller is coming out with. I'm currently using Auto-Py-To-Exe, but would be fine doing it in CLI. How would I go about importing matplotlib.pyplot, without including the entirety of matplotlib? I first tried including --exclude-module matplotlib and --hidden-import matplotlib.pyplot but that didn't work, and I got error ModuleNotFoundError: No module named 'matplotlib' upon running the EXE.