Having investigated and experimented further, I've been able to solve this problem. I've requested a new certificate for an alternate email address from StartSSL and saved it to and exported it from firefox (iceweasel).
Trying to add this new certificate with smime_keys worked out of the box! It seems that the .p12 files I had generated from Apple's keychain application were missing the root and/or intermediate certificates from the bundle. This also explains why I had this problem with all certificates I tried to load. With this new knowledge, I was also able to create and validly add my old keys for signing and decrypting to mutt. That said, given that I was able to manually get my keys working, I think perhaps smime_keys is being too harsh on refusing to load files without a root certificate chain? Both thunderbird and firefox accept these certificates without complaint.
signature.asc
Description: Digital signature