The problem on Windows Outlook

Setup address book (no GAL). e.g., ldapx.ends2ends.com [2]

Try to send encrypted email to someone in the address book that you do not have keys for already. e.g., natsec@dc.org (me)

After Outlook connects to ldap server and requests and gets userSMIMECertificate;binary attribute, it fails

Above works for userCertificate;binary attribute if intermiedate certs are pre-installed (follow AIA cert chain) or not used (eg 9Folders email client, etc..). Also works for Outlook for Mac address book.

Solution (thanks to Gary Nebbett)

$ cmp -lb original patched
 12419  70 8     14 ^L
 12420 150 h    154 l
Hex:
00003083 38 0C
00003084 68 6C

$ cmp -lb original patched
 16586 173 {    313 M-K
 16587 174 |    200 M-^@
Hex:
000040CA 7B CB
000040CB 7C 80

$ cmp -lb original patched
 15857 200 M-^@ 130 X
 15858  26 ^V    32 ^Z
Hex:
00003DF1 80 58
00003DF2 16 1A

openssl smime -sign -in empty.txt -text -outform der -out userSMIMECertificate.der -nodetach -signer mycert.pem -inkey mykey.pem -certfile intermediatecerts.pem

Alternative Fix

[HKEY_CURRENT_USER\software\policies\microsoft\office\16.0\outlook\security]
"enableaiacertextension"=dword:00000001

Making it all work.

1. Either 1) apply the 2-byte patch to your EMABLT32.DLL to see userSMIMECertificate;binary; or 2) allow Outlook to chain (follow AIA path) from the cert in userCertificate;binary by making the Registry fix.
2. Start Outlook
3. Add your ldap server to the Outlook Address Books. You can use ldapx.ends2ends.com[2] (no SSL and default Base) for testing.
4. Restart Outlook
5. If you installed the patch correctly or alternate fix, you should be able to send an encrypted email to natsec@dc.org (me) or checkme@dc.org. You should also be able to send encrypted emails to anyone using [2] or whatever ldap/AD server you have set.

• [1] ldap.ends2ends.com This is a ldap-2-dns gateway with dnssec validation and hence securely scales globally. Send a signed email (no body) to checkme@dc.org which should respond with the dns record you need to paste into your dns zonefile. Your dns must be secured with dnssec.
• [2] ldapx.ends2ends.com This is an example of a ldap "key-server" similar to what pgp does. Send a signed email to ldapx@ends2ends.com. In approx 5 min your public key data (userSMIMECertificate;binary and userCertificate;binary) will be publicly available to anyone that includes this address book and they will be able to send you encrypted email w/o first exchanging cetificates. This one is just for test so it does not do a full certificate validation before accepting new certs/signatures. A production version would validate certs chains to CAs before publishing.
• userSMIMECertificate as presented here and in [1]+[2] works out of the box for Outlook for Mac, 9Folders, .. many more w/o bug fix. Works with MS Outlook w/ bug fix.
• A Win10 batch script is here to help automate the patch. For safety, you have to take the patched DLL file created and replace the original one yourself as Administrator.

• Exchange GAL pkcs7 (nodetach) https://docs.microsoft.com/en-us/openspecs/exchange_server_protocols/ms-oxoabk/2653a83f-ace5-4543-8908-d6557e2f6ed6
• AIA sledgehammer https://support.microsoft.com/en-us/topic/ms13-094-description-of-the-security-update-for-outlook-2010-november-12-2013-c4ff9950-d9cf-84cc-4cc8-125451518520
• Stackoverflow https://stackoverflow.com/questions/67172459/what-exactly-is-the-contents-of-the-ldap-usersmimecertificate-attribute
• Old stackoverflow https://crypto.stackexchange.com/questions/60537/what-exactly-is-the-contents-of-the-ldap-usersmimecertificate-attribute
• Microsoft Docs https://docs.microsoft.com/en-us/answers/questions/372996/working-sample-of-ldap-usersmimecertificate-attrib.html
• Actalis free s/mime cetificates. https://www.actalis.it/en/certificates-for-secure-electronic-mail.aspx

copy EMABLT32.DLL foo2.dll
certutil -encodehex -f foo2.dll foo2.dll.hex 12 1>NUL
@PowerShell "(GC .\foo2.dll.hex)|%%{$_ -Replace 'eb0cbee8780110','eb0cbe0c7e0110'}|SC .\foo2.dll.hex.mod"
certutil -decodehex -f foo2.dll.hex.mod foo2_mod.dll
copy foo2_mod.dll EMABLT32.DLL

Here is Ghidra disassembly of where the incorrect ref to "userSMIMECertificate" is:

int __thiscall FUN_10003c6b(void *this,uint param_1,uint *param_2,undefined4 param_3)
{
  uint *puVar1;
  char cVar2;
  byte bVar3;
  bool bVar4;
  int *piVar5;
  int iVar6;
  char *pcVar7;
  char **ppcVar8;
  char *pcVar9;
  uint uVar10;
  int iVar11;
  char *pcVar12;
  uint local_1c;
  uint local_10;
  int local_8;
  
  *param_2 = param_1 & 0xffff000a | 10;
  pcVar12 = (char *)0x0;
  param_2[2] = 0x8004010f;
  ppcVar8 = *(char ***)(*(int *)((int)this + 8) + 0xc);
  if (param_1 == 0x3a701102) {
    pcVar12 = "userSMIMECertificate"; <---Change to "userSMIMECertificate;binary"
LAB_10003ce1:
    bVar4 = true;
  }
  else {
    if (param_1 == 0x8c6a1102) {
      pcVar12 = "userCertificate;binary";
      bVar4 = true;
      piVar5 = (int *)FUN_10008e74(ppcVar8,"userCertificate;binary");
      if (piVar5 != (int *)0x0) goto LAB_10003ce8;
      pcVar12 = "userCertificate";
      ppcVar8 = *(char ***)(*(int *)((int)this + 8) + 0xc);
      goto LAB_10003ce1;
    }
    bVar4 = false;
  }
  piVar5 = (int *)FUN_10008e74(ppcVar8,pcVar12);
LAB_10003ce8:
  if ((pcVar12 == (char *)0x0) || (piVar5 == (int *)0x0)) {
    iVar6 = 0x40380;
  }
...