The JET Blue Story
The Exchange Server product used the Extensible Storage Engine (ESE, aka JET Blue) to store data for decades. The story of the JET Blue (in contrast to JET Red which is used for the Access database) can be read here (https://en.wikipedia.org/wiki/Microsoft_Jet_Database_Engine). In the ancient days of data storage, the ESE database was the best choice for storing mostly unstructured data with many dynamic properties.
The Messaging API (MAPI) was developed in the 1990s to provide programmers with unified interfaces for easier message exchange. The current Outlook 2013 MAPI Reference has replaced the MAPI documentation at TechNet. It is not easy to find reliable resources about the original MAPI implementation in today’s world. The only printed resource is Inside Mapi (Microsoft Programming Series), ISBN 978-1572313125, published in 1996.
At Ignite 2015, Ross Smith VI joked about moving the Exchange storage engine to SQL. This was true back in the day, with Exchange 2013 in production and Exchange 2016 coming. But Ross laid the tracks for the evolution of Exchange.
But it seems that the Exchange Product Team realized that in today’s world with heavily standardized communication and less dynamic requirements than in the 1990s, the days of JET blue are over. At the same time, SQL Server evolved into a mature database solution capable of handling big data. The question was, if it can store SharePoint data, why not Exchange data? After twenty years of Exchange Server using the good ole ESE engine, it was time to move on.
The SQL scripts that Exchange uses to configure SQL are located in $exbin\SQL
Example
CREATE TABLE [dbo].[MAPI_PROPERTIES]( [MAPI_PROPERTTY_ID] [int] IDENTITY(1,1) NOT NULL, [MAPI_PROPERTY_NAME] [nchar](127) NOT NULL, [IsWellKnownProperty] [bit] NOT NULL, [MAPI_TYPE_ID] [int] NOT NULL, CONSTRAINT [PK_MAPI_PROPERTIES] PRIMARY KEY CLUSTERED ( [MAPI_PROPERTTY_ID] ASC) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]) ON [PRIMARY]GOALTER TABLE [dbo].[MAPI_PROPERTIES] WITH CHECK ADD CONSTRAINT [FK_MAPI_PROPERTIES_MAPI_TYPES] FOREIGN KEY([MAPI_TYPE_ID])REFERENCES [dbo].[MAPI_TYPES] ([MAPI_TYPE_ID])GOALTER TABLE [dbo].[MAPI_PROPERTIES] CHECK CONSTRAINT [FK_MAPI_PROPERTIES_MAPI_TYPES]GO
Exchange Server goes SQL
The current Exchange 2016 CU2 Preview supports an undocumented registry key to activate SQL Server support for Exchange. I do not know if this was supposed to be officially included in a public release. So maybe the SQL support was made available by error and is already removed from the most current build again.
The famous SqueakyLobster registry key has been used in Exchange 5.5 to troubleshoot performance issues. The new “Lobster” key activates the hidden code in the Exchange Server product. The name of the key is LobsterMapiDB.
This key activates support for Exchange modern storage. Without this key, you won’t be able to move mailboxes from ESE legacy storage to modern SQL storage.
It is assumed that a SQL Server 2014 instance is available. A SQL Server 2014 Express edition is sufficient for testing purposes.
Note:
Any configuration changes or the registry should be validated in a test environment first. Never try this in production right away.
The high-level steps required to activate SQL support for Exchange 2016 are:
- Create a configuration file to provide the SQL connection string
- Create a SQL server login for the Exchange Trusted Subsystem security group
- Add a registry key to the local Exchange Server registry
- Restart Information Store Service (MSExchangeIS)
- Execute PowerShell script to migrate mailboxes to SQL
The detailed steps are as follows:
- Create a new config file named Microsoft.Exchange.Data.SQL.exe.config in $exinstall/bin
- Create a SQL login for Exchange Trusted Subsystem
CREATE LOGIN [DOMAIN\Exchange Trusted Subsystem] FROM WINDOWS
- Create a new DWORD named LobsterMapiDB in
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15
and set the data value to 1
- Restart Exchange Information Store
Restart-Service MSExchangeIS
- Use Move-MailboxToModernStorage.ps1 script to move selected mailboxes to modern storage
More can be found here:
- Migration scripts and config files are available at Github, https://github.com/Apoc70/LobsterMapiDB
Links
- From crush to product documentation: The story of Squeaky Lobster, http://blogs.technet.com/b/exchange/archive/2006/06/15/3394229.aspx
- Setting “Squeaky Lobster” performance counter settings, https://support.microsoft.com/en-us/kb/556030
- Extensible Storage Engine, https://en.wikipedia.org/wiki/Extensible_Storage_Engine
- Outlook 2013 MAPI Reference, https://msdn.microsoft.com/en-us/library/cc765775.aspx
- SQL Server 2014, https://www.microsoft.com/en/server-cloud/products/sql-server/
- Final information: https://goo.gl/QiTtDo
Enjoy Exchange for the next 20 years…
