Monday, April 23, 2018

Third Party Transport Agents delaying Emails and Crashing the Transport Databases

I had a customer experiencing extensive delays receiving email through the Exchange Transport stack both for internal and external mail relay.

When we ran the Get-Queue or Get-Message cmdlets, the following error was experienced.

Exchange can't connect to the Exchange Transport service on computer "localhost". Verify that the service is started.


Interestingly, sometimes when we ran the Get-Queue and Get-Message command, the correct output was returned so this issue was intermittent.

During the issue, I noticed that the ExchangeTransport.exe service enters a suspended state.


Looking at the Application Logs on the server, I noticed the Transport ESE Databases (the Sender Reputation and IP Filtering) were constantly being rebuilt.

  • edgetransport (34164) Sender Reputation Database: The database engine is initiating recovery steps. 
  • edgetransport (34164) Sender Reputation Database: The database engine has begun replaying logfile C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\SenderReputation\trn.log. 
  • edgetransport (34164) Sender Reputation Database: The database engine has successfully completed recovery steps. 
  • edgetransport (34164) Sender Reputation Database: The database engine started a new instance (2). (Time=0 seconds) 


Enabling verbose logging on the Exchange server for the Transport Stack we saw that the issue was being caused by third party transport agents.

Get-EventLogLevel "MSExchangeTransport" | Set-EventLogLevel -Level Expert


Log Name:      Application
Source:        MSExchangeTransport
Date:          23/04/2018 3:43:10 PM
Event ID:      10003
Task Category: PoisonMessage
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      COMPUTERNAME
Description:
The transport process failed during message processing with the following call stack: System.Runtime.InteropServices.COMException (0x800706BE): Creating an instance of the COM component with CLSID {9DE1008F-1451-4E2E-8767-A1DAF9BE3560} from the IClassFactory failed due to the following error: 800706be The remote procedure call failed. (Exception from HRESULT: 0x800706BE).
   at ContentSecurity.ExchangeAgents.MailRouteItemSourceContext.processItem(String sender, ArrayList recipients, IStream pRawData, UInt32& pFlags)
   at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.ProcessMailRouteMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e, Boolean& bDeleteFromSource)
   at ContentSecurity.ExchangeAgents.GfiAsRtSubmittedAgent.SubmittedMessage(SubmittedMessageEventSource source, QueuedMessageEventArgs e)
   at Microsoft.Exchange.Data.Transport.Routing.RoutingAgent.Invoke(String eventTopic, Object source, Object e)
   at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.Dispatcher.Invoke(MExSession session)
   at Microsoft.Exchange.Data.Transport.Internal.MExRuntime.MExSession.AsyncInvoke(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Disabling the Exchange Transport Agents for GFI and restarting the MSExchangeTransport service resolved the issue.


If anyone experiences these symptoms, please make sure you check your third party Transport Agents!

Monday, April 2, 2018

Azure AD Domain Services Overview - Removing the need for Domain Controllers in Azure IaaS clouds

Late 2017 Microsoft released some very cool technology in Azure called Azure AD Domain Services.  This service provides Azure Customers with Virtual Machines in Azure the ability to use Domain Services such as Kerberos, NTLM and Group Policy lock down without the need for deploying Domain Controllers in the cloud.

It is important to note, Azure AD Domain Services a paid service, once enabled in your Azure Tenancy, you will be billed monthly.  Azure AD Domain Services unlike other cloud services in Azure cannot be stopped or paused, it must be deleted from the Azure Tenancy to avoid further billing.  To understand how this service is charged, please see https://azure.microsoft.com/en-au/pricing/details/active-directory-ds/

What Azure AD Domain Services offers customers is the ability to remove the need for building domain controllers in the cloud and instead utilize Domain Controllers as a service.  You essentially consume these domain functionality as a service without the need to deploy, manage and patch domain controllers in the cloud.

As companies adopt cloud technologies, we want to transition our workloads into Software as a Service (SaaS) offerings.  What I tell my customers is 95% of what your IT department maintains are the applications and services provisioned within the virtual machines.  For on-premises deployments the Virtualization Layer, Servers, Storage and Network typically get updated every 5 years as part of an Infrastructure Refresh project which is generally done by an IT Integrator.  When transitioning to cloud computing, simply moving your virtual machines from your on-premises environment to an Infrastructure as a Service platform does not generally reduce your IT expenditure and for many clients is more expensive in most cases.  For majority of companies, none of what the IT team works on in a daily basis is revolves around the Virtualization, Servers, Storage and Networking layer!

We want to transition as many services as we can onto Software as a Service offerings where we consume a service but have no need to maintain the virtual machines and the applications which run on these.  Office 365 is a prime example of a Software as a Service offering, you consume Exchange, Sharepoint and Skype for Business services without the need for maintaining anything in the back end.


Well you might have guest it, Azure AD Domain Services is a Software as a Service offering for "Domain Services" removing the need for Domain Controllers.

Azure AD Domain Services utilises the user accounts, security groups and data stored in Azure Active Directory to provision domain services.  It acts as a service which communicates with Azure Active Directory in the back-end.  Servers in an Azure IaaS offering or even on-premises via Express Route or VPN Tunnel can utilise Azure Active Directory for authentication and domain services functionality.


If you have an on-premises Active Directory environment and are looking to move some virtual machine workloads up to the cloud, you can also leverage Azure AD Domain Services which I will explain.

Through utilizing Azure AD Connect (the same tool most of you are familiar with for Office 365 migrations), you can synchronize your user accounts and password hashes up to Azure AD which then can be leveraged by Azure AD Domain Services.

Important: You must synchronize password hashes to Azure AD for Azure AD Domain Services to work correctly.  Azure AD Domain Services cannot authenticate against passwords in an Active Directory Environment using Active Directory Federation Services or Passthrough Authentication.

Important: You can only create one managed domain serviced by Azure AD Domain Services for a single Azure AD directory



Setting up Azure AD Domain Services

In this article I'm going to run through the setup process of Azure AD Domain Services.  In order to be able to setup Azure AD Domain Services you must first have an Azure Tenancy and billing setup.  You will also need setup in place a Resource Group, Virtual Network for Azure AD Domain Services to operate on within Azure and ideally a Virtual Network Gateway to provide connectivity back to on-premises.

The step is to find Azure AD Domain Services from the Azure Market place.


Like creating a real Active Directory Domain in a new Forest, specify the DNS Namespace you want for the new Azure AD Domain Services domain.  Also specify the subscription you want to use, the resource group and the Azure Datacentre.

I'm using Southeast Asia (Singapore) as I get the same latency from my Office to Singapore as what I get to the Azure Sydney datacenter - and it's cheaper to put my infrastructure in Asia!


Next specify the virtual network and server subnet you want to use within Azure.  I created this earlier for other infrastructure residing in my Azure tenancy.


Within the Azure AD Domain Services there is a group called "AAD DC Administrators".  This is our traditional "Domain Admins" we are custom to in Active Directory.  Add at least one account which exists in the Azure AD tenancy we are linking to be an administrator.



Give it some time to deploy.  It took almost an hour to provision within my tenancy.


Once it finishes provisioning, you will see it in your list of resources within the Azure portal.  Go to Azure AD Domain Services and note down the DNS servers.  These will automatically be provisioned within the IP range of your virtual network.


Find the Virtual Network resource you have provisioned earlier.


Update the DNS of the Virtual Network to use the Azure AD Domain Services DNS.


After updating the Virtual Network you will need to reboot your Azure VMs on the network.  Remember the golden rule of Azure is never configure static IP addresses on your VM's.  Use the Azure Portal to configure DNS, Gateway settings and IP addressing for everything!

You should see the DNS settings referencing that of Azure AD Domain Services.


Now you can simply just join the member servers to the Azure AD Domain Services managed domain using an administrative account in the "AAD DC Administrators" group.



Administrating Azure AD Domain Services

As mentioned previously Azure AD Domain Services is Domain Services managed by Microsoft.  As a result, it is not possible to login to the AD Domain Services via Remote Desktop like you can do with Traditional Domain Controllers.  Provided you have administrative rights and are part of the "AAD DC Administrators" group in Azure AD you can however connect to the Domain Services using the Microsoft Management Console (MMC) Active Directory snap-ins which I will show you shortly.

On a member server joined to Azure AD Domain Services, simply add the Active Directory Remote Server Administration Tools.


Then you can simply Open Active Directory Users and Computers and query what is in Azure AD Domain Services.  You will be able to see all the objects which exist in Azure AD which are automatically synchronized.  As you see, my Azure AD has lots of objects! - 2 users... :)

It is important that all changes are conducted through the Azure AD portal.  This includes creating new user accounts, changing attributes, resetting passwords, creating/deleting groups and group membership management.  Any changes you make to the objects in Azure AD Domain Services will be overwritten automatically by Azure AD which is the "authoritative source".


Azure AD Domain Services supports full Group Policy functionality and provides member servers with a SYSVOL for Group Policy Objects.


If you install Group Policy Management Console on a member server you can create and manage Group Policy against your users and member computers.  All Group Policy functionality is available in Azure AD Domain Services.


There are domain controller objects in Azure AD Domain Services.  These are automatically generated and maintained by Microsoft.  You must not touch these objects within the Domain Controllers organisational unit.

Important: Encase you were wondering, you cannot add additional domain controllers to a domain running in Azure AD Domain Services.



Limitations of Azure AD Domain Services

There are a few limitations of Azure AD Domain Services that i'm aware of which I would like to share with you.

  • The Schema is administrated by Microsoft for the managed domain.  Schema extensions whilst possible are not supported by Azure AD Domain Services.  If you want to customize the Schema, create your own domain controllers using the traditional method.  This means no Microsoft Exchange, Skype for Business or other applications which change the Schema.  Use Office 365!
  • Any changes made in Azure AD take approximately 20 minutes to reflect in your managed domain.  This is a bit slow and you will need to be patient.
  • There is no way to pause or stop Azure AD Domain Services.  From a billing perspective make sure your aware of this and are fine paying the monthly fee for this service.  It is cheaper then creating your own virtual machines in Azure and configuring them as Domain Controllers.
  • All group and user changes must be conducted through Azure AD, not Azure AD Domain Services.
  • You cannot add additional domain controllers to Azure AD Domain Services.  As this service is managed by Microsoft, they will automatically provide additional servers should the number of LDAP queries be excessive or you log a support case saying its slow!

Migrating Servers to Azure AD Domain Services


If you have Azure AD Connect in place from your on-premises Active Directory Domain to Azure AD, all your user accounts and groups will be available in the cloud and can be used in Azure AD Domain Services.  This is provided your synchronizing the password hashes for the accounts and not utilizing ADFS or Passthrough authentication.

After you migrate virtual machines up to Azure, if you join them to the new Azure AD Domain Services, you will be able to continue logging into the servers using the same synchronized accounts.  Please note however, Azure AD Domain Services will provide resources with a new Security Identifier (SID) and as a result, some resource access may break.  I'm not aware of any tools at this stage that will deal with this.  In on-premises world we had the Active Directory Migration Tool (ADMT) which performed SID Translation and allowed us to utilise SID History to continue accessing old resources over a transitive forest trust during co-existance.

We can however migrate group policy objects across to Azure AD Domain Services using Group Policy with Group Policy Migration tables.  This will allow you to provide the same policies in Azure AD Domain Services as what you have on-premises without manual reconfiguration.  If you want more information on this google "Group Policy Migration Tables".

I hope this post has been helpful in giving you an understanding of Azure AD Domain Services!

Monday, March 26, 2018

Binding a Certificate breaks IMAP or POP on Exchange - NO LOGIN failed error

A customer of mine running Exchange 2013 CU17 went to renew the Exchange Certificate after it expired.  The customer even though they were not using secure IMAP bound the new certificate to IMAP, POP and IIS.

The customer has a line of business application which uses unsecured PlainTextLogin on the internal network to a single mailbox to receive invoices and email traffic.  When using PlainTextLogin without encryption you want to ensure this is not exposed to the Internet.  In my case, my customer did not have IMAP directly exposed to the Internet and was kept secure.

The following screenshot shows the new certificate installed on the Exchange server bound to IMAP and POP services.


As soon as the customer bound the certificate to POP and IMAP services, it broke IMAP.  From my reading the same symptom is also experienced for POP.

There are many forum threads on the Internet about this issue none with a resolution.  For example:
  • https://social.technet.microsoft.com/Forums/exchange/en-US/b14005e4-6416-463b-91db-a4f14620c9f2/imap-connection-failure-no-login-failed-proxynotauthenticated?forum=exchangesvrclients
  • https://www.experts-exchange.com/questions/28162501/Exchange-2013-IMAP-authentication-fails.html
  • https://confluence.atlassian.com/jirakb/unable-to-authenticate-against-microsoft-exchange-imap-server-after-upgrade-jira-376839181.html
  • https://serverfault.com/questions/760417/imap-issue-proxynotauthenticated
  • https://community.spiceworks.com/topic/1980834-exchange-2016-can-t-authenticate-with-pop-or-imap

Symptoms of the Issue

The Test-ImapConnectivity command returned the following error:

RunspaceId                  : ecef0b6b-183e-4211-929d-e046af7c062f
LocalSite                   : Default-First-Site-Name
SecureAccess                : False
VirtualDirectoryName        :
Url                         :
UrlType                     : Unknown
Port                        : 993
ConnectionType              : Ssl
ClientAccessServerShortName : Server
LocalSiteShortName          : Default-First-Site-Name
ClientAccessServer          : Server.domain.local
Scenario                    : Test IMAP4 Connectivity
ScenarioDescription         : Connect to server using IMAP4 protocol, search for the test message, and delete it along
                              with any messages that are older than 24 hours.
PerformanceCounterName      : ImapConnectivity-Latency
Result                      : Failure
Error                       : IMAP Error: aYKG NO LOGIN failed.
UserName                    : administrator
Latency                     : 00:00:00.0368441
EventType                   : Error
LatencyInMillisecondsString :
Identity                    :
IsValid                     : True
ObjectState                 : New

The IMAP4 Protocol Logs on the Exchange Server showed the following HealthMailbox error over and over again:

22T07:30:46.994Z,0000000000000003,2,127.0.0.1:993,127.0.0.1:32258,HealthMailbox1f3addd86ba64b7c84868136b32a82f9,1196,78,97,login,HealthMailbox1f3addd86ba64b7c84868136b32a82f9@DOMAIN.COM *****,"R=""z NO [Error=ProxyNotAuthenticated Proxy=EXCHANGESERVER.DOMAIN.COM:1993:SSL] LOGIN failed."";Msg=Proxy:EXCHANGESERVER.DOMAIN.COM:1993:SSL;ErrMsg=ProxyNotAuthenticated"

Microsoft Outlook clients configured using IMAP would prompt for authentication in an infinite loop despite entering the correct details.

Get-ServerHealth would return all IMAP services on the Exchange server in an unhealthy state.

When performing a Telnet to the IMAP Service using Plain Text Login I would get "NO LOGIN failed" also visible in the IMAP protocol log.



Issue Explanation

For this issue to take effect you must have two things consistent in your Exchange environment.
  1. You must be using PlainTextLogin instead of SecureLogin
  2. You must have bound a custom certificate to IMAP.

Also unfortunately if you bind a custom certificate to IMAP, you cannot unbind it according to Microsoft - great feature.


https://technet.microsoft.com/en-us/library/aa997231.aspx

So how do we fix this, what happened?


The X509CertificateName should be the common name of the certificate that's enabled for IMAP.

By default the X509CertificateName is the FQDN of the server, server.domain.com (whatever it is called in your environment.

As soon as we bound the certificate to IMAP, Exchange Automatically updates the X509CertificateName to the common name of the new certificate.  Now if you have SecureLogin enabled, this does not cause an issue.  However if your using PlainTextLogin, as soon as you bind the certificate IMAP breaks even if we are connecting to unsecured IMAP on 143 (secure IMAP is on 993).

If your using PlainTextLogin IMAP or POP must be set to reference the default "self signed certificate" generated by the Exchange installation process which matches the FQDN on the service.

To reset the IMAP service to reference the default self signed certificate, we simply need to change the X509CertificateName back to the FQDN of the server.

Set-ImapSettings -X509CertificateName ServerFQDN

After making this change restart the IMAP4 backend and frontend services.

Test again using an IMAP client or Telnet.  We can see that it now works as it is referencing the self signed issue.


Final Thoughts

In theory because the X509CertificateName matched the common name on the new certificate, and because we weren't even using secure IMAP at all, it should have worked.  It appears to be a bug in the source code around how these services were coded.

I hope this post has been helpful to other people experiencing this same issue.

Saturday, March 24, 2018

SCCM - TS environment is not initialized

I was in a process of upgrading a customer to the latest SCCM 1710 build for rolling out Windows 10 across the companies network.

To upgrade to SCCM 1710 you must first upgrade to SCCM 1702 then do the next upgrade in the SCCM console.

While upgrading a client to SCCM build 1702, we ran into an issue where the following error was generated in the smsts.log file on the SCCM Distirbution Point.  This error was experienced when imaging a client with our existing Windows 7 deployment task sequence via PXE Boot.

"TS environment is not initialized"


On the client being imaged, the following error was experienced:

The Windows Boot Configuration Data (BCD) file from the PXE server does not contain a valid operating system.  Ensure the server has boot images installed for this architecture.

Error Code: 0xc0000098


This issue occured because I installed The Windows Assessment and Deployment Kit (Windows ADK) version 1709 on the server.

SCCM 1702 does not support ADK 1709.

To fix the issue, I simply performed the next step of the upgrade - using the SCCM console to perform an in-place upgrade of SCCM to 1710.

For future reference:

If your running SCCM 1702, make sure you run ADK 1703.

If your running SCCM 1710, use ADK 1709.
\
Hope this post was helpful.

Friday, March 16, 2018

Remote Desktop Services 2016 Microsoft Office Not Aligning

I came across an issue with Remote Desktop Services 2016 on Server 2016 where Microsoft Office applications did not align properly when deployed via a RemoteApp.

A small bar on the left and top of the applications is present and whilst not a big deal, was annoying for users.  I have shown this in the following screenshot, notice the black bar on the top and left of the window.


This issue was flagged as a bug in Server 2016.  Microsoft released the fix under KB4013429 along with a whole lot of other updates. 

https://support.microsoft.com/kb/4013429

The patch can be downloaded from here:

http://download.windowsupdate.com/d/msdownload/update/software/secu/2017/03/windows10.0-kb4013429-x64_ddc8596f88577ab739cade1d365956a74598e710.msu

After installing this patch and rebooting the server, the users will not longer experience this issue.

Hope this post has been helpful.

Thursday, February 15, 2018

KMS Activation Error 0xc004f074

I recently had issues deploying a new KMS server to activate Windows 10 LTSB 2016 at a customer.  We deployed LTSB to remove all the bloatware from the Windows Store, Edge and other unwanted items like candy crush saga which Microsoft believes enterprise organisations should have in an enterprise version of Windows!

After setting up the KMS server, the following error were experienced on the endpoints:

We can't activate Windows on this device because we can't connect to your organisation's server. Make sure that you're connected to your organisation's network and try again. If you continue having problems with activation, contact your organisation's support person. Error code: 0xC004F074.


Activating Windows(R), EnterpriseS edition
Error: 0xC004F074 The Software Licensing Server reported that the computer could not be activated. No Key Management Service (KMS) could be contacted. Please see the Application Event Log for additional information.


License Activation (slui.exe) failed with the following error code:
hr=0C004F074

Event ID: 8198
Source: Security-SPP


The KMS Server we deployed was running "Windows Server 2012 R2 Datacentre edition".

The key we used from our Microsoft VLSC portal was "Windows Srv 2012R2 DataCtr/Std KMS for Windows 10" - the key recommended for Activating Windows 10.

After liaising with Microsoft with regards to the error, Microsoft advised me that the KMS Key "Windows Srv 2012R2 DataCtr/Std KMS for Windows 10" can only be used for "Windows 10 LTSB 2015".

In order to activate Windows 10 LTSB 2016, you need to use the "Windows Svr 2016 DataCtr/Std KMS" Key.  This will also activate all other builds of Windows 10, Windows 8.1 and Windows 7.  This is what it should look like in your Microsoft VLSC Portal.


In order to install this Server 2016 KMS Key on Windows Server 2012 R2, you must first install the KB3172614 patch.  This can be downloaded from the following URL:


Once this patch is installed, perform the following:

1. Uninstall any current KMS keys on the KMS server with the following command:

slmgr.vbs /upk

2. Install the new "Windows Svr 2016 DataCtr/Std KMS" Key from the your VLSC Portal.

slmgr.vbs /ipk XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

3. Activate the KMS Server

slmgr.vbs /ato

4. Check the key was installed correctly

slmgr/vbs /dlv



Next on your workstation's install the correct KMS Client Key which can be viewed here:


To install use the same command on the server, just using the KMS Client Key instead:

slmgr.vbs /ipk XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

Then activate it against the KMS Server:

slmgr.vbs /ato

I hope this post has been helpful.

Tuesday, February 13, 2018

Outlook 2016 Connects First Time, Prompts for Password Every Other Time

A customer of mine recently deployed Outlook 2016.  The customer had an issue where they could create a new Outlook 2016 Profile, but after the Outlook Profile was created and Outlook was closed, the second time they open Outlook they would be prompted for password in an infinite loop.

Disabling Cached Exchange Mode on the Outlook Profile would fix the issue but we needed Cached Exchange mode to be enabled!

The protocol we were using is MAPI over HTTPs.

Outlook 2010 and 2013 connected fine with MAPI over HTTPs, only Outlook 2016 had this issue.

The issue turned out that the customer had disabled oAUTH on the MAPI over HTTPS virtual directory.

Below is the authentication settings setup on the customers Exchange 2016 server:


This is what the default authentication settings look like:


To fix the issue we needed to set the settings back to default with the following command:

Get-MapiVirtualDirectory | Set-MapiVirtualDireectory -IISAuthenticationMethods NTLM,OAuth,Negotiate -InternalAuthenticationMethods NTLM,OAuth,Negotiate -ExternalAuthenticationMethods NTLM,OAuth,Negotiate

Make sure you reset the correct IIS App Pool or do an iisreset so the change takes effect immediately.

Monday, January 29, 2018

Finding a Device in SCCM for Unknown Computer TS Deployment

1. Currently I have a task sequence deployed to the "All Unknown x86/ x64 Computers" device collection. SCCM will only PXE boot machines it has never seen before in its database.



2. I PXE boot the new client PC and it downloads the policy and completes imaging successfully.

3. At this point SCCM knows about this computer via MAC address but lets say I need to reimage the client computer later on in life or right now as a test.

4. When trying to reimage I get the error that fails on "looking for policy" and then it aborts.


5. The reason why you cannot reimage it is because SCCM can tell that it was already imaged with the same exact MAC address and task sequence name. This is by design.

6.  What do to do reimage it?  You have two options.

a. You could add known device collections to the task sequence.

b. Alternatively you can remove the device from SCCM so it is treated again as an unknown device.

How do you find the device though if you don't know the previous hostname?

Look at the SMSPXE.log file on the SCCM Distribution Point.  Find the "Client boot action reply" entry in the log file and get the ItemKey number.


Now search for the ItemKey in SCCM "Assets and Compliance" --> Devices.


Deleting the device asset from SCCM will then once again treat the resource as an unknown computer.

Hope this post has been helpful.