BatchTest File Formats

A BatchTest uses a XML config file to describe the test(s) to run, and it outputs results in one of several formats. Both files are emailed to a specified Delivery-To address as attachments. The config file format is designed to be machine created, i.e. from a database or from an Excel workbook. The results formats allow the results to be machine parsed.

XML Config File Format

While there is just one config file format, many of the elements are optional or default to the right thing.

Simple Config File Format

The simplest form of the config file looks like this (user values in bold):


<BatchTest TestType="receiver">
  <Target>test@checktls.com</Target>
  <Delivery>
    <To>support@checktls.com</To>
    <Format>csv</Format>
  </Delivery>
</BatchTest>

See this XML File in a browser window. This file describes one simple test:

<BatchTest TestType="receiver">
Top level element that opens the XML. The TestType attribute makes this a Receiver test.
<Target>
The email address to be test, same as the Email field on the interactive Receiver test.
<Delivery>
Starts a group of elements that describe where and how to send the test results.
<To>
The email address to which results should be sent (because it's under <Delivery>). Be sure to add @CheckTLS.com to your approved senders list to make sure the email gets through.
<Format>
Indicates which of the output formats to use to report results -- see Results Formats below.
</Delivery>
Ends the group of elements describing where and how to send the test results.
</BatchTest>
Ends the XML.

Complete Config File Format

While the simple XML format above works for most testing, the config file has options to roll-up multiple tests into a single number, suppress results if they are good enough, and test private email systems. A complete config file looks like this (user values in bold):


<BatchTest TestType="thru" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.CheckTLS.com BatchTest.xsd">
  <Target Count="2" Host="mail2.checktls.com"  Port="587" AuthType="PLAIN" AuthUser="support@checktls.com" AuthPass="secret" SMTPTimeOut=30" ReplyTimeOut="10">test@forcetls.com</Target>
  <Delivery HidePasswords="false">
    <To>support@checktls.com</To>
    <Format>csv</Format>
    <Suppress Function="minimum" Test="ge" Value="100" />
  </Delivery>
  <Total Function="average" />
</BatchTest>

See this XML File in a browser window.
The XML Schema Definition (XSD) for BatchTest XML files is here: BatchTest XML Schema (xsd)

Target

The <Target> element has attributes that can set all the options from Custom/Private Systems. These include a specific MX host, an email port other than the standard port 25, different methods of authenticating to the mail system, and a specific user-id and pass-code for access. There is also an attribute to quickly test a single target address many times in succession. The <Target> attributes as shown above are:

Count
The number of times to test this <Target> in rapid succession; useful to stress test an email system/server or test how various MX servers are being selected.
Host
A single host to test, typically an MX server through which email is routed. This is different from targeting a specific mail host: targeting "user@host.domain.com" sends the email to "user" on "host.domain.com", whereas this Host attribute tests "user@domain.com" as delivered to mx1.domain.com.
Port
A non-standard (i.e. not 25) port on which the email server for the <Target> is listening. Non-standard ports are used in "hidden" email systems that are not open to the public, but are Internet-connected.
AuthType
The type of Authorization that this test should use when authenticating to the email server. Valid options are: PLAIN, LOGIN, CramMD5, NTLM.
AuthUser
The user-id to authenticate with.
AuthPass
The pass-code to authenticate with.
SMTPTimeOut
Tells the test how long to wait (in seconds) for the SMTP server to respond to a command. See TimeOut in TestReceiverFull.
ReplyTimeOut
Only used with BatchTest Thru; tells the test how long to wait (in minutes) for the email to come back. See BatchTest Thru.
Hide Passwords

BatchTest can hide the authentication password for email servers it tests so results reports can be shared without compromising server security. Add the attribute "HidePasswords" to the Delivery element and set it equal to "true" to suppress all the passwords in the results of this test.

Totals

BatchTest can compute two types of totals for all the <Target>s in a given test. This is obviously most useful when testing more than one email address at a time, although single <Target> totals are useful for suppressing results (next section). Total functions can be "average" or "minimum", or two total elements can be used to get both.

Suppressing Results

IT departments with their own monitoring system (e.g. Nagios) always want their BatchTest results sent. They monitor the test results and follow the correct notification sequence depending on the results. This is one of the primary reasons BatchTest results are available in machine-parsable XML. See MonitorTLS to do this as a web service rather than from email.

Other users only want to know if something is wrong. For these, BatchTest has the option to not send the results based on user supplied conditions. The <Suppress> element under the <Delivery> node defines when to not send the results email.
The different <Suppress> attributes are:

Function
Either "average" or "minimum" to select which total to compare.
Test
One of:
"ge" Greater than or Equal to
"gt" Greater Than
"eq" Equal to
"ne" Not Equal to
"le" Less than or Equal to
"lt" Less Than
Value
The given number to compare the total to.

The attributes in <Suppress> read like a sentence:
If [function] is [test] than [value], then don't send the email. For example:
If minimum is greater than 90, then don't send the email. This would only send an email if any of the <Target>s in the Batch had a Confidence Factor that was under 90.

Results File Format

The <Format> element in the config file selects the format of the results file. The different <Format> options are:

text
The results file is a plain text (.txt) file that shows the Confidence Factor and the target address, for example:

100 "onetest@checktls.com"
100 "twotest@checktls.com"
csv
The results file is an Excel spreadsheet (.csv) file that shows the Confidence Factor and the target address, for example:

100,"onetest@checktls.com"
100,"twotest@checktls.com"
xml
The results file is an XML (.xml) file that shows the Confidence Factor and the target address, for example:

<CheckTLS test="BatchTestReceiver">
  <Results>
    <Result>
      <Score>100</Score> 
      <Target>onetest@checktls.com</Target> 
    </Result>
    <Result>
      <Score>100</Score> 
      <Target>twotest@checktls.com</Target> 
    </Result>
  </Results>
</CheckTLS>
xml-score
The results file is an XML (.xml) file that shows the Score level results of the interactive TestReceiver test, for example:

<CheckTLS test="BatchTestReceiver">
  <Results>
    <Result>
      <EMailAddress>testone@checktls.com</EMailAddress> 
      <ConfidenceFactor>100</ConfidenceFactor> 
      <MXConfidenceFactor>100</MXConfidenceFactor> 
      <MXCount>1</MXCount> 
  </Result>
    <Result>
      <EMailAddress>testtwo@checktls.com</EMailAddress> 
      <MXConfidenceFactor>100</MXConfidenceFactor> 
      <MXCount>1</MXCount> 
  </Result>
</Results>
</CheckTLS>
xml-matrix
The results file is an XML (.xml) file that shows the Matrix level results of the interactive TestReceiver test, for example:

<CheckTLS test="BatchTestReceiver">
  <Results>
    <Result>
      <EMailAddress>test@checktls.com</EMailAddress> 
      <ConfidenceFactor>100</ConfidenceFactor> 
      <Connected>100</Connected> 
      <Allowed>100</Allowed> 
      <SignedOn>100</SignedOn> 
      <TLSAvailable>100</TLSAvailable> 
      <NegotiatedTLS>100</NegotiatedTLS> 
      <CertOK>100</CertOK> 
      <SenderOK>100</SenderOK> 
      <ProofedAddress>100</ProofedAddress> 
      <MXCount>1</MXCount> 
      <MX exchange="www.checktls.com" preference="20">
        <Connected>0.020458</Connected> 
        <Allowed>2.00779</Allowed> 
        <SignedOn>2.012302</SignedOn> 
        <TLSAvailable>2.017546</TLSAvailable> 
        <NegotiatedTLS>4.275149</NegotiatedTLS> 
        <CertOK>4.258508</CertOK> 
        <SenderOK>4.328082</SenderOK> 
        <ProofedAddress>4.334522</ProofedAddress> 
      </MX>
    </Result>
  </Results>
</CheckTLS>
xml-matrixssl
The results file is an XML (.xml) file that shows the Matrix level results of the interactive TestReceiver test along with information about the SSL Certificate used.
xml-detail
The results file is an XML (.xml) file that shows the Detail level results of the interactive TestReceiver test, for example:

<CheckTLS test="BatchTestReceiver">
  <EMailAddress>test@checktls.com</EMailAddress> 
  <ConfidenceFactor>100</ConfidenceFactor> 
  <Connected>100</Connected> 
  <Allowed>100</Allowed> 
  <SignedOn>100</SignedOn> 
  <TLSAvailable>100</TLSAvailable> 
  <NegotiatedTLS>100</NegotiatedTLS> 
  <CertOK>100</CertOK> 
  <SenderOK>100</SenderOK> 
  <ProofedAddress>100</ProofedAddress> 
  <MXCount>1</MXCount> 
  <MX exchange="www.checktls.com" preference="20">
    <Connected>0.023249</Connected> 
    <Allowed>0.102055</Allowed> 
    <SignedOn>0.103797</SignedOn> 
    <TLSAvailable>0.105284</TLSAvailable> 
    <NegotiatedTLS>0.517981</NegotiatedTLS> 
    <CertOK>0.503749</CertOK> 
    <SenderOK>0.557888</SenderOK> 
    <ProofedAddress>0.562938</ProofedAddress> 
    <detail>
      <![CDATA[ 
[000.023]   Connected to server
[000.102]<--220 www.checktls.com ESMTP Sendmail 8.14.4/8.14.4; 
            Fri, 12 Nov 2010 13:46:30 -0500
[000.102]   We are allowed to connect
[000.102]-->EHLO checktls.com
[000.104]<--250-www.checktls.com Hello www.checktls.com 
            [192.168.1.165], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH DIGEST-MD5 CRAM-MD5
250-STARTTLS
250-DELIVERBY
250 HELP
[000.104]   We can use this server
[000.104]   TLS is an option on this server
[000.104]-->STARTTLS
[000.105]<--220 2.0.0 Ready to start TLS
[000.105]   STARTTLS command works on this server
[000.505]   Cipher in use: DHE-RSA-AES256-SHA
[000.505]   Cert Authority: /C=US/ST=Arizona
            /L=Scottsdale/O=GoDaddy.com, Inc.
            /OU=http://certificates.godaddy.com/repository
            /CN=Go Daddy Secure Certification Authority
            /serialNumber=07969287
[000.506]   Cert Owner: /O=www.checktls.com
            /OU=Domain Control Validated/CN=www.checktls.com
[000.506]   Cert CommonName: www.checktls.com
[000.506]   Cert SubjectAltName: 2/checktls.com
[000.507]   Cert Hostname VERIFIED (www.checktls.com)
[000.507]   Cert chain VERIFIED (0 (ok))
[000.507]   Connection converted to SSL
[000.507]~~>EHLO checktls.com
[000.518]<~~250-www.checktls.com Hello www.checktls.com 
            [192.168.1.165], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH DIGEST-MD5 CRAM-MD5 LOGIN PLAIN
250-DELIVERBY
250 HELP
[000.518]   TLS successfully started on this server
[000.519]~~>MAIL FROM: <test@checktls.com>
[000.558]<~~250 2.1.0 <test@checktls.com>... Sender ok
[000.558]   Sender is OK
[000.558]~~>RCPT TO: <test@checktls.com>
[000.563]<~~250 2.1.5 <test@checktls.com>... Recipient ok
[000.563]   Recipient OK, E-mail address proofed
[000.563]~~>QUIT
[000.569]<~~221 2.0.0 www.checktls.com closing connection
      ]]> 
    </detail>
  </MX>
</CheckTLS>
xml-certdetail
The results file is an XML (.xml) file that shows the CertDetail level results of the interactive TestReceiver test (similar to xml-detail above but with more info about the certificates used).
xml-ssldetail
The results file is an XML (.xml) file that shows the SSLDetail level results of the interactive TestReceiver test (similar to xml-certdetail above but with details of the SSL "conversation" used to make the connection displayed in their unencrypted form).
pager
The results are sent directly in the body of the email rather than in an attached file and include the Confidence Factor, BatchTest total(s), and the Matrix level results of the interactive TestReceiver test in a very terse format so it fits on a small smart phone screen (i.e. pager), for example:

EMail:CFa:Conne:Allow:Signe:TLSAv:CertO:Negot:Sende:Proof
support@checktls.com:100:100:100:100:100:100:100:100:100
average: 100
pager-html
Same as pager above but formatted in HTML, for example:
EMailAddress ConfidenceFactor Connected Allowed SignedOn TLSAvailable CertOK NegotiatedTLS SenderOK ProofedAddress
support@checktls.com 100 100 100 100 100 100 100 100 100
average: 100

Create XML Config Files With Excel

While config files are designed to be machine created, i.e. from a database, we also provide two Excel worksheets that let you create config files manually: "simple" and "complete".

The simple workbook creates Simple Config File Format files. This Simple Format is used for all but the most specialized test cases.

The complete workbook creates Complete Config File Format files. This Complete Format includes every option and parameter available.

Both workbooks work the same way and contain instructions for how to use them. The "complete" workbook just has more rows and more columns to cover all the possible entries.

To use a workbook, load it in Excel and maximize the window so you can see everything at once. If you open the workbook directly from the link on CheckTLS.com, newer versions of Excel will open the workbook in "Read-Only" or "Compatibility" mode. You can safely turn this off. The workbook does not include any macros or Visual Basic code. All it does is use Excel's built-in XML Map ability to map cells in a spreadsheet to the BatchTest XSD.

The top of the workbook has the fields there are only one of, like the type of test (receiver or thru), who to send the results to, etc. These are shaded grey. Refer to the config files section above. The blue fields are the elements that may be repeated, typically a list of target email addresses to test in this batch. To add more of these, insert a row in the spreadsheet between the dark blue header and the light blue value row(s). Be sure any new lines you insert are automatically shaded blue; if they are not, the spreadsheet will not map them into the XML file.

Once you've filled in the spreadsheet, follow these steps to create the XML file:

  • Open the Developer menu (how to display Developer Tab in Excel)
  • Under XML, Choose Export
  • Verify Save as type is XML (*.xml)
  • Browse to where you want to save the XML file
  • Enter a name for the new XML file
  • Click Export

If you get a message about data validation, you have a bad entry in one (or more) fields.

A filled-out "complete" sheet looks like this:

And the XML it generated:


<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<BatchTest TestType="receiver">
        <Target Host="mail1.checktls.com" Port="25" AuthType="PLAIN" AuthUser="S
ecretUser" AuthPass="SecretPassword">test@CheckTLS.com</Target>
        <Target Host="mail2.checktls.com" Port="25" AuthType="LOGIN" AuthUser="S
ecretUser" AuthPass="SecretPassword">test@CheckTLS.com</Target>
        <Delivery HidePasswords="false">
                <To>support@CheckTLS.com</To>
                <Format>csv</Format>
                <Suppress Function="minimum" Test="ge" Value="100"/>
        </Delivery>
        <Total Function="minimum"/>
</BatchTest>