Thursday, December 24, 2015

Route 5: DNS and Network Enumeration

Do you remember our last day before the break, how there were some problems and at they asked students to not begin any exams? All exercises stopped working, and we waited for an hour or so for the fix. In the end, when service was restored, the facilitator mentioned that the problem originated from a DNS problem.

If you look back the post on Route 3, a DNS problem could be the direct cause of no response from [vcenter].  We may just have the wrong IP.

I mean, I don't know their network, and I certainly don't know which IP should be the vcenter server.

But, I bet we can figure it out if we allow ourselves to do some network enumeration.

Using a for loop, we can scan the /24 network.

for /L %g in (1,1,254) do ping -n 1 x.x.x.%g >> ne_result.txt

This command could take a few minutes to complete.  There are probably faster ways to do this.  A program that would send out all of the pings at once, instead of waiting for a response before sending the next would work quicker.  But, meh.  A command line for loop is good enough for me.

Searching through the ne_result.txt file for 'reply' I've found that there are 14 active boxes on the /24 network.

With a list of active IPs, we can check to see if any of them respond to vcenter's administration port 9443.  Two of the fourteen boxes were already identified in Route 3, leaving twelve left to scan.

nmap -sS [ip] -p 9443

This scan does a dns probe as well, so I got a dns name for all but one of the responding boxes. 
  1. [ad2] is filtered
  2. [chat] is closed
  3. [media] is closed
  4. [report] is filtered
  5. [esxi3] is filtered
  6. [esxi4] is filtered
  7. [esxi5] is filtered
  8. [esxi6] is filtered
  9. [esxi7] is filtered
  10. [esxi8] is filtered
  11. [unk] is closed 
  12. [metting] is closed
Unfortunatly, nothing responded on port 9443.  This indicates that it is not a DNS problem.  It's more likely that the vcenter box is currently off, or there is an authentication problem on my part.

It's Christmas eve now, and I've got to go put presents under the tree.  Have a great night and thanks for troubleshooting with me!

-dh

Route 4: Use a Different Computer

In the middle of testing out some of my solutions in Route 3 and I realized that there was probably an easier fix.

Just use my wife's computer.

She's not a big fan of me messing with her stuff, so after seeking permission and getting the OK plus a stink eye, I check out her laptop.

Firefox isn't installed.  So I install version 43

Flash is outdated.  I upgrade to version 20

Java is up to date.

AnyConnect isn't installed.  Quickly fixed by visiting the schoolhouse website.

I load up the VPN, connect to the school house and navigate to the certificate trust slide.

It says I already have the certificate installed.  I double check the fingerprint.  It matches what was published in Route 1.

Everything looks great.  I do one more check on firefox's plugins (about:plugins).  It looks good too.

Launch a VM.  Get the black screen for 10 seconds, and the refresh page message right after.

Blegh.

At this point it's pretty clear to me that either this is an operator error and I'm repeatedly doing something stupid, or there is a problem with the schoolhouse.

I've found it's always best to assume I'm the problem.  With no background information, the chances are 50/50 and I'd rather not prove myself an idiot by blaming the school.  I need to go spend more time in the student guides and handouts to see if I missed anything.

Route 3: Using Wireshark to Identify Failing Traffic

Using Wireshark on the AnyConnect interface, I captured the process of logging into the schoolhouse, selecting a course, and opening up a VM.  The packet capture included over 900 packets.

I'm not in the mood to parse that much material.

To whittle that down to less than 100 packets, I ran a second capture that only encapsulates launching a vm and the subsequent 10 seconds of black screen until the error message.

Analyzing using Follow >> TCP Stream is a must here.

9 TCP conversations are parsed below.

tcp.stream eq 0
  [my_ip]:49246 to [webserver]:80
  HTTP GET /user_section_items/9202/vm_view/420e9af3-8c03-caa9-3ede-8fe07bbbbfe1
  Results in successful file transfer.  HTML file has html, css and javascript.
Javascript has functions to connect to vcenter console.  Includes error text "The 
console has been disconnected. Refresh the page to reconnect."

tcp.stream eq 1
  [my_ip]:49247 to [webserver]:80
  HTTP GET /img/buttonbar-bg.png,
  Results in 404

tcp.stream eq 2
  [my_ip]:49248 to [webserver]:80
  HTTP GET /css/button-leftside.png HTTP/1.1
  Results in 404

tcp.stream eq 3
  [my_ip]:49249 to [webserver]:9292
  HTTP GET /faye?message=%5728[blahblahblah]
  Returns OK & javascript/json object

tcp.stream eq 4
  [my_ip]:49250 to [webserver]:9292
  HTTP GET /faye
  Returns OK, switching protocols to websocket

tcp.stream eq 5
  [my_ip]:49251 to [vcenter]:9443
  SYN never receives response.  Re-transmitted twice.

tcp.stream eq 6
  [my_ip]:49252 to [webserver]:9443
  HTTP GET /faye?message=%3423[blahblahblah]
  Returns OK & javascript/another json object

tcp.stream eq 7
  [my_ip]:49253 to [vcenter]:9443
  SYN never receives response.  Re-transmitted twice.

tcp.stream eq 8
  [my_ip]:49254 to [vcenter]:9443
  SYN never receives response.  Re-transmitted once.

If you noticed the lack of response from [vcenter] like me, you might start feeling like this maybe isn't an user error.  I don't think the http 404 responses are a big deal, they're just to retrieve the image of a button and some button css.  That could easily be left over from the original site construction.

A quick google search of port 9443 identifies it as a vcenter administration port.  Seeing as no response is received, I figured we should see if the box is even up.

C:\>ping [vcenter]

Pinging [vcenter] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for [vcenter]:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

That would seem to be the problem, vcenter doesn't appear to be up.

... at least, that's what I hope.  Remember, we don't trust computers, so it's possible vcenter filters ping requests, and I've maybe not authenticated correctly with the vcenter, so it's ignoring my SYN on port 9443.

But, that's a bit of a stretch, and I'm feeling like is less of an issue of PEBKAC, and more of a school house turned of the [vcenter] problem.

Route 2: Verifying Software Versions

On the classroom help page there is a troubleshooting pdf.  It says: 

Technical Requirements:  We require Firefox 40.0 (or newer), Adobe Flash 19 (or higher), and Java 8 (or higher).

Verifying the technical requirements is easy.  Just:
  • Compare installed programs
    • Open Control Panel >> Programs >> Uninstall Programs 
    • Verify version numbers
  • Check Firefox's plug-ins
    • Navigate to about:plugins
    • Verify version numbers
Here are my version numbers for comparison.
Firefox            43.0.2
Adobe Flash        20.0.0.235
Java               8.0.660.18
Cisco Anyconnect   3.1.12020
 ------------------------

Despite my version numbers seeming OK, I revert to a lack of trust for computers.  Just because the troubleshooting.pdf says 'or higher', doesn't mean that newer versions have been verified to work.  Lets un-install all our required software, search the Internet for those exact versions, and try them!

Firefox 40.0, released November 8, 2015
Adobe Flash 19, released November 10, 2015
Java 8 build 66, released November 16, 2015

 Today's version is 8 build 66.  It was released on November 16, 2015.  That matches the above time frame:
It's also possible we need an earlier build.  Older versions could be found at  
If you use older versions, you'll need to create a free Oracle account.  I didn't find benefit after trying a few older versions, but you may have better luck.

Cisco AnyConnect 3.1
  • Navigate to the schoolhouse site
  • Login using your user credentials
  • Run installer
 ------------------------

After all that work, I noticed that Firefox checks and disables outdated plug-ins. We need to disable the disabling.
  • Navigate to about:config
  • Accept the warning
  • Search for extensions.blocklist.enabled
  • Set to false
  • Restart browser
At this point, I check my VM status and find out that they are still not working.  Downgrading all that software made me vulnerable to web attacks and I didn't get any improvements.  Time to revert back to the updated versions.

Poo.

-dh

Route 1: Checking Certificate Trust.

Navigate to the introduction lesson 1.1 slide 1 to check the certificate.

Firefox tells me the certificate is already installed.  It has always told me this, despite having never installed a certificate.  Perhaps I already trusted the right certificate or maybe the site code is missing something.

As you can tell, I don't really trust computers.  The site doesn't explain which certificate it installed, so I can't verify anything.  I figured the best way to locate the specific certificate was by digging through the site code.  While viewing the same slide:
  • Open Firefox's Developer Console Debugger
  • Search for '!cert'
  • E0M1S1_0000_edgeActions.js contains the line
window.open("https://certs.godaddy.com/repository/gdig2.crt", "_blank");
  • Assume this is the certificate.
  • Check by deleting all Go Daddy certs from Firefox.
    • Select Options >> Advanced >> Certificates >> View Certificates
    • Navigate through the 5 tabs looking for certificates from GoDaddy.
    • Highlight each and then delete all.
  • Refresh the page
  • Click the Cert link.....different prompt!
  • Select 'This certificate can identify websites.' 
If you want to double check your cert fingerprints, see below.

Name:  Go Daddy Root Certificate Authority -G2
SHA-256 Fingerprint:       45:14:0B:32:47:EB:9C:C8:C5:B4:F0:D7:B5:30:91:F7:32:92:08:9E:6E:5A:63:E2:74:9D:D3:AC:A9:19:8E:DA

Name:  Go Daddy Secure Certificate Authority - G2
SHA-256 Fingerprint:

97:3A:41:27:6F:FD:01:E0:27:A2:AA:D4:9E:34:C3:78:46:D3:E9:76:FF:6A:62:0B:67:12:E3:38:32:04:1A:A6

Download location:
   https://certs.godaddy.com/repository/gdig2.crt


Unfortunately, I still get the same error when loading a VM.  It looks like the certificate isn't the problem.

-dh

Screenshots


Here's some verification screen shots showing my OS and hardware, the required certificates and installed program versions.

Operating System and Hardware

Go Daddy Root Certificate

Go Daddy Secure Certificate

Firefox 43 registry key

a
Adobe Flash 19 registry key

Java 8.66 registry key

Cisco AnyConnect 3.1 registry key

about:plugins Java version

about:plugins Flash version