More HomeLab Pitfalls

This is going to be another one of those blogs of the pitfalls I’ve ran into. I really want to get into how to fix specific errors and troubleshooting. However, since I’m trying my hardest to get this lab setup to dig into the new releases that came out a couple weeks ago,

VUM – Update Manager

So if your like me you know that the first thing you do when deploying anything new is to update it. So why not update your hosts? Well here are some findings that I ran across with my 3 node vSAN cluster. First DNS and I know the statement, “99% of all IT issues are DNS issues.” But thats true! In my case I built a standalone DC with DNS before I deployed my VCenter so that I could use my A records to resolve the traffic. This worked great for the VCenter. It didn’t work great for VUM. Turns out that on your ESXI hosts when you have a single DNS DC it must be running on the first DNS server field. If its running on the second, it will not be able to scan the nodes for compliance or reach them. (If you setup vSAN you would already see the errors showing it cant scan when its doing its vSAN health checks). Second iSCSI controller, if its VMware certified, why are we having to deal with the warning before we can run the updates? Well because their really not… Here is a great Blog on it to look through. Pretty simple fix but something that should just work right?

The whole Network thing.

I was pretty sure that my decision for a 2 subnet with the ability to cross-talk was not a bad idea. One for the home lab, and one for the house would make a stable environment. Nope… For this to those out there about to start, I’d say to do your homework or start your home networking solution first. Ubiquity has a ton of great solutions to look into and a lot of blogs out there to help set you up. There are, of course, a lot of other options that do the functions you may want, but try to be as specific as possible and get things setup before your lab devices come in.

For me, I went from 1 wifi router with 0 ability to create subnets, to a wifi router and ER-x from Ubituity. To the full ubiquity setup.jYjmXdNWTmGEVADkHgbokg

However, I found out the best practice is to setup a gateway -> router -> AP. So because I’m missing that middle piece my AP is running at 1/3 of the speed of my gateway(ie. Hardline in the AP I get 450mb down. Now on my AP…Image 4-27-19 at 7.24 AM       Obviously the speed just isn’t where it should be in this configuration. Once I get my ER-X I’ll update as to what the changes look like. But for now, its actually stable. so I cant complain. But to state again. Plan your goals, pre-define the architecture, research and verify the solutions, then implement it. Its a lot more stressful when its your $$ and you don’t have support available on the phone.

Quick pitfalls

  1. Don’t put your VCenter on your vSAN deployment. It “Should” keep working if the VCenter crashes, but its’ not easy to get back. I found that in my configuration it was actually faster to just rebuild… and thats not too fun.
  2. Remember to get your VCenter off your switch before you disconnect your uplinks… U2 actually made it so that if it fails it falls back. Not that I’ve done it on both updates or anything…
  3. When troubleshooting networking issue, having a centralized location for logging makes your life so much easier. Ubiquity gave me that help.
  4. VROPs deployment thats now built into VCenter will only deploy a thick provisioned VM. This can be annoying when trying to move it off and get it to a thin-Provisioned VM for vSAN.

Its sad that all this stuff came about over months of dragging myself through the mire, but now that I’m stable, I hope to start getting into things soon.

I’m thinking of cutting my blogs into shorter quicker blogs more technically focused. It wont be that crazy, but trying to find the best and newest stuff out there is starting to slow me down in terms of just getting content out there. I really want this content to support people and help them in their IT journeys with VMware products. I know I’ve come a long way because of others.

 

HomeLab Rookie – Networking Mis-steps-stakes

So going along with last post about how I am really not great with the administration of vSphere, or the setup(last time was 5.5). Its time to look at the more fundamental stuff, and how bad I am with the Network component.

Addendum: I hope these help someone out there grow themselves. I know I’m growing in leaps and bounds as I learn through doing.

My Goal was to create 2 subnets, one for home, and one for the lab. I want these to be open to each other to an extent(l2) but still be stable(still working on that part)

Ubiquity

So I decided to grab some ubiquity networking pieces to start. I grabbed the Ubiquity Edge Switch to go along with my Netgear Nighthawk router. I was looking for VLAN capabilities, and my goal was to setup the subnets on the router and then pass them through on the switch.

Learning point 1: VLANs

So its worth pointing out that VLANs on the features of a product don’t really mean SUBNET/VLANs. This kinda bit me in the butt a while, because I ended up trying to create one basic subnet and tried to create a VLAN with a different subnet… No Bueno… In fact when I created VLANs on the router, the whole thing crashed.  However, I found out that the Ubiquity switch I had was a dream to work with (after I updated the firmware). However, I looked and looked and the switch can pass through VLANs, but again, not subnet it.

Learning point 2: Devices

So for anyone looking into doing this, Its worth looking into your ISP and see if more speed is needed. I found out that I was more than doubling my devices and would need to look into my speed usage. It may not be an issue, but for me I found out that for a small figure I’d double my speed. So sure! I got a lot of OVA’s to download anyway 😉

Learning point 3: Unified Management

So there I was swapping from Netgear to Ubiquity and back. Finally I gave in and bought the Ubiquity Edge Router. I went with this one cause the price point didn’t phase me and the functionality of the router looked tremendous. Well, I learned how much this thing could do. I literally love this little box that could. It does the subnet VLANs that I wanted as well as DHCP servers for both subnets. I went with the WAN+2LAN2 connection and set my home to a 192, and my lab to a 10. Oh man, I love this thingubntrouter

The firewall, services, natting, just so much for me to learn in this tiny little box. Once I got this setup, I changed my Netgear into an access point, and set the ISP connection to static from the router. One thing about the Ubiquity Edge Router though, these small boxes use a big plug than so they take up like 2-3 spots in the UPC. Which leads me to the next point.

Learning point 4: Power

If you saw where I am going, it gets better. So every hour or so my whole network would just crash. I’d lose both my LAB and home networks and it would cause some severe anger in my brain(I think I have a couple extra knots in my back from it). I went through SOOO many settings to figure out what it is. I reset the firewall settings(which wasn’t easy considering all I’ve done before was Windows Firewall). Set specific VLAN subnets and reset. Set port forwarding when I couldn’t figure out why I needed to. Well, this went on for about a month(which is also why I have been slacking on posts). I just couldn’t move forward with an unstable lab. Well yesterday I was at the end of my tether. I troubleshooted each device one at a time. During troubleshooting each device my anger boiled. Finally I found out the little box that could was the culprit. It would crash and everything would just die. So I pulled it out to RMA it to get another one, or the gateway(As I hear good things about it). When lo and behold I realized it… I had plugged the central router into a crappy extension cord. *Le Sigh*. Just… no…. If you use a UPC like me and find your missing ports… Get These

Thoughts

I cant express how much I’ve grown doing these things. I’ve figured out so much and learned in this past month more about architecture in the past couple years.

IT is so segregated right now that we lose sight that each feature has to troubleshoot differently, and its really hard. Especially for a Rookie to try to keep swapping gears. I’ve learned from the pure windows standpoint, then PowerShell automation, then vRA. I’ve never been allowed to play with the other parts. But with this Lab, I’m getting to. If your on the edge thinking if a home lab is worth it… It is. Even in a corporate lab, I still wouldn’t learn this much. However, If your not interested in the whole stack, why deal with the trouble right?(and it is Trouble).

RELEASES

This week VRA7.6 was released doing some EXTREMELY needed updates to Orchestrator, Vrops 7.5 was also released plus ESXI 6.7U2 Get to downloading and updating folks! Now go break stuff, and learn how to fix it.

 

A Good Adjustment

I’m busy working on the homlab trying my best to duplicate a homelabber and failing miserably. But more information will be coming on that later.

For now I found a great KB that needs some sharing! VMware has been known for some great pointers to fix issues. This one fell into my lap from an issue I was seeing.

The Problem

So every VRealize Automation environment is different so let me be straight. This change will only help vRO extensiblility actions and automations. For me it was a good improvement over the vRO XaaS workflows that I had published.

We were seeing timeouts and errors showing “Form not found” when trying to open workflows that had actions to pull specific information(AD, VSphere, etc.) because of this the workflows were in the tank and sometimes even IaaS Deployments would return with an error 400.

The KB can be found here: https://kb.vmware.com/s/article/2147109

The Steps:

In Embedded vRealize Orchestrator Server:
  1. Open the /usr/lib/vco/app-server/bin/setenv.sh file using a text editor.
  2. Modify the memory by setting the Xmx and Xms values to the MB value required.For example:

    2.5 GB memory is allocated to each Xmx and Xms(this is the default setting):

    JVM_OPTS=”$JVM_OPTS –Xmx2560m Xms2560m -Xmn896m -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=1024m -Xss256k”

  3. Edit the /etc/vr/memory-custom file using a text editor.
  4. Add this entry:add_service_mem vco-server *NUMBER*

    Note: The *NUMBER* is equal to the sum of -Xmx and -MetaspaceSize as configured in step #1. The memory is in MB.

  5. Stop the vRealize Appliance and increase/decrease the memory to match the increased/decreased memory of the vRealize Orchestrator.
  6. Start the vRealize Appliance.
  7. Repeat steps #1 to #6 to rest of the nodes in the cluster.

Just a short, quick blog for today, but this was a very good change for me, and I saw a marked improvement in response time from my embedded vRO.

Hopefully some hilarity from homelabbing coming, but I hope it helps someone out there. Some highlights:

  1. ISP MAC LOCKING!
  2. WAN +2LAN2? or WAN+LAN2??
  3.  MODEM TO WHAT ON THE WAAAAAT???
  4. “Its just making them talk to each other right?”

The Life of a Home Lab…Rookie

Normally my blogs are more technical and at least get the information accross about how to do different things within VMware’s toolbox. Today, I’m starting a series(ish) on doing a homelab. Yes this is new for me and i’m working through some basic things i havn’t had to do since ESXI 5.5. SO there is some learning to do. Thoughts so far

  1. Distributed switches… where did they go??
  2. Oh man supermicro boot delay is killing me…
  3. Cable management… this is why i got into coding…
  4. Networking… should the edge go to the router to the system or a different way?

These are all thoughts that went through my brain. Not very helpful i know, but maybe some of this can help those like me, who spend all their time in already built enterprise solutions, has a rack/stack team that bring the server up, then a networking team to add the needed networking.

But, thats the annoying, The good is I’ve got some great stuff to dig into and to worth through. I’m going to be slamming through this now and getting this done. First though..

The Setup

Current List of assets:

  1. 3 Supermicro E200s
  2. 1 Ubiquity edge switch
  3. 3 Samsung nvme SSDs for storage
  4. Alot of cables

q9jMhoe6QVKQ5exdkYIaHg.jpg

So first things first, I put the SSD into M2 slot on each of the machines. There was two philips screws on the back, then an overhead plate covering the ram/SSD slot etc. Once i removed that i was able to access the screw that would hold the SSD in place.

Cabling the three with the switch wasn’t too bad, I purchased a miniature server case to put everything in from amazon. It doesn’t look too bad, I pulled a 1×6 from the garage and built a table with no topper to allow cables to come up from the bottom. Pictures incoming!!

This slideshow requires JavaScript.

So cabling is completed, and everything is “racked”… lol

ESXI 6.7

YUP, Lets get to imaging.

For those that did anything with Supermicro the pain of that 1 ms default boot screen is rough. Immediately save yourself the pain and change that to 10000 once you get through the bios.

I wont go through the settings etc for imaging ESXI because its pretty well documented and not too difficult.

However, Imaging for VCSA has been a bit of a sludge for me, but again, there is alot of already written documentation. Heres a good youtube vid that will lead you the right way.

Which leads us to where we are currently… I’m trying to figure out how to setup networking and get it setup properly.

 

 

VRealize Deployments: Part Two – The Network

So this is FAR overdue.. Sorry about that. In Part one we basically looked through setting up an AD structure for the computer object and setting the machine to install and join domain. Basic stuff really, but going from a purely manual build to this process pretty sweet. Here is the next part. IPAM, and the Network.

IPAM

There are a lot of solutions out there for IPAMs, from Solarwinds, to Infoblox the IPAM space has a lot of prospect. Some even use an excel spreadsheet right out of the 90s. Well, In all these prospects the built in Network Profiles of VRealize Automation can definitely make your life easier.

The research

The first thing you need to do is get a block of IP’s. Lots of different ways to do this. If your IPAM solution is extremely trusted you would go through that solution and reserve a block(depending on your resources for the VLAN, would determine the size of the block). From here you would need to go to VRealize portal to create your profile.

  1. From the login of the portal you will go to “Infrastructure” -> “Reservations” -> “Network Profiles” and click “New” -> “External”network1
  2. From here you will create a Name for the profile(this will be used later). Description. IPAM endpoint for this will just be the internal IPAM. Select the proper Subnet mask from the drop-down and input the Gateway.network2
  3. On the DNS tab you will input the Primary  and  Secondary DNS, DNS suffix, DNS Search Suffices and any appropriate WINS inputs.network3
  4. Finally you will input you IP block. Inputting the Name, Description, Start IP and End IP.
  5. Next you will need to set this in the Reservations.
  6. On the Infrastructure tab go to Reservations, and select the reservation that will be using that profile. On the Network tab of the Reservation click the drop-down to select the newly created profile.                                    network4
  7. Now Save your settings.
  8. Now you can go to Design and create a new blueprint to take advantage of your profile.                                                                                                        network5
  9. On the blueprint Canvas to the left, select Network & Security and drag over Existing Network. This will put a new item on the canvas to go with your machine type.                                 network6
  10. Click the “…” and select the profile that should be utilized for the machine.network7
  11. Now Click on the Machine Item and go to the Network Tabnetwork8
  12. Select the same profile on the machine, and click OKNetwork9
  13. You should now see a link on the machine, and that machine will not pull from that IP block for its IP settings.

Pretty awesome in my opinion and far superior to 5 custom properties. VMware has really done a lot to help engineers get the best out of their solution.

Now the next step follows what needs to be adjusted in your IPAM solution, weather that is to set the IP to a specific setting or call it and adjust some notes. This is specific to your environment and to your solution. I would suggest an extensibility subscription that runs on deployment/Destroy and adjusts as needed. Both Infoblox, and Solarwinds have good plugins for vRO along with custom API calls that can be utilized to solve these issues.

Thanks for reading! Hope this helps!

Docker runs on Windows 2016 Core

So I’ve read alot of blogs out there for getting ready for containers, but since I’ve hated anything dealing with fruit and tech for so long, I’ve disregarded a whole side of scripting. This is something I’m remediating in further blogs, but for this one, lets focus on Windows 2016 non-desktop experience containers…. And throw in some vRA because its fun…

The CORE

So first you need to get your hands on a Windows 2016 iso. From my research the nonn-desktop experience is built-in to the basic ISO(HUZZAH).

Go through the normal Setup for your environment to setup a new machine in vSphere or whatever IAAS your using.

Once you can Console in you’ll be greeted with a familiar friend:docker1Thats right friends, its OLD SCHOOL TIME!

So set a password thats totes legit, and get to work. Once your in the cmd prompt its time to whip up an old friend:

SCONFIG

YUP, Sconfig will setup everything from here on out in terms of firewall, domain, ip, etc. For this purpose I’ll just set the ip on the machine.

The Docker

Now run the following
powershell
Install-Module DockerMsftProvider -Force

This will ask for confirmation so hit the ‘Y’.docker2
Now run the next command:
Install-Package Docker -ProviderName DockerMsftProvider -Force

This will finish up the docker install and you will probably want to reboot the machine.

GREAT Docker is now installed and your ready to go! not really…

For people like me, you want to get Docker within a centralized management(VRA would be nice). So for this you need to continue some setup.

The Management

First you would create the docker config file otherwise known as “daemon.json”.

Run the following to create the file:
CD C:\Programdata\docker\config
New-Item -ItemType 'file' -name daemon.json

This creates the blank file. Now to populate it. First stop Docker service…
Stop-Service docker

Now run “notepad” and open the file. Insert the following
{
"hosts": ["tcp://0.0.0.0:2375", "npipe://"]
}

Where 0.0.0.0 = your Ip address.

Finally run,
docker trust key generate role

Where role = the role of the public key. Once created open the key (the command should show you where) in “notepad” and copy from “—-” to “—–“. Now login to vRA.

On the “Containers” tab go to “Identity Management” and click on the “+CREDENTIAL”docker6Create a name for the creds(this will be used later) and paste the public Key that you copied from creation in the text field.

Now on the “Containers” tab go to “Container Host Clusters” and click on the “+CLUSTER”docker7This should create your Docker Container Master.Docker8

From here you can pull from the registry to build a “hello-world” container and deploy it from vRA

***BONUS***

Installing a core OS can be difficult for those that don’t remember those days(i had to google myself). Installing VMtools can be difficult… Mount the Tools on the OS and run the following on the disk drive:
.\setup64.exe /s /v "/qn reboot-r"

Happy containering!

Custom Forms = Custom Naming?

UPDATE:

So the code I used has a bit of a bug. When going from 1 digit length up to 2 it adds the 0 to it still. I believe its based off the default “inum” in the code. I’ve updated the code below. Notice, I went a different route, so that I can run if/else clauses per length as needed(depending on how many digits you want to go). I’m ok with just 2, so I’m running the following:
if(type == null)
{
return ""
}
else{
name = app + "-" + env + "-" + type
var inum = ""
sSearchPattern = name + inum
// Get a list of computers matching the pattern in strComputer
var computers = ActiveDirectory.getComputerADRecursively(sSearchPattern,defaultADServer);
if(computers[0] != null)
{
for each (computer in computers)
{
inum = Math.max(inum, computer.name.substring(sSearchPattern.length));
}
}
inum=inum+1;
var snum=String(inum)
if(snum.length == "1") {snum = "0" + snum;}
else{
//donothing
}
//build the new name
var newName = sSearchPattern+snum
// Validate the newly minted name, to make sure it doesn't exist
var computers = ActiveDirectory.getComputerADRecursively(newName,defaultADServer);
if(computers[0] == null)
{
newComputerName = newName;
}
else
{}
return newName;
}

Original blog below:

So, the white whale for me in automation has been custom naming. VRA allows for specific naming with some numerical iteration, but that’s not always fit the bill. Besides, why can’t I use the naming convention as a dynamic field within custom forms? IS ANYONE READING THIS?

wcft

Anyway. There are SO MANY WAYS to do this, it seems silly to add another, but, since I learned from so many people out there, I figured giving back may help someone else.

So your going to first create an action in vRO. Once you login set the dropdown to “Design” – Right-Click on the folder you want to create the action and click “Add action…”

customnaming4

THE CODE!


name = location + "-" + domain + "-" + app
var inum = 0
sSearchPattern = name + inum
// Get a list of computers matching the pattern in strComputer
var computers = ActiveDirectory.getComputerADRecursively(sSearchPattern,defaultADServer);
if(computers[0] != null)
{
foreach(computer in computers)
{
inum = Math.max(inum, computer.name.substring(sSearchPattern.length));
}
}
inum=inum+1;
var snum=String(inum)
while (snum.length < (2)) {snum = "0" + snum;}
var newName = sSearchPattern+snum
var computers = ActiveDirectory.getComputerADRecursively(newName,defaultADServer);
if(computers[0] == null)
{
newComputerName = newName;
}
else
{}
return newName;

NOTE!! It is assumed each input will be static and have a set default (i.e. Never null). So make sure each input has a default setting…

Basic theory here is that our naming convention is AA-BBB-CCC001(you can change this to anything, it’s not really the crux of this). But what your looking for is the “Math.max”; it’s just delightful. The inputs I used are as follows:

customnaming3

1. location–string
2. domain–string
3. app–string
4. defaultADServer–AD:AdHost

With this as an action, you can do some awesome stuff with custom forms.
For instance, you can set a deployment per app, or environment, create those inputs as static, and type as a dynamic dropdown.
Pretty simple.

    1. Create the Action with the code above and the inputs, and give it a pretty cool name(like edgar)
    2. Create Custom Properties for each input needed BUT the Ad:host (we’ll discuss that later.)
    3. Use custom forms! Activate it, and get a beer… you deserve it, you smart person you.
    4. Bring each input over that you need to use (if the are static only bring over the ones needed for dynamic).
    5. customnaming1For “Hostname”, use external source, and under action, type in the name of your action: (edgar)
    6. For the inputs select “FIELD”, then select the proper custom  property input you created. Then for “defaultADServer” use “Constant”. This will pull the AD Endpoints you have created. Point it to the AD you intend to use (if you want this dynamic you can use it as a field dropdown).
    7. For whatever you wish to create dynamic, set the field to a dropdown, and set the values to “Value|Label,Value|Label”.

Now the external Action will populate the Hostname, and wonderful custom naming “should” be yours.

customnaming2

Good job, get beer.