Corporate

When I look at what goes on in a corporate environment for someone like me that thrives on ever changing challenges I always felt that I wasn’t being pushed to my limits. I would maintain the same servers day in and day out, patch Tuesday would come and go and I knew what I was going to be doing. I would be patching those servers once again during my tiny maintenance window. The technology for me wasn’t changing much, I was pushed to learn a new product maybe once every 6 months. For the most part if you’ve seen 1 Windows server you’ve seen them all. There were some aspects though that I never fully learned such as Exchange and the deep dark innards of multiple domains inside a forest for example. I also saw the opportunity to attend training events like Cisco Live as something that just wasn’t in the budget nor was training really. What I did like though from corporate America was the fact that for the most part I knew what my hours were, it was M-F 8-5 typically, very little travel involved, so it was easy to plan activities with the family.

VAR (Value Add Reseller)

Ok let me start off this one first of all is the fact that we, as employees of VARs, really need to remember what the heck the VA stand for! We are not just a seller of a product, we add value to the sale of a product. We are the experts, the ones that should know better when the customer asks us to do something crazy. Ok, rant off, back on topic! Working for a partner for the past 6 years I’ve learned a ton, and I mean a ton. The challenge is always there, I’m constantly having to push myself. Although it might be the same style of deployment (a controller, few APs, some voice, etc) there is always some curve ball that gets thrown your way and you have to work around. I’m pushed by my peers to learn more and better understand technologies, even if it isn’t my core technology, I still need to know enough to help them when they need it. Working for a partner has opened up my network as well by allowing me to attend certain events such as Cisco Partner VT’s where you get together with the business unit and other partners from around the country (sometimes world) to learn about the upcoming products, issues with the current line, and collaborate. I’ve always said it, we might be competing but we are all in this together, either we all succeed or no one does. The relationship that this has allowed me to create with Cisco has opened up many doors and helped me create life long allies that I know I can trust to help me if I get stuck. At the end of the day the thing I like the most is when a customer realizes that I am living up to that “Value Add” portion of the purchase. They see my expertise and war wounds from going through a similar install with other customers and I can let them learn from past mistakes/issues that cost us in the end.

Vendor

Time for the vendor, the gradual next step for most from the VAR world as it seems. Now I haven’t personally worked for a vendor directly so my opinion here is based on what I feel happens when you go to a vendor. Typically if you work for a vendor you are drunk on their kool-aid to the point of no return, you have to be. If you were caught saying something great about a competitors product that could be a huge PR issue. I know this is going to come as shock to some but I don’t bleed Cisco, I go with what technology fits the solution needed. I love Cisco’s controller based solution, I think it is a highly efficient solution that for me is easy to configure, but their autonomous stinks. Aruba has an awesome autonomous-like setup with their Instant AP, if you haven’t looked at it please do. Aerohive is making strides with their branch and teleworker solutions, I usually always travel with either my BR100 or BR200. Meraki (now owned by Cisco) has an amazing ease of use that no one comes close to touching. Ruckus has an insanely talented antenna team, I love listening to them and having my brain melt from the physics that Victor Shtrom starts flowing, and he does it so effortlessly! I would have a hard time working for a vendor knowing that I would have to sort of look the other way when a customer is looking at a solution that I know would actually be the better fit. I would have a hard time trying to shoehorn vendor X into that sale and knowing the customer would be happy. Working for a vendor also means no more Tech Field Day events which I’ve found to be another great source of networking. The ability to meet all of the CEO’s and other amazing team members from these vendors has given me more “phone a friend” cards in my Rolodex for when I need something.

Final Thoughts

Ultimately you can see why I have stayed in a partner space. The flexibility and ever changing environment it allows me to work in is great. The downside? Travel. Depending on the size of the partner you work for you will either be local with minimal travel or have national accounts with 80-100% travel which is where I sit today. The burden on my family is tough while I’m traveling so we’ll see how much longer it lasts! One final comment on this subject is pay, hence the title of this post. You need to work where you feel you are being paid what you are worth. Being 1 of less than 200 (estimated) CCIE Certified Wireless Engineers is a big deal for me, it gives me a nice Ace in my card deck. It’s worth more to a partner than it is to a vendor or corporation. That number gives the partner special perks so hopefully you are compensated as such to reflect that. In the end do what makes you happy, work for who you want to that makes it not feel like work. If that means you go to a vendor for a few years to get an inside look and then go back to a partner, do it. If you are at a point in your life where you don’t want to have to learn new products every 3 months, cozy up in a cube and make your mark there. If you are still reading this, maybe taking a chance and starting your own business is the right idea as well! I would love to simply be a technical architect consulting with others and helping them navigate the plethora of options, doesn’t matter what vendor it is.

Going forward being a wireless engineer really needs to change. No longer should a wireless engineer be a person that knows how to install an AP, controllers, etc. The person needs to be a true RF engineer, understanding that makes you the wireless engineer. (Yes one could argue that this has been always the case and I agree, but that isn’t really what we see in the real world all the time!) With a converged access/distributed/vendor-x method there is a larger focus on understanding how a wired network properly works to troubleshoot packet flows. I know I’ve changed some of my development plans to focus more on route/switch training to better understand technologies such as SDN. I still have a desire to obtain my CWNE but now am looking at things like the CCIE R/S or CCIE Security.

Are you changing your development plans? Chime in on the comments if you are!

What is AVC?
Now one could argue that this isn’t anything new either as we have had Netflow for awhile now but we really haven’t had the data presented in this way before right from the vendors. AVC is simply visualizing traffic destinations in such a way that you can see how much of the data on your network is destined for HTTP/S for example and what sites such as Facebook, Twitter, etc. AVC also allows you to see how much data is for streaming services such as iTunes, AirPlay, etc.

Why Should I Care?
So why should I really care about this? What does this give me? As we become more of a connected society our bandwidth consumption has increased tremendously. Companies constantly struggle with having enough bandwidth do get their business tasks accomplished while meeting the needs of their employees. AVC allows a company to really see where their bandwidth is going and allows you to control it’s usage, hence the name. We can create profiles to drop traffic so instead of relying on a proxy server to do that for us we can stop it at the point of presence for a connection. Or maybe you don’t really need to drop it, you just want to be able to say what your top bandwidth usage is. A great example is when I installed my Meraki switch at my house, right away I was able to see that my Samsung Tab 8.9 that ran a simple security camera app was creating close to 158GB worth of traffic every 24 hours on my network, I wouldn’t have known that without AVC! I’ve also found how much our Netflix adds to our bandwidth cap usage each month.

The Future
Network design is really starting to change, we are progressing away from a switch port VLAN mapping to more of a traffic/user/device mapping with technologies like Software Defined Networking, policy based routing, and identity & device profiling. This new era of a dynamic network is an interesting one and will be interesting to see how all the new technologies come together to reshape how we architect networks. Ultimately in order for this new design concept to work we need the right tools, networking vendors are being forced to become software companies as well. The big names such as Aerohive, Aruba, Cisco, Meraki, for example all have this in their GUIs now at least for wireless traffic. I think Meraki does it best though in how they present the data to you. Their UI’s have always outshined the competitors, hopefully Cisco will take note of that as the Meraki talent is brought into Cisco post-acquisition. Data sources will also be key for the success, deep packet inspection is really the best solution instead of relying simply on DNS resolutions or known ports. When I look through my logs every now and then I’ll see WoW traffic, something that I don’t ever play, but according to Meraki’s AVC I’ve passed 2GB worth of traffic over a week, this could simply be web ad data, which flaws the data slightly.

Does AVC seem all that important to you? Chime in on the comments! I will leave you with a snippet from my Meraki dashboard for a week in December, this was a light week as I wasn’t home the entire time.

I’ve had experience using their wireless, security, switching, and mobile device management platforms but never really understood how it worked from the back end services. With Meraki being a cloud based solution performance is key, the scalability for their infrastructure is very important and it needs to be able to handle numerous different client requests for data through their dashboard application. I’ve had the benefit of building numerous web applications using PHP and MySQL so I understand where they are coming from. I’ve had applications in the past that I wrote that were querying MySQL tables in excess of a million rows and I can tell you that isn’t a quick action unless done right.

So let’s take a look at the Meraki cloud from about 1000 feet:

• Customers are partitioned across Meraki servers
  • Each partition is called a shard.
  • It’s a beefy 1U RAID server plus a 1u backup geographically dispersed.
  • One shard, which is the master, acts as a demultiplexer. When you hit the dashboard GUI you can see which master you are on.
• Example scale from a representative shard
  • Thousands of Meraki devices
  • Hundreds of thousands of clients
  • About 300GB of stats, dating back over a year
  • Gathers new data from every device every 45 seconds

Those numbers from the representative shard create some interesting engineering challenges though.

• How to connect to thousands of Meraki devices per shard?
  • How do you allow access to devices behind NATs? Meraki created a tunneling software that is encrypted as well as the data being encrypted while streamed.
  • This custom secure tunnel requires only 2 pkts/device/25 secs allowing for scalability.
• How to gather stats from those devices every 45 secs?
  • You need to minimize network overhead.
  • You need to minimize CPU costs.
• How to quickly store and retrieve all that data for analysis?
  • Customer database tuned for statistical data specifically.
  • SSD like speeds from inexpensive spinning discs which reduces expenses for scaling.

The challenge was solved by a poller (if you are familiar with Cacti) of sorts called Poder. At first it didn’t scale well because of how the daemon would grab data from the devices. Threads were used with each grabber (1 per data point such as cpu usage, uptime, etc) having their own process. This was a very simple approach relying on basic features:

• HTTP, XML libraries which are widely available
• TCP based which easily carries arbitrarily large responses
• XML which is unstructured allowing one to easyily to add new stats
• Blocking code is easy to write
• It’s easy to restart individual grabbers during testing/dev

But this approach is expensive at scale:

• Single HTTP fetch with empty payload: 10 pits, 510 bytes
• Each grabber does its own fetch, no sharing connections
• Lots of threads for I/O parallelism

So a new high performance approach was created based on an event driven RPC engine. Each information module then has a thread for LLDP, probing, etc.
Each module creates a request and sends to the RPC engine, each device then sends a response which goes to a DB. This allows for a single networking core for all statistics where the transaction is 1 fetch thread + non-blocking read/write + select loop. A switch to use UDP by default, fall back to TCP for large responses allows quicker packet transfer times. Binary encoding via Google Protocol Buffers allows for efficient use of buffers. There are still separate modules for each type of stat but the modules are single threaded and blocking on DB access. Most are 200-400 liens of straight line Scala code and don’t present a locking issue. The backend DB was changed as well, Meraki creates an interesting tree node concept keeping more information in RAM before writing it to disk. Once written to disk though there are index identifiers that allow for quicker disk seeking increasing performance by limited the number of reads required to find a data section containing the stats needed to display to the dashboard.

This new performance approach allows Meraki to scale their dashboard application which is crucial to the user experience and use of their equipment to levels that were once out of reach. I really enjoyed getting an in depth drink from the fire hose look at the back end services running the dashboard GUI. While it wasn’t directly related to RF or wireless I think it is important to understand how Meraki handles the complexity of running a hosted platform and growing to support larger amounts of customers. Kudos to Meraki for giving us this behind the scenes look.

In order to understand the need for location analytics there are a few challenges that one is presented with:

1. Business and venue operators face increasing pressures from changing business landscape.
2. Business operates have very limited visibility into their business environments. Where, when, and how do people move around the venue?

The key is to understanding what can we do with the data once we have it. How can someone solve those challenges with location analytics? Finding the most optimum walking path is key for large venues. How does the traffic flow at the beginning, during, and end of an event? Do we need to find more efficient routes of moving people around? With the move towards online shopping brick and mortar stores need to find new ways to keep people in their stores and buying, not window shopping. As we move towards a digital shopping experience with NFC, PayPal at the checkout, and personal payment channels like Square we need retailers to adapt and improve their methods of connecting to their patrons.

There really are a couple of trends that are driving this right now that helps us overcome those challenges:

1. Explosion of mobile devices and BYOD connecting everyone, everywhere, all of the time.
2. WLAN networks are becoming increasingly sophisticated enabling new capabilities and services.

The onslaught of mobile devices such as smartphones and tablets has changed the shopping experience. I can now look-up and compare prices quickly and easily. It’s very rare that you find someone that doesn’t have a smartphone these days and that doesn’t use it for at least window shopping comparisons. Venues are looking at ways to build these new robust wireless networks that we as consumers expect to have access to.

So what is it really? We can break location analytics down into three sections:

Detect the presence in the venue. First we need to detect the patron and where they are. We can do this by listening for the wireless probes that a smart device sends when looking for a wireless network. We can also use the MSAP protocol to assist in getting people to realize there’s an experience here.

Connect, understand network utilization, peak usage, number and types of devices on the network. How can we quickly and easily offer wireless services to the patron to get them on our wireless network? We don’t want the patron to have to burn through their cellular data, albeit we are talking small amounts of traffic.

Engage allow insights into customer online and onsite behavior, most traffic paths, dwell times, location density etc. At this point once we have detected the patron and allowed them to connect we want to engage with them. This would be through an application that runs on their smart device tailoring the product information to them as part of a loyalty program for example. We can even provide them with a GPS like navigation to where something is in the store in relationship to their location.

At the end of the day though there is a huge concern here, are we giving up privacy at the sake of what? If we simply don’t join a loyalty program at most the venue is allowed to record a MAC address as an identifier, this doesn’t really associate you in any shape or form so there really isn’t a privacy concern. A venue doesn’t know a MAC is a person. Now when we couple a loyalty program with the MAC though we run into the concerns about privacy. The fact that one can easily spoof a MAC could introduce some fun loyalty program games creating false advertising directed at an individual or profiling to occur. This technology is relatively young so it’ll be interesting to see how it pans out.