<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://a-ayrapetov.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://a-ayrapetov.github.io/" rel="alternate" type="text/html" /><updated>2026-11-13T10:05:31+00:00</updated><id>https://a-ayrapetov.github.io/feed.xml</id><title type="html">Arkadiy Ayrapetov</title><subtitle>Cybersecurity Architect. Field notes on network access control, segmentation, OT and ICS security, detection and the read-only tooling I use on client work.</subtitle><author><name>Arkadiy Ayrapetov</name></author><entry><title type="html">An AI assistant on a production network that never gets a session</title><link href="https://a-ayrapetov.github.io/2026/11/13/ai-assistant-that-never-touches-the-network.html" rel="alternate" type="text/html" title="An AI assistant on a production network that never gets a session" /><published>2026-11-13T00:00:00+00:00</published><updated>2026-11-13T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/11/13/ai-assistant-that-never-touches-the-network</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/11/13/ai-assistant-that-never-touches-the-network.html"><![CDATA[<p>I use an AI assistant on almost every engagement now: switch and controller work, ISE policy, monitoring platforms, industrial networks. It has never once held a session on a client device, and that is the reason it is safe to use at all.</p>

<p>Think of a surgeon and a scrub nurse who knows every procedure by heart. The nurse prepares, checks and hands over the right instrument at the right moment. Only the surgeon touches the patient. An assistant on a production network works best in the same split: it writes the commands, scripts and click paths faster than any person, and a person runs them.</p>

<h2 id="what-goes-wrong-without-rules">What goes wrong without rules</h2>

<p>The failures are predictable, and none of them are exotic:</p>

<ul>
  <li>Commands written for a different software release than the one running.</li>
  <li>“Done” or “fixed” reported with no output to back it.</li>
  <li>The same context explained again in every new chat.</li>
  <li>Work that stops whenever the one person who remembers the details is away.</li>
</ul>

<p>Each rule below exists because one of these happened.</p>

<h2 id="the-rules">The rules</h2>

<p><strong>Split the roles.</strong> The assistant has no session on any device or console. It writes; a person runs, then hands back the output as a log file, an export or a screenshot. In regulated and availability-first environments this is also the only arrangement a change board will accept.</p>

<p><strong>Keep one record.</strong> One knowledge base per engagement holds the rules, the facts, a status board and a tracker. Every chat reads it first and writes back what changed in the same reply. A chat has no reliable end, so nothing is saved for later. Anyone, or any other assistant, pointed at the record can pick up the work.</p>

<p><strong>Source every fact that can change.</strong> Addresses, VLANs, tags, versions, CLI syntax and GUI paths come from output in the conversation, from the record, or from the vendor document for the exact platform and release. Anything else is marked as unverified, with the command that would settle it. Never from the model’s memory, because the model’s memory is where commands from the wrong release come from.</p>

<p><strong>Climb a fixed ladder for missing facts.</strong> Output already given, then the record, then local files, then vendor documentation, then one read-only command for the person to run. The assistant climbs the ladder before it asks a question.</p>

<p><strong>No completion claim without evidence.</strong> “Done” requires output read in that same reply. A colleague saying it worked, a plan, or a script that exists does not count. A scheduled job finished only if its final status line is in the log.</p>

<p><strong>Ship every change in one shape.</strong> Scope, operational impact, the verification command and the value it should return, a rollback ready to paste, and how the device is reached if the change locks you out. One change at a time, a canary first, stop at the first device that does not verify.</p>

<p><strong>Scripts are read-only unless asked.</strong> A change script has a dry run, an explicit apply switch, a backup before, a verification after and a rollback file. Credentials come from the operating system’s credential store, never from the script, a log or the chat.</p>

<p><strong>Keep a fallback.</strong> The record is plain text behind a standard protocol and mirrored to a folder. If one assistant is unavailable, another reads the same files. If none is, the runbooks are complete enough for a person.</p>

<h2 id="why-the-split-works">Why the split works</h2>

<p>An assistant is strongest at recall and drafting and weakest at knowing what is true on a live network right now. The split puts each side where it is reliable: the person supplies ground truth, the assistant supplies speed, and the record carries memory across sessions and people.</p>

<p>The test I use is simple. Pick any statement the assistant made today and ask where it came from. If the answer is a path in the record, a line of output, or a vendor page for the right release, the method is working. If the answer is “it just knew”, it is not.</p>

<h2 id="where-to-find-more">Where to find more</h2>

<p>Two of my playbooks carry the method further: <a href="https://github.com/a-ayrapetov/a-ayrapetov/blob/main/playbooks/gated-production-change.md">gated production changes</a> and <a href="https://github.com/a-ayrapetov/a-ayrapetov/blob/main/playbooks/script-contract.md">the script contract</a>. The gates and proofs those describe are <a href="https://github.com/a-ayrapetov/a-ayrapetov/blob/main/playbooks/templates/netops_gates_and_proofs.py">code with tests</a>.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[I use an AI assistant on almost every engagement now: switch and controller work, ISE policy, monitoring platforms, industrial networks. It has never once held a session on a client device, and that is the reason it is safe to use at all.]]></summary></entry><entry><title type="html">Closing access ports on industrial switches, one port at a time</title><link href="https://a-ayrapetov.github.io/2026/11/13/closed-mode-mab-one-port-at-a-time.html" rel="alternate" type="text/html" title="Closing access ports on industrial switches, one port at a time" /><published>2026-11-13T00:00:00+00:00</published><updated>2026-11-13T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/11/13/closed-mode-mab-one-port-at-a-time</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/11/13/closed-mode-mab-one-port-at-a-time.html"><![CDATA[<p>Closing access ports on an industrial network is like changing the locks in a building that never closes. Every door has to keep working for the people already inside while you work on it. The devices on these ports (cameras, intercoms, sensors, controllers, signs) mostly cannot run 802.1X, so they get MAC Authentication Bypass. In closed mode, a port passes nothing until the policy server says yes. One wrong group, one stale MAC address or one device that has not spoken yet, and something that matters goes dark.</p>

<p>This is the order of work I use for those rollouts with Cisco switches and Cisco ISE. I am using it on a rollout that spans hundreds of devices across many sites, one switch at a time.</p>

<h2 id="agree-the-scope-before-touching-a-port">Agree the scope before touching a port</h2>

<p>Run a short test with the owners on one port of each device type. Write down which types passed, which are approved, which are excluded and which are on hold. Anything tied to safety stays on hold until its owner approves in writing. That list is the contract for the rest of the rollout.</p>

<h2 id="keep-a-safety-net-in-policy">Keep a safety net in policy</h2>

<p>While ports are being closed, the catch-all authorization rule in ISE stays at permit. That is deliberately not the end state. It means a device that profiles oddly still gets on the network while you learn about it. Tightening the catch-all is a separate change, later, with its own approval.</p>

<h2 id="trust-no-single-device-list">Trust no single device list</h2>

<p>Before writing a single command for a site, build the device list from three sources and reconcile them: a fresh read-only capture of the MAC address tables from the switches, the endpoint export from ISE, and the tracking sheet the owners keep. Fix stale MACs first. Most outages in a rollout like this start with a wrong assumption about what is plugged in.</p>

<p>Then check ISE read-only: the site’s switches exist as network devices, every endpoint sits in the right identity group, and the authorization rules for those groups exist.</p>

<h2 id="one-switch-at-a-time-always-in-the-same-order">One switch at a time, always in the same order</h2>

<ol>
  <li>Confirm the switch is a network device in ISE with the right shared secret.</li>
  <li>Open a fresh admin session while the old one stays open. A working TACACS+ login proves nothing about RADIUS, so test each service on its own.</li>
  <li>Pre-check that the switch reaches the policy servers and that the identity template is on the switch.</li>
  <li>Apply the ports from one paste block with real names inline: a description that says what the device is, then the closed-mode template.</li>
  <li>Test every port. Save only after the tests pass.</li>
</ol>

<p>A port with several MAC addresses behind it needs its host mode chosen on purpose and agreed with the owner. Multi-host authenticates the first MAC and then opens the port for the rest. Multi-auth authenticates every MAC on its own. They are not interchangeable, and the wrong one either blocks the second device or lets in anything that plugs into the first.</p>

<h2 id="never-bounce-a-poe-port-to-retry">Never bounce a PoE port to retry</h2>

<p>The reflex to shut and unshut a port to restart authentication is wrong here. On a PoE port, shut cuts power to the camera or intercom, which then has to boot again. Clear the session instead:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>clear access-session interface GigabitEthernet1/0/12
</code></pre></div></div>

<p>On releases or configurations that still use the legacy form, the command is <code class="language-plaintext highlighter-rouge">clear authentication sessions interface ...</code>. Check with <code class="language-plaintext highlighter-rouge">?</code> on the switch.</p>

<h2 id="quiet-devices-are-not-failed-devices">Quiet devices are not failed devices</h2>

<p>MAB starts when the device sends its first frame. Some devices are quiet for minutes. No session and no MAC usually means the device has not said anything yet, so give it a few minutes and check link and power before you blame policy. A show command right after clearing a session can also come back empty for a few seconds. Run it again before you react.</p>

<p>And know whether ISE reacts to profile changes on your deployment. If a device’s profile changes and ISE sends a change of authorization, a device that worked yesterday can drop today.</p>

<h2 id="what-counts-as-done">What counts as done</h2>

<p>Per port: <code class="language-plaintext highlighter-rouge">show access-session interface &lt;port&gt; details</code> shows the session authorized by MAB with the expected tag or VLAN, the device answers, and the ISE live log shows the pass. Per switch: the save is confirmed. Per site: every port in scope is on the tracking sheet as verified from output, not from memory.</p>

<p>Rollback is per port: remove the template and any host mode you added, then clear the session on that interface.</p>

<p>The full checklist is in my <a href="https://github.com/a-ayrapetov/a-ayrapetov/blob/main/playbooks/closed-mode-mab-site-rollout.md">closed-mode MAB playbook</a>. Before the rollout, the <a href="/2026/11/13/passive-ot-asset-inventory.html">passive inventory</a> is how the device list gets built without touching the devices.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[Closing access ports on an industrial network is like changing the locks in a building that never closes. Every door has to keep working for the people already inside while you work on it. The devices on these ports (cameras, intercoms, sensors, controllers, signs) mostly cannot run 802.1X, so they get MAC Authentication Bypass. In closed mode, a port passes nothing until the policy server says yes. One wrong group, one stale MAC address or one device that has not spoken yet, and something that matters goes dark.]]></summary></entry><entry><title type="html">Cutting NDR alarm noise without going blind</title><link href="https://a-ayrapetov.github.io/2026/11/13/ndr-alarm-noise-without-going-blind.html" rel="alternate" type="text/html" title="Cutting NDR alarm noise without going blind" /><published>2026-11-13T00:00:00+00:00</published><updated>2026-11-13T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/11/13/ndr-alarm-noise-without-going-blind</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/11/13/ndr-alarm-noise-without-going-blind.html"><![CDATA[<p>A noisy detection platform is a smoke alarm that goes off every time someone makes toast. After a week nobody looks up, and the one real fire gets the same shrug. Flow analytics and network detection platforms on large networks often raise thousands of alarms a day. Most of them come from a handful of rules and a handful of hosts nobody ever classified.</p>

<p>The tempting fix is to switch the loud rules off. The dashboard goes quiet and the network goes invisible. Here is the loop I use instead. It works with any flow analytics or NDR platform that has host groups and custom rules, and a team of two or three can run it a few items a day.</p>

<h2 id="measure-hosts-not-alarms">Measure hosts, not alarms</h2>

<p>Count <strong>hosts in alarm per day</strong> for a rule, not raw alarms. Raw counts swing wildly with traffic volume and reward the wrong fixes: suppress one chatty host and the count drops by thousands while nothing about your visibility improved.</p>

<h2 id="the-loop">The loop</h2>

<ol>
  <li><strong>Size it.</strong> Pick the rule with the most hosts in alarm. Record seven daily values. That is your before number. Skip anything small.</li>
  <li><strong>Find who.</strong> List the sources and targets for that rule. One target repeated across many sources is usually one cause.</li>
  <li><strong>Prove what it is.</strong> Host report, flow search, reverse DNS, the asset inventory. A host nobody can classify is a finding in its own right.</li>
  <li><strong>Decide.</strong> It is noise only if you can name the owner and the reason. Answers from the internet, traffic that crosses zones, scanning by unknown hosts, and anything without an owner stay as alarms and get reported.</li>
  <li><strong>Change one thing.</strong> Read the rule’s existing exclusions first. Take a screenshot of the rule before editing: that picture is your rollback. Prefer putting the host into an existing functional group that the rule already excludes. If the rule does not exclude that group, classify the host and add the exclusion. Narrowest scope, one rule per change.</li>
  <li><strong>Log and prove.</strong> One row per change in a shared tracker: before, after, where it was applied, how to undo it.</li>
</ol>

<h2 id="proof-that-the-change-worked">Proof that the change worked</h2>

<ul>
  <li>After the save time, the device still sends the same traffic (a flow search shows it), and no new alarm for that rule starts after the save time. Traffic continues and no new alarm: that is the proof.</li>
  <li>The rule now has exactly one more exclusion than before. Fewer means something was dropped during the edit.</li>
  <li>The next day’s hosts in alarm is lower than the before number. Fixes for overnight scanners can only be proven the next morning.</li>
  <li>Run the same check from an export or an API script, so the numbers do not depend on who looked at the dashboard.</li>
</ul>

<h2 id="traps">Traps</h2>

<ul>
  <li>Adding a host to a group does nothing if the rule does not exclude that group. Read the exclusions first.</li>
  <li>Some rule editors drop existing exclusions while you edit. Count them before and after.</li>
  <li>Old alarms can stay marked active after the fix. Judge by when an alarm first became active.</li>
  <li>Group names go stale. Check what is actually in a group before tuning around it.</li>
  <li>Scanners left unclassified make quiet devices look like they are breaking segmentation, simply because they answer the scan.</li>
  <li>Safety-related systems never go into an exception group. Report them to their owner instead.</li>
  <li>Temporary exception groups need a written membership rule, an owner and an exit date.</li>
</ul>

<h2 id="why-it-works">Why it works</h2>

<p>Most of the noise comes from hosts that were never put where they belong. Classifying them fixes many rules at once, and what is left on the dashboard starts to mean something again. A week after each change, the before and after numbers are in the tracker, and nothing that was excluded has an unknown owner.</p>

<p>The same loop, written as a checklist, is in my <a href="https://github.com/a-ayrapetov/a-ayrapetov/blob/main/playbooks/ndr-alarm-noise-reduction-loop.md">NDR alarm noise playbook</a>. Classifying hosts is far easier with a current device list, which is what the <a href="/2026/11/13/passive-ot-asset-inventory.html">passive OT inventory</a> is for.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[A noisy detection platform is a smoke alarm that goes off every time someone makes toast. After a week nobody looks up, and the one real fire gets the same shrug. Flow analytics and network detection platforms on large networks often raise thousands of alarms a day. Most of them come from a handful of rules and a handful of hosts nobody ever classified.]]></summary></entry><entry><title type="html">Inventory an OT network without sending it a single packet</title><link href="https://a-ayrapetov.github.io/2026/11/13/passive-ot-asset-inventory.html" rel="alternate" type="text/html" title="Inventory an OT network without sending it a single packet" /><published>2026-11-13T00:00:00+00:00</published><updated>2026-11-13T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/11/13/passive-ot-asset-inventory</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/11/13/passive-ot-asset-inventory.html"><![CDATA[<p>Every OT security program starts with the same question: what is actually on this network? On an office network the usual answer is a scan. On a network full of controllers, drives, HMIs and remote terminal units, a scan is the one answer you cannot use. Older controllers and protocol stacks can stop responding or fault when they are probed, and a scan during operations is a change nobody approved.</p>

<p>The good news is that the network already knows. Think of a hotel that needs a guest list. You could knock on every door, or you could read the key-card log at the front desk. The switches are the front desk.</p>

<h2 id="four-tables-the-switch-already-keeps">Four tables the switch already keeps</h2>

<p>Every access switch builds these tables as a by-product of forwarding traffic. Reading them adds nothing to the OT endpoints.</p>

<ol>
  <li><strong>ARP table</strong> on the layer 3 device: IP address to MAC address.</li>
  <li><strong>MAC address table</strong> on the access switch: MAC address to port and VLAN.</li>
  <li><strong>Interface status</strong>: port to VLAN, link state and the description someone typed years ago.</li>
  <li><strong>CDP and LLDP neighbors</strong>: which ports lead to other network gear, so you can leave them out of the endpoint list.</li>
</ol>

<p>Join them on MAC address and port, and you get a list that says: this MAC, at this IP, sits on this port of this switch, in this VLAN, and the port says it is “Line 3 PLC”. That is more useful than a scan result, because it tells you where the device physically is.</p>

<h2 id="what-the-join-tells-you-that-a-scan-cannot">What the join tells you that a scan cannot</h2>

<ul>
  <li><strong>An access port with several MAC addresses behind it</strong> usually means an unmanaged switch, a hub, or a device with a small switch built in. Those are the places where segmentation quietly stops.</li>
  <li><strong>A MAC address with no ARP entry</strong> is either a layer 2 only device or a device whose gateway lives somewhere you have not collected yet. Both are worth knowing.</li>
  <li><strong>A MAC that moves</strong> between runs is either maintenance or something you need to ask about.</li>
</ul>

<h2 id="treat-the-existing-inventory-as-a-lead-not-the-truth">Treat the existing inventory as a lead, not the truth</h2>

<p>On one large industrial network the monitoring system’s export was the official device list. In almost every row the device name was simply the IP address repeated, and the real name was buried in the first line of a notes field. Use whatever list exists to decide where to look first, then let the switch tables decide what is true.</p>

<h2 id="collect-like-a-guest-not-an-inspector">Collect like a guest, not an inspector</h2>

<p>The collection itself has to be as quiet as the method.</p>

<ul>
  <li>Read-only show commands on the switches only. Nothing goes to the OT devices. No ping sweeps, no scans on OT VLANs.</li>
  <li>Never poll safety systems, interlocks or serial-to-Ethernet converters, even with show commands.</li>
  <li>Throttle. A handful of concurrent sessions, a pause between commands, a pause between switches.</li>
  <li>Poll no more often than every 15 minutes. Hourly is a sensible default.</li>
  <li>Start with two switches you know well, check the result by hand, then scale.</li>
  <li>Keep every raw output next to the parsed result. When someone questions a line, the evidence is one file away.</li>
</ul>

<h2 id="three-things-that-will-catch-you-out">Three things that will catch you out</h2>

<p><strong>ARP is per VRF.</strong> On IOS-XE, <code class="language-plaintext highlighter-rouge">show ip arp</code> shows the global table. If the OT networks live in VRFs, collect <code class="language-plaintext highlighter-rouge">show ip arp vrf &lt;name&gt;</code> for each one, or half the IP column stays empty.</p>

<p><strong>Quiet devices age out.</strong> A MAC address only stays in the table while the device keeps sending frames, and the default aging time on Catalyst switches is five minutes. A controller that talks once an hour may not be there when you look. Run the collection several times across a day and keep the union.</p>

<p><strong>Not every switch speaks Cisco.</strong> Industrial switch vendors have their own CLIs, some with quirks that break automation. For those, read the same tables over SNMP from the standard bridge MIB instead of fighting the command line.</p>

<h2 id="run-it-again-and-diff">Run it again and diff</h2>

<p>A single inventory is a snapshot. The value comes from the second run. New MACs, MACs that vanished and MACs that moved ports are a change log for a network that is supposed to change rarely. A new MAC on a controller VLAN at two in the morning is the kind of event worth reading.</p>

<p>This is also where the frameworks point. The joint guidance CISA and its partners published in August 2025, <a href="https://www.cisa.gov/resources-tools/resources/foundations-ot-cybersecurity-asset-inventory-guidance-owners-and-operators">Foundations for OT Cybersecurity: Asset Inventory Guidance for Owners and Operators</a>, treats a maintained OT asset inventory as the base of a defensible architecture, and IEC 62443 zones and conduits cannot be drawn without one.</p>

<h2 id="the-tool">The tool</h2>

<p>I wrote <a href="https://github.com/a-ayrapetov/ot-asset-inventory">ot-asset-inventory</a> to do the join. You give it a folder per switch with the saved show output, and it gives you a CSV of endpoints per port with flags for multi-MAC ports, MACs with no IP and locally administered addresses. Vendor names come from the IEEE OUI file if you supply it. A diff mode compares two runs and exits non-zero when something is new, gone or moved, so it can run on a schedule.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install git+https://github.com/a-ayrapetov/ot-asset-inventory
ot-inventory collect/ --oui oui.csv --csv inventory.csv
ot-inventory diff last-week.csv inventory.csv
</code></pre></div></div>

<p>It only reads text. It never connects to anything, which is the point.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[Every OT security program starts with the same question: what is actually on this network? On an office network the usual answer is a scan. On a network full of controllers, drives, HMIs and remote terminal units, a scan is the one answer you cannot use. Older controllers and protocol stacks can stop responding or fault when they are probed, and a scan during operations is a change nobody approved.]]></summary></entry><entry><title type="html">Moving switches to ISE TACACS+ without locking yourself out</title><link href="https://a-ayrapetov.github.io/2026/11/13/tacacs-migration-without-lockouts.html" rel="alternate" type="text/html" title="Moving switches to ISE TACACS+ without locking yourself out" /><published>2026-11-13T00:00:00+00:00</published><updated>2026-11-13T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/11/13/tacacs-migration-without-lockouts</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/11/13/tacacs-migration-without-lockouts.html"><![CDATA[<p>Moving switch logins from local accounts to Cisco ISE TACACS+ is routine work, and it is also the change most likely to lock you out of your own network. On one migration of older switches, a method list in the wrong shape cost a full day and three lockouts. The failure looked exactly like a bad password every time, which is why it took so long to see.</p>

<p>Think of a door that reads your keycard before it checks whether the card reader has power. If the reader is dead and nobody planned for that, you stand outside with a perfectly good card.</p>

<h2 id="the-rule-that-decides-whether-your-fallback-works">The rule that decides whether your fallback works</h2>

<p>Cisco IOS walks a method list in order, and it moves to the next method only when the current one gives no answer. Cisco’s <a href="https://www.cisco.com/en/US/docs/ios-xml/ios/sec_usr_aaa/configuration/15-2mt/sec-cfg-authentifcn.html">authentication configuration guide</a> spells it out: if the server does not respond, that is an error and the next method is tried; if the server answers with a rejection, the process stops there.</p>

<p>So <code class="language-plaintext highlighter-rouge">local</code> at the end of a list protects you from an outage of the policy servers. It does not protect you from a wrong password, a missing user on ISE, or a policy that says no. Those are answers, and an answer ends the login. Plan for both cases separately.</p>

<h2 id="a-shape-that-survives-a-bad-day">A shape that survives a bad day</h2>

<p>The pattern that held up across fleets of older and current switches:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aaa new-model
username breakglass privilege 15 secret &lt;strong-local-secret&gt;
!
aaa authentication login default local
aaa authentication login VTY_AUTH group ISE_TACACS local
aaa authorization exec VTY_AUTH group ISE_TACACS local if-authenticated
aaa accounting exec default start-stop group ISE_TACACS
aaa accounting commands 15 default start-stop group ISE_TACACS
!
ip tacacs source-interface Vlan100
!
line vty 0 15
 login authentication VTY_AUTH
 authorization exec VTY_AUTH
</code></pre></div></div>

<p>What matters in it:</p>

<ul>
  <li><strong>A real local account with a secret</strong>, so the fallback has someone to let in.</li>
  <li><strong><code class="language-plaintext highlighter-rouge">local</code> after the server group</strong> on login and exec authorization, and <strong><code class="language-plaintext highlighter-rouge">if-authenticated</code></strong> at the end of exec authorization, so an outage of the policy servers does not leave a user who already logged in without a shell. Prove that on one canary switch by taking ISE out of reach on purpose: how the list falls through for a user who exists only on ISE is worth seeing with your own eyes on your release.</li>
  <li><strong>The default login list stays local</strong> and only the vty lines use the server list. The console keeps working the old way while you test.</li>
  <li><strong>Accounting to the server</strong>, so you get a record of who typed what.</li>
</ul>

<p>On some older IOS 15 releases the named <code class="language-plaintext highlighter-rouge">tacacs server</code> form parsed without complaint but never resolved the server address, and the legacy <code class="language-plaintext highlighter-rouge">tacacs-server host &lt;ip&gt; key &lt;key&gt;</code> form worked. That is a field observation, not something from a Cisco document, so check <code class="language-plaintext highlighter-rouge">show tacacs</code> after you configure it, whatever form you use.</p>

<h2 id="the-source-address-has-to-match-what-ise-expects">The source address has to match what ISE expects</h2>

<p>ISE identifies a network device by the source address of its requests. <code class="language-plaintext highlighter-rouge">ip tacacs source-interface</code> pins that address. Without it, the switch uses whatever interface the route points out of, ISE sees an address it has no device entry for, and every login with an ISE account fails. A one digit typo in the ISE device entry does the same thing.</p>

<h2 id="the-management-vrf-trap">The management VRF trap</h2>

<p>The most confusing version of this failure happens when management lives in a VRF. The switch sends its TACACS+ requests from the global routing table, the replies never come back, and every login waits on a timeout of a few seconds before it falls back. With command authorization on, so does every command. <code class="language-plaintext highlighter-rouge">show tacacs</code> shows packets sent and none received.</p>

<p>The fix is a server group that is pinned to the VRF, as Cisco documents in <a href="https://www.cisco.com/en/US/docs/ios-xml/ios/sec_usr_tacacs/configuration/15-1mt/Per_VRF_for_TACACS_Servers.html">Per VRF for TACACS+ Servers</a>:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aaa group server tacacs+ ISE_TACACS
 server-private 192.0.2.10 key &lt;key&gt;
 server-private 192.0.2.11 key &lt;key&gt;
 ip vrf forwarding MGMT
 ip tacacs source-interface Vlan100
</code></pre></div></div>

<p>Prove reachability from the right table with <code class="language-plaintext highlighter-rouge">ping vrf MGMT 192.0.2.10 source Vlan100</code>. A plain ping from the global table proves nothing. While you are there, align logging, SNMP traps, NTP and NetFlow sources with the same VRF and interface.</p>

<h2 id="before-you-bind-the-vty-lines">Before you bind the vty lines</h2>

<ol>
  <li>Test the server group directly with <code class="language-plaintext highlighter-rouge">test aaa group ISE_TACACS &lt;user&gt; &lt;password&gt;</code>. Options differ by release, so check them with <code class="language-plaintext highlighter-rouge">?</code> on the device first. Count it as a pass only when the output literally says the user was authenticated.</li>
  <li>Keep your current session open. Open a second session with the new login and prove it works before you close the first.</li>
  <li>Change the vty lines and the console in separate windows. Never both at once.</li>
  <li>Know your way back in before you start: a vty line still on local authentication, then taking the switch out of ISE so the server stops answering and <code class="language-plaintext highlighter-rouge">local</code> takes over, then the console, and a reload only if the bad state was never saved.</li>
</ol>

<h2 id="check-the-config-before-it-checks-you">Check the config before it checks you</h2>

<p>I wrote <a href="https://github.com/a-ayrapetov/aaa-lockout-check">aaa-lockout-check</a> to read a running-config and flag the shapes above before anyone binds a line: server lists with no fallback, console authorization that closes the last door, method lists or server groups that are referenced but never defined, a missing or dead source interface, and a source interface in a VRF that the server group does not use.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install git+https://github.com/a-ayrapetov/aaa-lockout-check
aaa-lockout-check "configs/*.cfg" --csv findings.csv
</code></pre></div></div>

<p>It reads text only and never connects to a device. Run it on the config you are about to push, not just the one that is running.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[Moving switch logins from local accounts to Cisco ISE TACACS+ is routine work, and it is also the change most likely to lock you out of your own network. On one migration of older switches, a method list in the wrong shape cost a full day and three lockouts. The failure looked exactly like a bad password every time, which is why it took so long to see.]]></summary></entry><entry><title type="html">Configuration drift as an audit control, not a chore</title><link href="https://a-ayrapetov.github.io/2026/10/07/config-drift-as-an-audit-control.html" rel="alternate" type="text/html" title="Configuration drift as an audit control, not a chore" /><published>2026-10-07T00:00:00+00:00</published><updated>2026-10-07T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/10/07/config-drift-as-an-audit-control</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/10/07/config-drift-as-an-audit-control.html"><![CDATA[<p>Every network I have inherited had changes nobody could explain. An ACL line added during an outage and never removed. A trunk carrying a VLAN that was supposed to be retired. A port left in a test VLAN. None of these are dramatic, and all of them are how segmentation quietly erodes.</p>

<h2 id="why-auditors-care">Why auditors care</h2>

<p>Configuration control is not an abstract best practice. It appears directly in the frameworks most clients answer to: NIST SP 800-53 CM-3 (configuration change control) and CM-6 (configuration settings), CIS Control 4 (secure configuration of enterprise assets), and for industrial environments IEC 62443-3-3 SR 7.6 on network and security configuration settings. Each one asks the same question in different words: can you show that what is running is what you approved?</p>

<h2 id="the-minimum-viable-control">The minimum viable control</h2>

<p>You do not need a commercial platform to answer that question. You need three things:</p>

<ol>
  <li><strong>An approved baseline</strong> of every device’s running configuration, captured at a known-good point, ideally right after a change window closes.</li>
  <li><strong>A scheduled comparison</strong> of the live configuration against that baseline, with volatile lines (timestamps, byte counts, clock values) filtered so only real changes surface.</li>
  <li><strong>A record</strong> of every comparison, including the clean ones, because “we checked and nothing changed” is evidence too.</li>
</ol>

<p>That is what <a href="https://github.com/a-ayrapetov/cisco-config-drift">cisco-config-drift</a> does. It sends exactly one command to each device, <code class="language-plaintext highlighter-rouge">show running-config</code>, keeps credentials out of files, writes a unified diff for anything that changed, and logs a line per device per run. Exit codes make it easy to alert from Task Scheduler, cron or a pipeline.</p>

<h2 id="read-only-on-purpose">Read-only on purpose</h2>

<p>The tool never pushes configuration, and that is a design choice, not a missing feature. A drift detector that can also remediate is a change tool, and a change tool needs change control of its own. Keeping detection read-only means it can run on a schedule against production, including OT networks where an unplanned change can stop a process, without becoming a risk itself.</p>

<h2 id="what-to-do-with-drift">What to do with drift</h2>

<p>Every diff ends in one of two places: it is a legitimate change that missed the paperwork, so you approve it and re-baseline, or it is not, so you roll it back and find out how it happened. Either answer is better than not knowing.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[Every network I have inherited had changes nobody could explain. An ACL line added during an outage and never removed. A trunk carrying a VLAN that was supposed to be retired. A port left in a test VLAN. None of these are dramatic, and all of them are how segmentation quietly erodes.]]></summary></entry><entry><title type="html">A network hardening assessment in two read-only steps</title><link href="https://a-ayrapetov.github.io/2026/10/07/hardening-assessment-workflow.html" rel="alternate" type="text/html" title="A network hardening assessment in two read-only steps" /><published>2026-10-07T00:00:00+00:00</published><updated>2026-10-07T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/10/07/hardening-assessment-workflow</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/10/07/hardening-assessment-workflow.html"><![CDATA[<p>Clients ask for a hardening review in one of two moods: before an audit, or after a scare. Either way they want the same deliverable: what is weak, how bad it is, and exactly what to change. Here is the workflow I use, built so that nothing touches production except one read-only command.</p>

<h2 id="step-1-collect-read-only">Step 1: collect, read-only</h2>

<p>Pull every running-config with a single <code class="language-plaintext highlighter-rouge">show running-config</code> per device. <a href="https://github.com/a-ayrapetov/cisco-config-drift">cisco-config-drift</a> does this as its snapshot step, so the collection doubles as the client’s first approved baseline:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install cisco-config-drift
cisco-drift snapshot --inventory inventory.csv
</code></pre></div></div>

<p>Credentials come from environment variables or a prompt and never touch disk. If the client prefers to collect configs themselves, they can, and nothing in step 2 changes.</p>

<h2 id="step-2-audit-offline">Step 2: audit, offline</h2>

<p>Run <a href="https://github.com/a-ayrapetov/ios-hardening-audit">ios-hardening-audit</a> across the collected configs:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install ios-hardening-audit
ios-harden "baselines/*.cfg" --csv findings.csv
</code></pre></div></div>

<p>It checks 23 baseline items drawn from CIS Cisco IOS benchmarks and vendor hardening guides: Telnet on vty lines, missing access-class, <code class="language-plaintext highlighter-rouge">enable password</code> and weakly hashed secrets, local users stored as passwords, default or read-write SNMP communities, the HTTP server, missing AAA, remote syslog, NTP, banners and brute-force protection. Every finding carries the evidence line and the fix, and secrets are redacted from the output so the report can be shared.</p>

<h2 id="step-3-turn-findings-into-a-plan-not-a-list">Step 3: turn findings into a plan, not a list</h2>

<p>A raw list of two hundred findings is not a deliverable. What the client needs is:</p>

<ol>
  <li><strong>The handful of highs, by device.</strong> Telnet, default communities and cleartext credentials are the ones attackers actually use. These go first, usually in one change window.</li>
  <li><strong>Fleet-wide patterns.</strong> If forty switches share the same missing <code class="language-plaintext highlighter-rouge">access-class</code>, that is one template fix, not forty tickets.</li>
  <li><strong>Version caveats.</strong> Some defaults differ by IOS release, so every “not found in config” finding is confirmed against the running version before it goes in the report.</li>
  <li><strong>A re-run.</strong> After remediation, the same two commands produce the after picture. The diff between the two reports is the evidence the auditor wants.</li>
</ol>

<h2 id="why-read-only-matters">Why read-only matters</h2>

<p>Assessments that start by pushing an agent or a config change need their own change approval, and that alone can delay the work by weeks. A read-only collection plus an offline audit can usually start the same day, and it can run safely against OT and other networks where an unplanned change is not an option.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[Clients ask for a hardening review in one of two moods: before an audit, or after a scare. Either way they want the same deliverable: what is weak, how bad it is, and exactly what to change. Here is the workflow I use, built so that nothing touches production except one read-only command.]]></summary></entry><entry><title type="html">The ISE certificate that takes 802.1X down on a Tuesday</title><link href="https://a-ayrapetov.github.io/2026/10/07/ise-certificate-expiry.html" rel="alternate" type="text/html" title="The ISE certificate that takes 802.1X down on a Tuesday" /><published>2026-10-07T00:00:00+00:00</published><updated>2026-10-07T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/10/07/ise-certificate-expiry</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/10/07/ise-certificate-expiry.html"><![CDATA[<p>Certificate expiry is the most predictable outage in network access control, and it still catches teams every year. The expiry date has been printed on the certificate since the day it was issued. What fails is the process around it.</p>

<h2 id="what-breaks-and-how">What breaks, and how</h2>

<p>Cisco ISE carries several system certificates per node, each bound to one or more roles. Two of them matter most:</p>

<p><strong>The EAP certificate</strong> is what ISE presents to supplicants during EAP-TLS, PEAP and TEAP. When it expires, endpoints that validate the server certificate (as they should) refuse to continue. Wired and wireless 802.1X authentications start failing, and depending on your fallback design, devices land in a remediation VLAN or nowhere at all. Endpoints that already hold a session may keep working until reauthentication, which makes the outage roll in over hours rather than all at once and makes it harder to diagnose.</p>

<p><strong>The Admin certificate</strong> secures the management GUI and ISE’s own communication between nodes in the deployment. Replacing it restarts services on that node, so the renewal itself needs a change window, which is exactly why it gets put off.</p>

<p>Trusted certificates expire too. An expired root or intermediate in the trusted store breaks validation of every client certificate chained to it.</p>

<h2 id="why-alarms-are-not-enough">Why alarms are not enough</h2>

<p>ISE raises certificate expiration alarms, and in most deployments I have seen they land in a dashboard nobody watches daily or an inbox filter nobody reads. The fix is not more alarms. It is a check that runs on a schedule, produces a short list, and fails loudly.</p>

<h2 id="a-read-only-check-you-can-schedule">A read-only check you can schedule</h2>

<p><a href="https://github.com/a-ayrapetov/ise-health-audit">ise-health-audit</a> uses three GET calls on the ISE OpenAPI to list deployment nodes, their system certificates and the trusted store, then flags anything expired or inside a 30 or 60 day window, plus nodes that are not connected, missing admin HA, and self-signed certificates still serving Admin or EAP:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python ise_audit.py --pan ise-pan1.example.com --ca-bundle corp-ca.pem
</code></pre></div></div>

<p>It never writes to ISE. If the client will not grant API access, they can run the three GET calls themselves and hand over the JSON, and the tool audits it offline.</p>

<h2 id="the-process-that-actually-prevents-the-outage">The process that actually prevents the outage</h2>

<ol>
  <li>Run the check weekly and route a non-zero exit to a ticket, not an email.</li>
  <li>Treat the 60 day mark as the start of the renewal, not a reminder: request the certificate, book the window for the Admin role restart, and test the new EAP certificate against each supplicant type before it goes live.</li>
  <li>After renewal, re-run the check and attach the clean output to the change record.</li>
</ol>

<p>Sixty days is enough time to do this calmly. Six hours is not.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[Certificate expiry is the most predictable outage in network access control, and it still catches teams every year. The expiry date has been printed on the certificate since the day it was issued. What fails is the process around it.]]></summary></entry><entry><title type="html">Triaging interface errors across a whole switch stack</title><link href="https://a-ayrapetov.github.io/2026/10/07/triaging-interface-errors-at-scale.html" rel="alternate" type="text/html" title="Triaging interface errors across a whole switch stack" /><published>2026-10-07T00:00:00+00:00</published><updated>2026-10-07T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/10/07/triaging-interface-errors-at-scale</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/10/07/triaging-interface-errors-at-scale.html"><![CDATA[<p>Most physical-layer problems announce themselves in <code class="language-plaintext highlighter-rouge">show interfaces</code> long before a user opens a ticket. The trouble is volume: a stack of four 48-port switches is close to two hundred interfaces, and nobody reads two hundred counter blocks carefully. Here is the triage order I use, and the tool I wrote to do the reading for me.</p>

<h2 id="read-the-counters-as-symptoms-not-numbers">Read the counters as symptoms, not numbers</h2>

<p><strong>CRC errors</strong> mean frames arrived corrupted. On a healthy copper or fiber link the count stays at zero. A handful after a reboot or a cable move is noise; a count that keeps climbing is a physical problem until proven otherwise: the patch cable, the optic, a damaged panel port, or interference. I judge CRCs as a rate against input packets rather than as a raw number, because 50 CRCs on a port that moved a billion packets is a different story from 50 on a port that moved ten thousand.</p>

<p><strong>Late collisions</strong> on a full-duplex network should not exist. When you see them, or see an up interface running half duplex at all, the classic cause is a duplex mismatch: one side hard-coded, the other auto-negotiating. The fix is symmetry, auto on both ends or the same fixed setting on both ends.</p>

<p><strong>Runts and giants</strong> point two ways. Runts usually travel with duplex problems or a failing NIC. Giants usually mean an MTU mismatch between neighbors.</p>

<p><strong>Output drops</strong> are a capacity question, not a cabling one. A tiny fraction is normal on a busy uplink; a sustained rate points at congestion, queueing policy or microbursts.</p>

<p><strong>Interface resets</strong> climbing over time suggest a flapping link. Check the logs for up and down events before touching anything.</p>

<p><strong>err-disabled</strong> means the switch shut the port on purpose: port security, BPDU guard, storm control or link flap detection. <code class="language-plaintext highlighter-rouge">show interfaces status err-disabled</code> names the reason. Fix the cause before you bounce the port, or it will go right back down.</p>

<h2 id="let-a-parser-do-the-reading">Let a parser do the reading</h2>

<p>I wrote <a href="https://github.com/a-ayrapetov/cisco-interface-health">cisco-interface-health</a> to apply those rules to raw output. It parses <code class="language-plaintext highlighter-rouge">show interfaces</code> text offline, never connects to a device, and ranks findings by severity with a suggested next step:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install cisco-interface-health
ifhealth show_interfaces.txt --csv findings.csv
</code></pre></div></div>

<p>Because it only reads text, it is safe to run on output a client emails you, and the CSV drops straight into a ticket or a report. Thresholds sit at the top of the script so you can tune them for your environment.</p>

<h2 id="then-work-the-layers">Then work the layers</h2>

<p>The ranked list tells you where to look, not what to conclude. Clear counters, wait, and re-collect before calling anything fixed. A counter that stops climbing after a cable swap is evidence; a counter you reset and never re-checked is not.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[Most physical-layer problems announce themselves in show interfaces long before a user opens a ticket. The trouble is volume: a stack of four 48-port switches is close to two hundred interfaces, and nobody reads two hundred counter blocks carefully. Here is the triage order I use, and the tool I wrote to do the reading for me.]]></summary></entry><entry><title type="html">When TrustSec looks like a routing problem</title><link href="https://a-ayrapetov.github.io/2026/10/07/trustsec-looks-like-routing.html" rel="alternate" type="text/html" title="When TrustSec looks like a routing problem" /><published>2026-10-07T00:00:00+00:00</published><updated>2026-10-07T00:00:00+00:00</updated><id>https://a-ayrapetov.github.io/2026/10/07/trustsec-looks-like-routing</id><content type="html" xml:base="https://a-ayrapetov.github.io/2026/10/07/trustsec-looks-like-routing.html"><![CDATA[<p>The ticket said the domain controllers were down. They were not. They answered DNS, Kerberos and LDAP all day. They just would not answer a ping, and every monitoring check, everyone’s first test and half the troubleshooting runbooks start with a ping. Two hours of looking at routing tables later, the cause turned out to be one cell in a TrustSec policy matrix.</p>

<h2 id="how-it-happens">How it happens</h2>

<p>TrustSec enforces policy between Security Group Tags (SGTs), not between IP addresses. Every packet gets a source tag; the destination gets a tag from an IP-to-SGT mapping, from the authorization result of the port it sits on, or, if nothing classifies it, the Unknown tag (SGT 0). The switch then applies whatever SGACL sits in the matrix cell for that source and destination pair.</p>

<p>Servers are the classic gap. Users and endpoints get tagged at authentication, but servers often connect to ports that never authenticate, or sit behind a routed hop where no IP-to-SGT mapping was ever created. They end up as Unknown, or under a broad catch-all tag shared with things nobody thought about.</p>

<p>Now add a reasonable-looking policy: someone writes an SGACL for that destination tag that permits the application ports and ends with a deny for everything else, or explicitly denies ICMP to keep scanners quiet. Every real service keeps working. Ping dies. The symptom points straight at Layer 3, and the cause is two layers of policy away from where anyone is looking.</p>

<h2 id="how-to-prove-it-in-minutes">How to prove it in minutes</h2>

<p>The switch keeps per-SGT-pair counters, and they settle the question faster than any trace:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>show cts role-based sgt-map &lt;server-ip&gt;
show cts role-based permissions to &lt;destination-sgt&gt;
show cts role-based counters to &lt;destination-sgt&gt;
</code></pre></div></div>

<p>The first tells you which tag the server actually has, which is often not the one you assumed. The second shows the SGACL in the cell. The third shows deny counters climbing while you ping. If the deny counter moves in step with your failed pings, routing is innocent.</p>

<p>On the ISE side, the policy matrix view for that destination tag shows the same cell and who last changed it.</p>

<p>Command options differ across IOS-XE releases, so check the exact syntax for your version.</p>

<h2 id="how-to-stop-it-happening-again">How to stop it happening again</h2>

<ol>
  <li>Classify servers deliberately. Static IP-to-SGT mappings or subnet-to-SGT bindings for server ranges, pushed from ISE, so no server is accidentally Unknown.</li>
  <li>Decide on ICMP as policy, not as a side effect. Permitting echo for troubleshooting into infrastructure tags is a small cost for a large gain in diagnosability.</li>
  <li>Put the per-pair counters into the troubleshooting runbook, right after “can you ping it”. The first question for a segmented network is not “is it routed” but “what tag is it, and what does the cell say”.</li>
  <li>Review matrix changes like firewall changes. A single cell edit has the blast radius of a firewall rule and deserves the same change control.</li>
</ol>

<h2 id="the-broader-lesson">The broader lesson</h2>

<p>Segmentation moves part of the forwarding decision out of the routing table and into policy. Once that happens, “it does not ping” stops being a Layer 3 symptom by default. Teams that adopt TrustSec need to update their mental model, and their first-response checklist, to match.</p>]]></content><author><name>Arkadiy Ayrapetov</name></author><summary type="html"><![CDATA[The ticket said the domain controllers were down. They were not. They answered DNS, Kerberos and LDAP all day. They just would not answer a ping, and every monitoring check, everyone’s first test and half the troubleshooting runbooks start with a ping. Two hours of looking at routing tables later, the cause turned out to be one cell in a TrustSec policy matrix.]]></summary></entry></feed>