Our goal as AV field techs should be first-call resolution. To make that happen and be pro-active, I try to follow some basic guidelines.
Brett Cohen
As soon as I get a ticket assigned, I call ahead to confirm contact info and address. I do this because tickets are often auto-populated by CRM software with outdated contact info or the HQ address, even if the ticket is for a remote site. Do everybody a favor, and let the help desk know about contact and address corrections.
I review the reported issue and if something doesn’t sound right, I ask the help desk for the actual e-mail from the customer or the reported issue in the customers own words.
DOCUMENTATION ESSENTIALS
Don’t wait till you are on-site to start downloading important documentation of your customer’s system. You may get blocked by their firewall. Grab Audio, Video, and Control diagrams, and your QRGs (Quick reference guides) while on your own company’s network.
Got a ticket for a complex site? Ask your CAD team for full-sized, and shove it in the rack when you are done for the next tech. Or, head to your local office supply super store and spend a few bucks on full-size print-outs. I’ve never had that expense questioned.
Room numbers and room names will change between design, install, commissioning, and client turnover. When you see those changes, report them to your team so they can be applied to the ticket systems and drawings.
DON’T LEAD THE WITNESS
When I arrive on-site, I ask to be shown the exact issue(s) they are experiencing. It’s good policy to squelch your pride, and ask them to show you how the system is supposed to work—for them. Personally, I can’t recall how every customer’s system works, and let’s face it, we work on hundreds of varying systems and touch panels. Often we can uncover a simple user error and be on our merry way.
Some key questions to ask are: When did you first notice the issues? Have there been any upgrades on the company’s network or operating systems? Any recent power outages, surges, or storms?
Has anybody been troubleshooting on their own? Sometimes after an innocent customer states “No, nobody’s touched it,” I’ll find an RS-232 3.5mm plugged into IR out jack, or a RJ-45 moved from a DM port to the LAN port. Hmmm.
BURN OUT, BURN OUT
Scenario: Last call of the day and you are exhausted and baffled. Here are some options that I use to clear my head and rethink my strategy. I call my wife and kids to tell them I’m not coming home anytime soon (so they won’t be calling every 10 minutes).
The monitor radiation, fluorescent lights, rack, and projector fan noise can often cloud my thinking, so I walk out of the room to clear my head. If I’m still stuck I prepare to call my help desk or another tech. Often, I say out loud what I am going say to my colleagues at the help desk about the symptoms and steps I have taken, and the fix or the cause comes to me before I have to make the call.
RIKKI DON’T LOSE THAT NUMBER
Create and keep a log of your company’s commonly used system passwords and default manufacturers passwords. Because I keep mine in a 99¢ analog iPad, of course I use code names to represent the devices and site names.
I also do the same for Wi-Fi SSIDs and passwords, bathroom, and room access codes.
I once picked up and turned in an iPhone that was left in a conference room I was servicing. Because the screen was locked, I could not determine to whom it belonged. That day I realized I was at risk of the same fate. So, I took my home screen photo, and using MS Paint, I added an alternate phone number to it. That’s one less thing I need to worry about.
MANUFACTURERS’ GOODIES: PORTALS YOU NEED TO BOOKMARK
HTTPS://V3.SAMSUNGGSBN.COM
Videowall color balance using Color Expert Software
HTTP://DISPLAYSOLUTIONS.SAMSUNG.COM
Video tutorials and their videowall configurators
HTTP://CRESTRON.COM> PROPANEL > ONLINE HELP
Search the published answer IDs. You might find info that could cut your troubleshooting in half, and avoid on-hold time when calling for support.
HTTPS://SUPPORT.BIAMP.COM
Technical Support Database a.k.a. Cornerstone
CHECK ONE, TWO
Ceiling mics can be a little hard to isolate for verification. To test for functionality, I use a long shipping tube, bring it close to each mic, and with a very gentle tap I can hear it through the mic loud and clear, without any bleed through into the other mics.
This can be also used for speaker verification, but I usually just cup my ears. Try this during the next concert you attend. It’s a whole new experience.
CRESTRON: WHERE WOULD WE BE WITHOUT THEM?
Make use of your Info, View, and Route buttons on the front panel of your Crestron DM Switchers.
Press Info, then select input and output buttons to get signal details and device names.
Press View, then select either an Input or Output to see what is routed to it by watching the In & Out indicator lights.
Using your Route button when in a pinch to restore service when your touch panel is not functioning:
Press Route then desired Input followed by desired Output (or vice versa depending on your model). Don’t forget to take the route by hitting Enter. If you want to access more details via the Menu button, you can often enter your password using the Input selection button instead of the tedious rotary dial.
If you are a site facilitator, it’s good practice to double-check the labels on the Crestron switcher. Re-label them to something more meaningful to your application. Instead of HDMI 1, label it name of the source device. You can also edit the display names for the info buttons. Got some extra time? Make a two-column Excel spreadsheet that has A-end and a B-end for every cable and tape it inside your rack. That will save you the time, hand cramps, skin cuts and abrasions of going hand-over-hand to trace out cables if you have a failure.
IS INSIDE-OUT THE SAME AS OUTSIDE-IN?
Only if you got dressed in the dark. But if you are upgrading Crestron firmware, you’ll want to use the Inside out method. Start with the DM Switcher firmware, then the cards, then the transmitters and finally scalers/receivers. When you do the transmitters and receivers, make sure you are not doing them from the device itself or you will lose your connection in the process. Do these remotely from the frame. Always use Ethernet for these upgrades as wireless and USB will be too slow or leave you hanging.
RETURN TO SENDER
Crestron RMAs: Don’t wait on hold to submit an RMA. Submit them via E-mail, include Item model #, S/N, firmware. Include the associated components (run a DM report) Firmware that it communicates with. E.g., If it’s a TX-200 transmitter, include the details of the receiver or the receiver card and DM chassis info. Also, include the trouble shooting techniques you have tried and the results. You already know what Crestron is looking for, so just include it. If you provide all the right info, you may get your RMA confirmation E-mail without further questions.
SEQUENCE DOES MATTER
We all know that a reboot is a quick way to restore service, albeit we don’t really find the root cause that way. There are many opinions about what order to boot-up your rack. Polling for states and initialization can vary depending on hardware and coding. So, suffice it to say, if you don’t get your desired results with one re-boot sequence, try another.
MACGYVER STUFF
How do you color balance a video wall if you can’t get the camera to rest against a completely upright monitor? Stand there and hold it with one hand while you run your software with the other? Not unless you’re Mr. Fantastic or Plastic Man. Instead pick up some shock cord (bungee cord) make a large loop and stretch it from one corner of the monitor to the other. Now tuck your camera in it. If you size it correctly, the pressure will be gentle yet secure. I’ve had a lot of success using an X-Rite i1 Display Pro and Samsung’s Color Expert Software.
Selfie sticks are great for capturing S/Ns off of hanging monitors if you don’t have the remote.
Carabiner-o-adapters: I saw one of those Apple Store Ninja guys with one of these. He had every adapter he needed right on his hip. Use wire ties to fashion a hook point on any connector or adapter.
PICK A PORT…ANY PORT
Need to connect to a device, and you can’t figure out what its IP is? Run a scan with WireShark and look for IP addresses associated with familiar manufacturer ports:
AMX ports: 1319, 1320, 292, 2930, 3839
Biamp ports: 12000 - 12003, 61451
Crestron ports: 41794, 41795, 41796, 41797
Once you have a suspect IP, you can use the ping and pull method to identify the device. That is pull the device’s Ethernet cable off to see if the ping drops.
ping 10.0.0.217 -t (the -t gives you a non-stop ping, stop it with cntrl-c.)
If you have already pinged off your screen, add the time stamp “switch” by adding a -s and any number between 1 – 4 Ex. ping 10.0.0.217 -t -s 2
WithWireshark or other network scanning tools, use caution and make sure your AV gear is on its own network and if not, make sure you have express permission to scan. Otherwise you could find yourself escorted out the door by IT Security staff.
REALLY. IT’S NOT YOU, IT’S ME.
It’s awful to get dumped, but Crestron’s Dumpdmipconfig is a useful command for providing the port numbers of attached scalers/transmitters which then allows you to start a telnet session with that device. Use switcher ip:port of connected AV devices to issue reset or initialize commands, check IP table setup, issue ping commands, etc.
INVASION OF THE PORT SNATCHERS
You want to have eyes and ears on a room as you work, but what if the rack is in some far off telco/data closet? Find yourself an unused data port in your room and use the corresponding patch panel connection in the telco room to jump over to your AV rack network port. Now you are in. Just don’t forget to restore it.
As an alternative, I’ve seen many field engineers and programmers use wireless routers to accomplish remote access.
If you do use a wireless router to connect your laptop to the AV subnet via the rack switch, take Dale Becker/FE’s advice: Double check that your router has its DHCP server turned off or you run the risk of dishing out an inappropriate IP address onto that subnet and your AV gear. Also, remove any connections that go back to the client network because you might cause a port shutdown when the customers’ data switch senses your rogue router.
Important: Don’t forget to connect the client connect back before leaving.
APPLE SOURCE
Another goodie from Dale Becker, CTS, DMC-E-4k: “Apple products have HDCP enabled at all times, regardless of whether the content being viewed on them is HDCP protected or not,” he said. “Devices like video conferencing codecs and digital recorders will not pass HDCP content, therefore your programmer will need to disable HDCP support for a source/sink pair like AppleTV/Codec, but enable it if your source/sink pair is for presentation purposes, e.g, DVD player/Samsung monitor.”
YOU AIN’T SEEN NOTHIN’ YET
Have tired or weak eyes and need to make network or DM cables? Try the Platinum Tools EZ-RJ45 Crimp Tools for Shielded and standard RJ45 EZ connectors. The wires pop out the end so you can pull the jacket all the way into the strain relief tab and you can make sure you’ve got the color sequence right.
ORDER OF THE PHOENIX
Isolate your point of failure. I’ve been saved so many times with a $20 pair of folding headphones and a 3.5mm barrel connector to a Hosa Technology Phoenix adapter to check for line level audio. Use the same barrel to take the output of your phone’s headphone jack to inject audio into a DSP or input card. Bring up you levels slowly on these to be safe!
QUICK TEST CALLS
Check out http://thetestcall.blogspot.com for a list of phone numbers that you can call to test DTMF tones, latency times, and audio quality with frequency sweeps.
TEST EQUIPMENT
Got one of those Atlona HD800 signal generators, the ones without a readout? Obviously, you can’t determine which resolution your monitor won’t pass if you don’t know what you are sending to it. Procure a 1 x 2 HDMI splitter so you can use your AT-DIS7-PROHD portable monitor to decipher what you are sending to the monitor in question.
Gripe #1
VELCRO, IT’S WONDERFUL THING, BUT…
Installers, please stop putting Velcro over the manufacturer’s stickers;we can’t see S/N, P/N or MAC addresses. And when we try to peel the Velcro off, you know what happens….d-oh!
Gripe #2
2 RU KIDDING ME?
One of a field tech's worst nightmares: a 2 RU ceiling box. These things are populated in the shop with all the necessary gear and cabling to run a whole room. The last time I encountered one of these, the cabling was so tight across both shelves that there was not one component I could safely access to test or service. Hey, rack builders, remember that us field techs are working on a ladder against gravity, not at a workbench. Show some love and think of us when you assemble these. Make them serviceable.
Brett Cohen’s career has had its highs and lows, from satellite communication to transatlantic fiber operations, and almost everything in between, including being an inventor and a chauffeur. Currently, he is an AV Project Coordinator at a large corporate facility. He holds Cisco CCNA and CTS certifications. Connect with him via linkedin.com/in/bretthcohen.
