Tech Support Intern Training Lesson Plan

Tech Support Basics

Aims

To give a new Tech Support intern an understanding of the basic tools and procedures used in Tech Support

Objectives

1. Understand the pupose of the ticket sytem
2. Know how to create a new ticket
3. " " " comment and reply on a ticket and understand the difference
4. " " " escalate a ticket
5. " " " resolve a ticket/change its status and when
6. " " " use the dashboards and how this fits in with our workflow
7. Be able to use command line tools including command line args
8. Use and interpet the output of free, du, df and top
9. Be able to mount an external drive
10. " " " ssh into a remote box
11. " " " view and edit files
12. " " " direct command output to a file or another command
13. " " " create and restore network backups using tstools
14. have a basic understanding of the manual equivalent
15. Be able to boot a box into recovery mode and make the hard drive writable
16. Be able to start a net boot
17. Know how to get useful debugging output from a command
18. Identify common logs
19. Have an understanding of the tools that can aid in interpreting them
20. Be able to identify a consistent approach to trouble shooting

Intro 5 mins 5 The First, Second and Third Line Model 5 mins 10 What it means in the real world Verbal Instruction 1st Line support desk, log calls, answer common questions from script, assign to technicians 2nd Line solve day to day problems, may do call backs, fix boxes 3rd Line Guru's. Solve new problems, train + document, proactively manage potential issues What it means in Free Geek 1st Line Answer phones, do intake, create tickets, call people to let them know their machines are ready 2nd Line Fix boxes, do callback for problems 1st line techs can't solve. 3rd Line Staff RT Basic concepts Ticket 15 mins 25 Purpose Heart of all TS departments. Enables issues to be tracked over time and people. Make a New Ticket Verbal Instruction Click on new ticket/http://data.fglan/work_orders/new?mode=ts Subject Format - Name, Summary Important Fields Phone No Box Source, type, ticket type. What to write and when Everything, as soon as it happens, if not before Owning a ticket Important in the outside world Who is working on this Need to take/unown tickets Escalating 1->2->3 When? When you don't have the answer When to call Everyday until you speak to someone in person Phone messages get three strikes Conventions and abbreviations Left VM -- Date CNR Avoid Jargon Comments vs Replies 2.5 mins Comments are internal Replies go to user Sopmetimes need to change requestor Be careful with subject lines Workflow 2.5 mins 27.5 New -> Open (->contact)->pending->resolved Status 5 mins 32.5 New untouched be careful not to change status if you add notes Open actively being worked on Contact custom status ready for pickup Stalled Pending boxes waiting to be picked up atypical (normally more like contact/stalled Resolved EXERCISE: Create Ticket Students to create 1 ticket in Sandbox Queue My name is... Comment on ticket Comment on ticket of person to left Resolved ticket of person to right Custom Fields etc 2.5 mins 40 Basic name etc Status fields specific to Free Geek Links refers to/by merge can not undo depends on parent/child People requestor cc owner Dashboards 2.5 mins Line 1 lists phone calls and messages, boxes ready for pickup Line 2 Boxes on the bench, new and open Always work in chronological order follow list Line 3 For staff Working on the command line Commands 5 mins 45 pwd, ls Commands are like work in a sentence simple verbs Fetch! list! e.g. pwd, ls, cd, rm, mv EXERCISE: Open Terminal type whoami, pwd read back and interpret results basic arguments nouns fetch this box, list this directory e.g ls /home Options 5 mins 50 adverbs, Fetch quickly, list fully e.g. ls -alh - and -- short and long (gnu) form common options -h, --help -v, --verbose -R, recursive differ between commands can be combined with noun form -o [file] EXERCISE: type ls, ls /home, ls ~/, ls -a, ls -al, ls -alh, ls --almost-all

Common commands 20 mins 1hr 10 man free df df -h du du -sh du -ch --max-depth=1 EXERCISE: Use df and du to determine total size of hard drive and size of usr directory ps ps aux kill, killall, pkill, pgrep top uses ncurses half way to gui EXERCISE: Use ps and top to idntify processes running under your user mount mount (-t option) device mountpoint ssh used for connecting to remote machine securely log into a box anywhere ssh me@there scp secure copy scp here me@there: note colon otherwise works as cp sftp secure ftp less, view etc head head file head -10 file tail tail -f pager used to view files less is more go backwards and forwards, search view vi in readonly mode, usefull for syntax and commands editors 10 mins 1hr 20 nano easy(ish) only standard on Debian vi on nearly every *nix box vi(m) -- vi (i)m(proved) steep learning curve but very powerful worth the effort EXERCISE: 05 min 1hr 25 open vim: vim hello-world write hello world i... duplicate line yy... ESC..p delete world dw write and quit (colon)wq

TS Tools ts_network_backup 15 mins 1hr 40 basic backup ts_network_backup -c [ticket no] does everything for you advanced backups backing up when a drive is attached to another box -p option alternative path to home command line options -a other things to back up -u, -d backup less restoring backups ts_network_backup -r [folder] EXERCISE: create command line for backup using one option, say what you would use to restore it. ts_identify 5 mins 1hr 45 identifies backup folder ts_identify_backups (-t) ticket number others do what they say on the tin, no options except -h File system basics 10 mins 1hr 55 / /bin etc bin, sbin,opt, root, tmp, lib var cache, mail, www where servers store things usr for users repeats hierarchy also doc share src /etc config files /etc/apt filesystem /etc/fstab & /etc/mtab network /etc/host /etc/resolv.conf users and groups passwd, shadow, group, gshadow /var/log where log files are stored\ whats going on /home where you live users files /proc & /sys not really files contain system info cat /proc/cpuinfo /mnt & /media mount discs Manual Backups 10 mins 2hr 05 Go over page, backup config backup users and groups get list of installed packages transfer data rsync -avzh here me@there:/var/tsbackup backup name ticket -iso date iso date why its important

Rooting a box 15 mins 2hr 20 why password security if you have physical possesion of box there is no security recovery mode press shift when after bios screen 1st menu recovery mode 2nd menu rootshell not networking remounting makes file system writable mount -o rw,remount / starting networking dhclient eth0 etc network booting the network boot menu change bios settings, go tech support -> debian rescue why use a recovery disk specialist tools efficient mounting a hard drive mount (-t) drive mountpiint chroot'ing chroot /mnt for i in dev proc sys dev/pts; do mount $i /mnt/$i; done EXERCISE: In pairs,Root box and netboot to debian rescue Diagnosing problems through log files and error message running in the terminal 5 mins 2hr 25 verbose mode -v --verbose multiple levels debug modes log levels warn, info, error dmesg 5 mins 2hr 30 /var/log/syslog essential info what the system does EXERCISE: tail -f /var/log/syslog I/O Redirection 5 mins 2hr 35 pipe to command | pipe to file < > >> stdout & stderr 1 > /dev/null redirect std out (makes it disappear 2 > error.file redirect stderr &> log both to log EXERCISE: with ls send stdout and stderr to /dev/null and observe results grep 10 mins 2hr 45 search tool global regular expression parser 'Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.' no need to use regex for most things a word is a regex that says match this word awk (and sed) sed was the cause of the regex quote can be very useful but unlikely you will need to use it command line search and replace awk was the source of the quote whole programming language for dealing with tabular data you only need to know tow things awk '{print $1}' awk '{print NF}' cat /var/log/apache2/access.log | awk '{print $2}' | sort | uniq -c |sort -g EXERCISE: use grep and awk to get time and date of kernel events using awk (as group on board, use questions and prompts)

How to troubleshoot 15 mins 3hr Importance of consistency Not your machine Do not break it Repeatable solutions Needs to diagnose problem not just fix it reinstall is not the answer Problem solving https://en.wikipedia.org/wiki/Problem_solving Read this page(write on board) OODA loop (Observe, Orient, Decide, Act) https://en.wikipedia.org/wiki/OODA_loop Comes from military/fighter pilots designed for stressful situations and quick reactions important part it is a loop constant feedback ability to change course Observe what is going on Orient where might the problem lie Decide how can we test this Act run test Observe what where the results Orient what did they tell us Decide whats the solution to fix the problem Act apply solution Observe did it work? etc... PDCA (Plan Do Check Act) https://en.wikipedia.org/wiki/PDCA Plan Study the problem, establish the objectives, devise test Do Carry out test, collect data where there any error meessages? what did the log files say? Check Study results, compare to what was expected Act Analyse differences, determine root causes, corrective actions, next steps repeat if necessary RPR (Rapid Problem Resolution) https://en.wikipedia.org/wiki/RPR_Problem_Diagnosis Discover, Investigate, Fix IT Specific Discover Gather and review information Build model Investigate Create and carry out plan to gather data analyse results and iterate Identify root cause Fix Determine and implement fix check root cause worked Example The Facebook isn't working' Discover is it facebook or the internet Investigate ping facebook No -- iterate ping google No -- iterate ping router No -- the internet is down Fix reboot router open browser visit facebook EXERCISE: A box comes in that the user reports is not booting In pairs, Chose one method and outline the steps to solve the problem

Total Time 3 hours, not incrluding a break.