Tech Support Intern Training Lesson Plan

Tech Support Basics

Aims

To give a new Tech Support intern an understanding of the basic tools and procedures used in Tech Support

Objectives

1. Understand the pupose of the ticket sytem
2. Know how to create a new ticket
3. " " " comment and reply on a ticket and understand the difference
4. " " " escalate a ticket
5. " " " resolve a ticket/change its status and when
6. " " " use the dashboards and how this fits in with our workflow
7. Be able to use command line tools including command line args
8. Use and interpet the output of free, du, df and top
9. Be able to mount an external drive
10. " " " ssh into a remote box
11. " " " view and edit files
12. " " " direct command output to a file or another command
13. " " " create and restore network backups using tstools
14. have a basic understanding of the manual equivalent
15. Be able to boot a box into recovery mode and make the hard drive writable
16. Be able to start a net boot
17. Know how to get useful debugging output from a command
18. Identify common logs
19. Have an understanding of the tools that can aid in interpreting them
20. Be able to identify a consistent approach to trouble shooting

Intro -- The First, Second and Third Line Model

	Time	Total
Intro	5 mins	5
The First, Second and Third Line Model	5 mins	10
What it means in the real world (Verbal Instruction) 1st Line support desk, log calls, answer common questions from script, assign to technicians 2nd Line solve day to day problems, may do call backs, fix boxes 3rd Line Guru's. Solve new problems, train + document, proactively manage potential issues What it means in Free Geek 1st Line Answer phones, do intake, create tickets, call people to let them know their machines are ready 2nd Line Fix boxes, do callback for problems 1st line techs can't solve. 3rd Line Staff

RT

Basic concepts

• • Comments are internal
  • Replies go to user
    • Sopmetimes need to change requestor
    • Be careful with subject lines

Ticket	15 mins	25
Purpose Heart of all TS departments. Enables issues to be tracked over time and people Make a New Ticket (Verbal Instruction) Click on new ticket/http://data.fglan/work_orders/new?mode=ts Subject Format - Name, Summary Important Fields Phone No Box Source, type, ticket type. What to write and when Everything, as soon as it happens, if not before Owning a ticket Important in the outside world Who is working on this Need to take/unown tickets Escalating 1->2->3 When? When you don't have the answer When to call Everyday until you speak to someone in person Phone messages get three strikes Conventions and abbreviations Left VM -- Date CNR Avoid Jargon
Comments vs Replies	2.5 mins
Workflow	2.5 mins	27.5
New -> Open (->contact)->pending->resolved
Status	5 mins	32.5
New untouched be careful not to change status if you add notes Open actively being worked on Contact custom status ready for pickup Stalled Pending boxes waiting to be picked up FG use is atypical (normally more like contact/stalled in regular Tech Support) Resolved EXERCISE: Create Ticket Students to create 1 ticket in Sandbox Queue My name is... Comment on ticket Comment on ticket of person to left Resolved ticket of person to right
Custom Fields etc	2.5 mins	40
Basic name etc Status fields specific to Free Geek Links refers to/by merge can not undo depends on parent/child People requestor cc owner
Dashboards| 2.5 mins
Line 1 lists phone calls and messages, boxes ready for pickup Line 2 Boxes on the bench, new and open Always work in chronological order follow list* Line 3 For staff

Working on the command line

Basic Commands

Commands	5 mins	45
pwd, ls Commands are like work in a sentence simple verbs Fetch! list! e.g. pwd, ls, cd, rm, mv EXERCISE: Open Terminal type whoami, pwd read back and interpret results basic arguments nouns fetch this box, list this directory e.g ls /home
Options	5 mins	50
adverbs, Fetch quickly, list fully e.g. ls -alh - and -- short and long (gnu) form common options -h, --help -v, --verbose -R, recursive differ between commands can be combined with noun form -o [file] EXERCISE: type ls, ls /home, ls ~/, ls -a, ls -al, ls -alh, ls --almost-all
Common commands	20 mins	1hr 10
man free df df -h du du -sh du -ch --max-depth=1 EXERCISE: Use df and du to determine total size of hard drive and size of usr directory ps ps aux kill, killall, pkill, pgrep top uses ncurses half way to gui EXERCISE: Use ps and top to idntify processes running under your user mount mount (-t option) device mountpoint ssh used for connecting to remote machine securely log into a box anywhere ssh me@there scp secure copy scp here me@there: note colon otherwise works as cp sftp secure ftp less, view etc head head file head -10 file tail tail -f pager used to view files less is more go backwards and forwards, search view vi in readonly mode, usefull for syntax and commands
editors	10 mins	1hr 20
nano easy(ish) only standard on Debian vi on nearly every * nix box vi(m) -- vi (i)m(proved) steep learning curve but very powerful worth the effort
EXERCISE:	05 min	1hr 25
open vim: vim hello-world write hello world i... duplicate line yy... ESC..p delete world dw write and quit (colon)wq

TS Tools

ts_network_backup	15 mins	1hr 40
basic backup ts_network_backup -c [ticket no] does everything for you advanced backups backing up when a drive is attached to another box -p option alternative path to home command line options -a other things to back up -u, -d backup less restoring backups ts_network_backup -r [folder] EXERCISE: create command line for backup using one option, say what you would use to restore it.
ts_identify	5 mins	1hr 45
identifies backup folder ts_identify_backups (-t) ticket number others do what they say on the tin, no options except -h

Linux File System

File system basics	10 mins	1hr 55
/ /bin etc bin, sbin,opt, root, tmp, lib var cache, mail, www where servers store things usr for users repeats hierarchy also doc share src /etc config files /etc/apt filesystem /etc/fstab & /etc/mtab network /etc/host /etc/resolv.conf users and groups passwd, shadow, group, gshadow /var/log where log files are stored whats going on /home where you live users files /proc & /sys not really files contain system info cat /proc/cpuinfo /mnt & /media mount discs

Manual Backups

Manual Backups	10 mins	2hr 05
Go over wiki page backup config backup users and groups get list of installed packages transfer data rsync -avzh here me@there:/var/tsbackup backup name ticket -iso date iso date why its important

Working Without Passwords

Rooting a box	15 mins	2hr 20
why password security if you have physical possesion of box there is no security recovery mode press shift when after bios screen 1st menu recovery mode 2nd menu rootshell not networking remounting makes file system writable mount -o rw,remount / starting networking dhclient eth0 etc network booting the network boot menu change bios settings, go tech support -> debian rescue why use a recovery disk specialist tools efficient mounting a hard drive mount (-t) drive mountpiint chroot'ing chroot /mnt for i in dev proc sys dev/pts; do mount $i /mnt/$i; done EXERCISE: In pairs,Root box and netboot to debian rescue

Diagnosing problems through log files and error message

running in the terminal	5 mins	2hr 25
verbose mode -v --verbose multiple levels debug modes log levels warn, info, error
dmesg	5 mins	2hr 30
/var/log/syslog essential info what the system does EXERCISE: tail -f /var/log/syslog I/O Redirection***********5 mins**2hr 35 pipe to command \| pipe to file < > >> stdout & stderr 1 > /dev/null redirect std out (makes it disappear 2 > error.file redirect stderr &> log both to log EXERCISE: with ls send stdout and stderr to /dev/null and observe results
grep	10 mins	2hr 45
search tool global regular expression parser 'Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.' no need to use regex for most things a word is a regex that says match this word awk (and sed) sed was the cause of the regex quote can be very useful but unlikely you will need to use it command line search and replace awk was the source of the quote whole programming language for dealing with tabular data you only need to know tow things awk '{print $1}' awk '{print NF}' cat /var/log/apache2/access.log | awk '{print $2}' | sort | uniq -c |sort -g EXERCISE: use grep and awk to get time and date of kernel events using awk (as group on board, use questions and prompts)

Trouble Shooting and Problem Solving

How to troubleshoot	15 mins	3hr
Importance of consistency Not your machine Do not break it Repeatable solutions Needs to diagnose problem not just fix it reinstall is not the answer Problem solving https://en.wikipedia.org/wiki/Problem_solving Read this page(write on board) OODA loop (Observe, Orient, Decide, Act) https://en.wikipedia.org/wiki/OODA_loop Comes from military/fighter pilots designed for stressful situations and quick reactions important part it is a loop constant feedback ability to change course Observe what is going on Orient where might the problem lie Decide how can we test this Act run test Observe what where the results Orient what did they tell us Decide whats the solution to fix the problem Act apply solution Observe did it work? etc... PDCA (Plan Do Check Act) https://en.wikipedia.org/wiki/PDCA Plan Study the problem, establish the objectives, devise test Do Carry out test, collect data where there any error meessages? what did the log files say? Check Study results, compare to what was expected Act Analyse differences, determine root causes, corrective actions, next steps repeat if necessary RPR (Rapid Problem Resolution)* https://en.wikipedia.org/wiki/RPR_Problem_Diagnosis Discover, Investigate, Fix IT Specific Discover Gather and review information Build model Investigate Create and carry out plan to gather data analyse results and iterate Identify root cause Fix Determine and implement fix check root cause worked Example The Facebook isn't working' Discover is it facebook or the internet Investigate ping facebook No -- iterate ping google No -- iterate ping router No -- the internet is down Fix reboot router open browser visit facebook EXERCISE: A box comes in that the user reports is not booting In pairs, Chose one method and outline the steps to solve the problem

Total Time 3 hours, not incrluding a break.