Tech Support Intern Training Lesson Plan

Aims
To give a new Tech Support intern an understanding of the basic tools and procedures used in Tech Support

Objectives
" " " use the dashboards and how this fits in with our workflow	Be able to use command line tools including command line args	Use and interpet the output of free, du, df and top	Be able to mount an external drive	" " " ssh into a remote box " " " view and edit files	" " " direct command output to a file or another command " " " create and restore network backups using tstools	have a basic understanding of the manual equivalent	Be able to boot a box into recovery mode and make the hard drive writable 	Be able to start a net boot	Know how to get useful debugging output from a command	Identify common logs	Have an understanding of the tools that can aid in interpreting them	Be able to identify a consistent approach to trouble shooting
 * 1) Understand the pupose of the ticket sytem
 * 2) Know how to create a new ticket
 * 3) " " " comment and reply on a ticket and understand the difference
 * 4) " " " escalate a  ticket
 * 5) " " " resolve a ticket/change its status and when

Intro														5 mins		5 The First, Second and Third Line Model 						5 mins		10 What it means in the real world							Verbal Instruction 1st Line support desk, log calls, answer common questions from script, assign to technicians 2nd Line solve day to day problems, may do call backs, fix boxes 3rd Line Guru's. Solve new problems, train + document, proactively manage potential issues What it means in Free Geek 1st Line Answer phones, do intake, create tickets, call people to let them know their machines are ready 2nd Line Fix boxes, do callback for problems 1st line techs can't solve. 3rd Line Staff RT Basic concepts Ticket												15 mins		25 Purpose Heart of all TS departments. Enables issues to be tracked over time and people. Make a New Ticket								Verbal Instruction Click on new ticket/http://data.fglan/work_orders/new?mode=ts Subject Format - Name, Summary Important Fields Phone No Box Source, type, ticket type. What to write and when Everything, as soon as it happens, if not before Owning a ticket Important in the outside world Who is working on this Need to take/unown tickets Escalating 1->2->3				When? When you don't have the answer When to call Everyday until you speak to someone in person Phone messages get three strikes Conventions and abbreviations Left VM -- Date CNR Avoid Jargon Comments vs Replies									2.5 mins Comments are internal Replies go to user Sopmetimes need to change requestor Be careful with subject lines Workflow											2.5 mins   27.5 New -> Open (->contact)->pending->resolved Status												5 mins		32.5 New untouched be careful not to change status if you add notes Open actively being worked on			Contact custom status ready for pickup Stalled Pending boxes waiting to be picked up				atypical (normally more like contact/stalled			Resolved		EXERCISE: 			Create Ticket				Students to create 1 ticket in Sandbox Queue					My name is...			Comment on ticket				Comment on ticket of person to left			Resolved ticket of person to right		Custom Fields etc									2.5 mins	40			Basic				name etc				Status				fields specific to Free Geek			Links				refers to/by				merge					can not undo				depends on				parent/child			People				requestor				cc				owner		Dashboards											2.5 mins			Line 1				lists phone calls and messages, boxes ready for pickup			Line 2				Boxes on the bench, new and open					Always work in chronological order						follow list				Line 3				For staff Working on the command line	Commands												5 mins		45		pwd, ls			Commands are like work in a sentence			simple				verbs					Fetch! list!						e.g. pwd, ls, cd, rm, mv			EXERCISE: Open Terminal type whoami, pwd read back and interpret results basic arguments nouns fetch this box, list this directory e.g ls /home Options									5 mins		50 adverbs, Fetch quickly, list fully e.g. ls -alh - and -- short and long (gnu) form common options -h, --help -v, --verbose -R, recursive differ between commands can be combined with noun form -o [file] EXERCISE: type ls, ls /home, ls ~/, ls -a,  ls -al, ls -alh, ls --almost-all

Common commands											20 mins		1hr 10 man free df df -h du du -sh du -ch --max-depth=1 EXERCISE: Use df and du to determine total size of hard drive and size of usr directory ps ps aux kill, killall, pkill, pgrep top uses ncurses half way to gui EXERCISE: Use ps and top to idntify processes running under your user mount mount (-t option) device mountpoint ssh used for connecting to remote machine securely log into a box anywhere ssh me@there scp secure copy scp here me@there: note colon otherwise works as cp			sftp secure ftp less, view etc head head file head -10 file tail tail -f pager used to view files less is more go backwards and forwards, search view vi in readonly mode, usefull for syntax and commands editors												10 mins		1hr 20 nano easy(ish) only standard on Debian vi on nearly every *nix box vi(m) -- vi (i)m(proved) steep learning curve but very powerful worth the effort EXERCISE:											05 min		1hr 25 open vim: vim hello-world write hello world i... duplicate line yy... ESC..p					delete world dw write and quit (colon)wq

TS Tools ts_network_backup									15 mins		1hr 40 basic backup ts_network_backup -c [ticket no] does everything for you advanced backups backing up when a drive is attached to another box -p option alternative path to home command line options -a other things to back up					-u, -d backup less restoring backups ts_network_backup -r [folder] EXERCISE: create command line for backup using one option, say what you would use to restore it. ts_identify											5 mins		1hr 45 identifies backup folder ts_identify_backups (-t) ticket number others do what they say on the tin, no options except -h File system basics											10 mins		1hr 55 /	/bin etc bin, sbin,opt, root, tmp, lib var cache, mail, www where servers store things usr for users repeats hierarchy also doc share src /etc config files /etc/apt filesystem /etc/fstab & /etc/mtab network /etc/host /etc/resolv.conf users and groups passwd, shadow, group, gshadow /var/log where log files are stored\ whats going on	/home where you live users files /proc & /sys not really files contain system info cat /proc/cpuinfo /mnt & /media mount discs Manual Backups												10 mins		2hr 05 Go over page, backup config backup users and groups get list of installed packages transfer data rsync -avzh here me@there:/var/tsbackup backup name ticket -iso date iso date why its important

Rooting a box												15 mins		2hr 20 why password security if you have physical possesion of box there is no security recovery mode press shift when after bios screen 1st menu recovery mode 2nd menu rootshell not networking remounting makes file system writable mount -o rw,remount / starting networking dhclient eth0 etc network booting the network boot menu change bios settings, go			tech support -> debian rescue why use a recovery disk specialist tools efficient mounting a hard drive mount (-t) drive mountpiint chroot'ing chroot /mnt for i in dev proc sys dev/pts; do mount $i /mnt/$i; done EXERCISE: In pairs,Root box and netboot to debian rescue Diagnosing problems through log files and error message running in the terminal									5 mins		2hr 25 verbose mode -v --verbose multiple levels debug modes log levels warn, info, error dmesg													5 mins		2hr 30 /var/log/syslog essential info what the system does EXERCISE: tail -f /var/log/syslog I/O Redirection											5 mins		2hr 35 pipe to command | 		pipe to file < > >>			stdout & stderr 1 > /dev/null redirect std out (makes it disappear				2 > error.file					redirect stderr				&> log					both to log		EXERCISE: 			with ls send stdout and stderr to /dev/null and observe results	grep													10 mins		2hr 45		search tool		global regular expression parser			'Some people, when confronted with a problem, think “I know, I'll use regular expressions.”  Now they have two problems.'			no need to use regex for most things 				a word is a regex that says match this word	awk (and sed)		sed was the cause of the regex quote			can be very useful but unlikely you will need to use it			command line search and replace		awk was the source of the quote			whole programming language for dealing with tabular data				you only need to know tow things					awk '{print $1}'					awk '{print NF}'			cat /var/log/apache2/access.log | awk '{print $2}' | sort | uniq -c |sort -g		EXERCISE: use grep and awk to get time and date of kernel events using awk (as group on board, use questions and prompts)

How to troubleshoot											15 mins		3hr Importance of consistency Not your machine Do not break it		Repeatable solutions Needs to diagnose problem not just fix it			reinstall is not the answer Problem solving https://en.wikipedia.org/wiki/Problem_solving Read this page(write on board) OODA loop (Observe, Orient, Decide, Act) https://en.wikipedia.org/wiki/OODA_loop Comes from military/fighter pilots designed for stressful situations and quick reactions important part it is a loop constant feedback ability to change course Observe what is going on					Orient where might the problem lie Decide how can we test this Act run test Observe what where the results Orient what did they tell us					Decide whats the solution to fix the problem Act apply solution Observe did it work? etc... PDCA (Plan Do Check Act) https://en.wikipedia.org/wiki/PDCA Plan Study the problem, establish the objectives, devise test Do Carry out test, collect data where there any error meessages? what did the log files say? Check Study results, compare to what was expected Act Analyse differences, determine root causes, corrective actions, next steps repeat if necessary RPR (Rapid Problem Resolution) https://en.wikipedia.org/wiki/RPR_Problem_Diagnosis Discover, Investigate, Fix IT Specific Discover Gather and review information Build model Investigate Create and carry out plan to gather data analyse results and iterate Identify root cause Fix Determine and implement fix check root cause worked Example The Facebook isn't working' Discover is it facebook or the internet Investigate ping facebook No -- iterate ping google No -- iterate ping router No -- the internet is down Fix reboot router open browser visit facebook EXERCISE: A box comes in that the user reports is not booting In pairs, Chose one method and outline the steps to solve the problem

Total Time 3 hours, not incrluding a break.