ADDITIONAL RESOURCES

SWC Python lessons online: http://swcarpentry.github.io/python-novice-inflammation/
SWC Shell lessons online: http://swcarpentry.github.io/shell-novice/
SWC Git lessons online: http://swcarpentry.github.io/git-novice/

David's Python exercise repository: https://github.com/dotsdl/swc_intermediate_python
David's summary Jupyter Notebook: http://nbviewer.jupyter.org/github/dotsdl/swc_intermediate_python/blob/master/intermediate_scientific_python.ipynb

-------------------------------

Welcome to Software Carpentry

This is the pad for the 2016-04-29 Software Carpentry Workshop at UCSF.
The website for the workshop can be found at <https://bsmith89.github.io/2016-04-29-ucsf>.

We will use this Etherpad to share links and snippets of code, take notes, ask and answer questions, and whatever else comes to mind.
The page displays a screen with three major parts:

* The left side holds today's notes: please edit these as we go along.
* The top right side shows the names of users who are logged in: please add your name and pick the color that best reflects your mood and personality.
* The bottom right is a real time chat window for asking questions of the instructor and your fellow learners.

To start, please add yourself to the attendee list below:

(discipline, institution)
(* denotes an instructor)

-     *Kevin Vilbig (Social Work, University of Texas at Arlington)
-     *David Dotson (Computational Biophysics, Arizona State University)
-     Alex Williams (Gladstone / UCSF, SF)
-     Lia Ball (Biophysics, UCSF)
- Peter Shin (Radiology, UCSF)

Users are expected to follow our code of conduct: http://software-carpentry.org/conduct.html
All content is publicly available under the Creative Commons Attribution License: https://creativecommons.org/licenses/by/4.0/

Sticky notes:
    Red means "I could use some help" or "I'm still working"
    Green means: "All good here"

Open a terminal, and type the following:

Unix shell

In your shell, you can run the line:

export PS1='$ '

this will set your prompt to a simple '$ '.

Bread and butter

cd <dirname> : change the present working directory to <dirname>
ls : list the contents of the current working directory
pwd : get the absolute path to the current working directory
mv <filename> <newname> : move a file or change its name (roughly the same thing)
rm <filename> : delete a file; NO UNDO, so be careful
rm -r <directory> : recursively delete a directory and everything inside; BE CAREFUL, NO UNDO

Working with files

cat <filename> : output contents of file to STDOUT
less <filename> : view contents of file with ability to scroll; exit with `q`
touch <filename> : make empty file, or if exists change access time of file
mkdir <directory> : make a new directory
rmdir <directory> : delete a directory, but only if it's empty
nano <filename> : open a file with the `nano` text editor

Working with text

sort <filename> : sort STDIN input and return to STDOUT

example: `cat myfile.txt | sort` will return sorted lines from 'myfile.txt'

wc <filename> : get linecount, wordcount, character count for a given file; can take input text from STDIN
uniq <filename> : get unique lines from file; relies on identical lines being adjacent (pre-sorted)

Piping and redirection

echo "some text" > file.txt

Use ">" to write the STDOUT of the commands on the left to a file. THIS WILL OVERWRITE THE FILE CONTENTS

echo "some text" >> file.txt

Use ">" to write the STDOUT of the commands on the left to a file. This will APPEND to the file.

echo "some text" | wc -l

Use "|" to pipe STDOUT of left-hand side into STDIN of right-hand side.

Looping

Example form:

for item in <thing1> <thing2>
do

echo $item

done

This loop will print <thing1>, then <thing2>. A loop can have do any number and variety of things in the loop body.

Creating a file in vim:
vim <filename>

If you get stuck in vim, type:
<ESC> :q

to get out of it.

Exercises links for shell pipe filter
http://swcarpentry.github.io/shell-novice/03-pipefilter.html
http://swcarpentry.github.io/shell-novice/04-loop.html

To cancel a running operation in the shell, hit:

Ctrl+C

http://swcarpentry.github.io/shell-novice/06-find.html

The Advanced Bash Scripting Guide: This starts simple, and gets deeper.
http://www.tldp.org/LDP/abs/html/
a
http://github.com/dotsdl/swc_ucsf_python/blob/master/Mosquito%20Analysis.ipynb
My "live" notebook:

To make one grid of plots as one person asked:

    fig = plt.figure(figsize = (24,6))
filelist = glob.glob('*.csv')
count = 0
for file in filelist:
    count += 1
    dataf = pd.read_csv(file)
    ax1 = fig.add_subplot(2, len(filelist), count)
    ax1.plot(dataf['temperature'], dataf['mosquitos'], 'ro')
    ax1.set_ylabel('Number of Mosquitos')
    ax1.set_xlabel('Temperature (F)')
    ax1.set_title(file)
    ax2 = fig.add_subplot(2, len(filelist), count+len(filelist))
    ax2.plot(dataf['rainfall'], dataf['mosquitos'], 'bs')
    ax2.set_ylabel('Number of Mosquitos')
    ax2.set_xlabel('Rainfall')



I was playing with how to load all of the files into a single data frame. This will not entirely make sense, because it uses some things we haven't gone over yet, but it's pretty cool.

import pandas as pd
import matplotlib.pyplot as plt
import glob

for each in glob.glob('*.csv'):
    try:
        data
    except NameError:
        data = pd.DataFrame(pd.read_csv(each))
    else:
        data = data.append(pd.read_csv(each), ignore_index = True)

data.plot.scatter(x = ['rainfall'], y=['mosquitos'])

plt.savefig('foo.png')

Version control with git

Shell history: https://www.dropbox.com/s/0c6bzp5xakjochy/History.txt?dl=0

setup
git config --global user.name "Your Name"
git config --global user.email "your@email.com"

git config --global color.ui "auto"
git config --global core.editor "nano -w"

...if you need a proxy server for remote work
$ git config --global http.proxy proxy-url $ git config --global https.proxy proxy-url
$ git config --global --unset http.proxy $ git config --global --unset https.proxy
Making a new repository
To make the current working directory a git repository, do

git init

Telling git what to include in the next commit
Use

git add <filename>

to tell git what files with changes to include in the next commit.

Making a commit
Use

git commit -m <commit message>

to make the commit from the changed files included from `git add` previously.

Finding out the state of the git repository
We can use

git status

to ask git what the status is of the staging area (`git add`-ed files), the current branch name (usually 'master'), and which commit the current branch points to

What changed?
We can find out what differences there are between the current files and the last commit (or staged files) with:
git diff

Viewing the history
We can have a look at the previous commits we have made with

git log

This will show you the unique identifiers for each commit in this branch (probably 'master'), in reverse chronological order (recent top). It will also show you the commit message for each one (which you added with `git commit -m <message>` when you made the commit, summarizing the change).

Going back in time
We can view our files as they were in a previous commit with

git checkout <previous commit unique identifier>

We can also go to commits relative the one HEAD currently points to withwith, e.g.:

git checkout HEAD~1

which will move HEAD to point to the previous commit. This will put you in a 'detached HEAD state', because HEAD doesn't currently point to a branch name, but instead a specific commit. To go back to the 'master' branch

Getting a previous version of a single file from an existing commit can be done with:
git checkout <commit identifier or HEAD~#> <filename>.

Defining a remote repository to push/pull from
If we make a repository on a repository hosting service like GitHub, we can add it as a remote to our local repo:

git remote add <remotename> <remoteurl>

Any name (no spaces) can be used for the remote, but typical ones are "origin".

Pushing to a remote repository
We can push commits we've made locally to a remote repository with:
    git push <remotename> <branchname>

If commits exist on the remote branch that don't exist locally, you'll need to pull first.

Pulling from a remote repository
We can pull commits from a remote repository to our local branch with:
    git pull <remotename> <branchname>

If there are changes from the remote commits that git cannot merge automatically, it will tell you there are MERGE CONFLICTS. Use `git status` to see which files have merge conflicts, open them in your favorite text editor, and make the changes to the marked regions that resolve the issues.

http://figshare.com/articles/How_Git_works_a_cartoon/1328266

Create a new Git repository on your computer called bio. Write a three-line biography for yourself in a file called me.txt, commit your changes, then modify one line, add a fourth line, and display the differences between its updated state and its original state.

Make an account on GitHub; make a new repository called `planets`.

https://guides.github.com/activities/citable-code/

Morin, A., Urban, J., and Sliz, P. “A Quick Guide to Software Licensing for the Scientist-Programmer” PLoS Computational Biology 8(7) (2012): e1002598.