PATH is a special environment variable. The "full path" to a file represents its exact location on the filesystem, such as /usr/bin/bash. Any time you type the name of a program without a full path, bash searches your PATH to find it. Here's a simple example of PATH:
The PATH is searched from left to right, with : as a separator. So when I type python in my terminal, bash will look in the following places (in order) to find it:
/home/user/pythons/27/bin /usr/local/bin /usr/bin /bin
You can also define relative PATH pathes, like:
Modifying your PATH
The safest way to add something to your path is to append it. Note that you do not name the executable itself in the path, just the folder that it sits in.