XPath is very powerful, it’s kind of a query language like SQL but for XML documents, including HTML documents, so it is a natural fit for web scraping. A lot of people never learn it and use CSS selectors and regex, and have a really hard time scraping anything complicated.
Regex is great for parsing text, but not XML/HTML – see this famous StackOverflow answer about it:
CSS selectors are useful but not quite as powerful as XPath.
A simple example of XPath looks like this:
//div[contains(@class, "price")]//text() This means “look for any div whose
class attribute contains ‘price’, and give me the text inside each of those divs”. The result might be something like [“$39”,”$42”] if our divs are
<div class='price foo'>$34</div> ... blah blah ... <div class='price spam'>$42</div>.
For testing out and tweaking your XPath queries, the best tool I know is this Firefox plugin: https://addons.mozilla.org/en-US/firefox/addon/xpath-checker/ – it gives you a pretty-looking CSS-styled view of what a given XPath query returns. It’s much easier than looking at the raw HTML. I don’t know of any good extensions in Chrome.
I learned from http://www.w3schools.com/xsl/xpath_intro.asp but I found that site really hard to follow for anything but the basics.
TODO: A REAL XPATH TUTORIAL HERE
Check out this incredibly handy cheat sheet: http://ricostacruz.com/cheatsheets/xpath.html