Python: Regular Expressions Part Two

Previously we have discussed about Introduction, Simple Metacharacters and Character classes. Checkout part one to get better understandings.  Today we will discuss about More MetaCharacters, Groups, Special Sequences and Email Extraction.

Metacharacters

Some more metacharacters are *, +, ?, { and }.
These specify numbers of repetitions.
The metacharacter * means “zero or more repetitions of the previous thing”. It tries to match as many repetitions as possible. The “previous thing” can be a single character, a class, or a group of characters in parentheses.
Example:

import re
pattern = r"egg(spam)*"
if re.match(pattern, "egg"):
   print("Match 1")
if re.match(pattern, "eggspamspamegg"):
   print("Match 2")
if re.match(pattern, "spam"):
   print("Match 3"

The example above matches strings that start with “egg” and follow with zero or more “spam”s.

The metacharacter + is very similar to *, except it means “one or more repetitions”, as opposed to “zero or more repetitions”.
Example:

import re
pattern = r"g+"
if re.match(pattern, "g"):
   print("Match 1")
if re.match(pattern, "gggggggggggggg"):
   print("Match 2")
if re.match(pattern, "abc"):
   print("Match 3")

To summarize:
* matches 0 or more occurrences of the preceding expression.
+ matches 1 or more occurrence of the preceding expression.

The metacharacter ? means “zero or one repetitions”.
Example:

import re
pattern = r"ice(-)?cream"
if re.match(pattern, "ice-cream"):
   print("Match 1")
if re.match(pattern, "icecream"):
   print("Match 2")
if re.match(pattern, "sausages"):
   print("Match 3")
if re.match(pattern, "ice--ice"):
   print("Match 4")

Curly Braces

Curly braces can be used to represent the number of repetitions between two numbers.
The regex {x,y} means “between x and y repetitions of something”.
Hence {0,1} is the same thing as ?.
If the first number is missing, it is taken to be zero. If the second number is missing, it is taken to be infinity.
Example:

import re
pattern = r"9{1,3}$"
if re.match(pattern, "9"):
   print("Match 1")
if re.match(pattern, "999"):
   print("Match 2")
if re.match(pattern, "9999"):
   print("Match 3")

“9{1,3}$” matches string that have 1 to 3 nines.

Groups

A group can be created by surrounding part of a regular expression with parentheses.
This means that a group can be given as an argument to metacharacters such as * and ?.
Example:

import re
pattern = r"egg(spam)*"
if re.match(pattern, "egg"):
   print("Match 1")
if re.match(pattern, "eggspamspamspamegg"):
   print("Match 2")
if re.match(pattern, "spam"):
   print("Match 3")

(spam) represents a group in the example pattern shown above.

The content of groups in a match can be accessed using the group function.
A call of group(0) or group() returns the whole match.
A call of group(n), where n is greater than 0, returns the nth group from the left.
The method groups() returns all groups up from 1.
Example:

import re
pattern = r"a(bc)(de)(f(g)h)i"
match = re.match(pattern, "abcdefghijklmnop")
if match:
   print(match.group())
   print(match.group(0))
   print(match.group(1))
   print(match.group(2))
   print(match.groups())

As you can see from the example above, groups can be nested.

There are several kinds of special groups.
Two useful ones are named groups and non-capturing groups.
Named groups have the format (?P<name>…), where name is the name of the group, and is the content. They behave exactly the same as normal groups, except they can be accessed bygroup(name) in addition to its number.
Non-capturing groups have the format (?:…). They are not accessible by the group method, so they can be added to an existing regular expression without breaking the numbering.
Example:

import re
pattern = r"(?P<first>abc)(?:def)(ghi)"
match = re.match(pattern, "abcdefghi")
if match:
   print(match.group("first"))
   print(match.groups())

Metacharacters

Another important metacharacter is |.
This means “or”, so red|blue matches either “red” or “blue”.
Example:

import re
pattern = r"gr(a|e)y"
match = re.match(pattern, "gray")
if match:
   print ("Match 1")
match = re.match(pattern, "grey")
if match:
   print ("Match 2")    
match = re.match(pattern, "griy")
if match:
    print ("Match 3")

Special Sequences

There are various special sequences you can use in regular expressions. They are written as a backslash followed by another character.
One useful special sequence is a backslash and a number between 1 and 99, e.g., \1 or \17. This matches the expression of the group of that number.
Example:

import re
pattern = r"(.+) \1"
match = re.match(pattern, "word word")
if match:
   print ("Match 1")
match = re.match(pattern, "?! ?!")
if match:
   print ("Match 2")
match = re.match(pattern, "abc cde")
if match:
   print ("Match 3")

Note, that “(.+) \1” is not the same as “(.+) (.+)”, because \1 refers to the first group’s subexpression, which is the matched expression itself, and not the regex pattern.

More useful special sequences are \d, \s, and \w.
These match digits, whitespace, and word characters respectively.
In ASCII mode they are equivalent to [0-9], [ \t\n\r\f\v], and [a-zA-Z0-9_].
In Unicode mode they match certain other characters, as well. For instance, \w matches letters with accents.
Versions of these special sequences with upper case letters – \D, \S, and \W – mean the opposite to the lower-case versions. For instance, \D matches anything that isn’t a digit.
Example:

import re
pattern = r"(\D+\d)"
match = re.match(pattern, "Hi 999!")
if match:
   print("Match 1")
match = re.match(pattern, "1, 23, 456!")
if match:
   print("Match 2")
match = re.match(pattern, " ! $?")
if match:
    print("Match 3")

(\D+\d) matches one or more non-digits followed by a digit.

Additional special sequences are \A, \Z, and \b.
The sequences \A and \Z match the beginning and end of a string, respectively.
The sequence \b matches the empty string between \w and \W characters, or \w characters and the beginning or end of the string. Informally, it represents the boundary between words.
The sequence \B matches the empty string anywhere else.
Example:

import re
pattern = r"\b(cat)\b"
match = re.search(pattern, "The cat sat!")
if match:
   print ("Match 1")
match = re.search(pattern, "We s>cat<tered?")
if match:
   print ("Match 2")
match = re.search(pattern, "We scattered.")
if match:
   print ("Match 3")

\b(cat)\b” basically matches the word “cat” surrounded by word boundaries.

Email Extraction

To demonstrate a sample usage of regular expressions, lets create a program to extract email addresses from a string.
Suppose we have a text that contains an email address:

str = “Please contact info@test.com for assistance”

Our goal is to extract the substring “info@test.com”.
A basic email address consists of a word and may include dots or dashes. This is followed by the @ sign and the domain name (the name, a dot, and the domain name suffix).
This is the basis for building our regular expression.

pattern = r”([\w\.-]+)@([\w\.-]+)(\.[\w\.]+)”

[\w\.-]+ matches one or more word character, dot or dash.
The regex above says that the string should contain a word (with dots and dashes allowed), followed by the @ sign, then another similar word, then a dot and another word.

Our regex contains three groups:
1 – first part of the email address.
2 – domain name without the suffix.
3 – the domain suffix.

Putting it all together:

import re
pattern = r"([\w\.-]+)@([\w\.-]+)(\.[\w\.]+)"
str = "Please contact info@test.com for assistance"
match = re.search(pattern, str)
if match:
   print(match.group())

In case the string contains multiple email addresses, we could use the re.findall method instead of re.search, to extract all email addresses.

The regex in this example is for demonstration purposes only.
A much more complex regex is required to fully validate an email address.

Courtesy: sololearn

Advertisements

Python: Regular Expressions Part One

Regular expressions are a powerful tool for various kinds of string manipulation.
They are a domain specific language (DSL) that is present as a library in most modern programming languages, not just Python.
They are useful for two main tasks:
– verifying that strings match a pattern (for instance, that a string has the format of an email address),
– performing substitutions in a string (such as changing all American spellings to British ones).

Domain specific languages are highly specialized mini programming languages.
Regular expressions are a popular example, and SQL (for database manipulation) is another.
Private domain-specific languages are often used for specific industrial purposes.

Regular expressions in Python can be accessed using the re module, which is part of the standard library.
After you’ve defined a regular expression, the re.match function can be used to determine whether it matches at the beginning of a string.
If it does, match returns an object representing the match, if not, it returns None.
To avoid any confusion while working with regular expressions, we would use raw strings asr”expression”.
Raw strings don’t escape anything, which makes use of regular expressions easier.

import re

pattern = r"spam"

if re.match(pattern, "spamspamspam"):
   print("Match")
else:
   print("No match")

The above example checks if the pattern “spam” matches the string and prints “Match” if it does.

Here the pattern is a simple word, but there are various characters, which would have special meaning when they are used in a regular expression.

Other functions to match patterns are re.search and re.findall.
The function re.search finds a match of a pattern anywhere in the string.
The function re.findall returns a list of all substrings that match a pattern.

Example:

import re

pattern = r"spam"

if re.match(pattern, "eggspamsausagespam"):
   print("Match")
else:
   print("No match")

if re.search(pattern, "eggspamsausagespam"):
   print("Match")
else:
   print("No match")
    
print(re.findall(pattern, "eggspamsausagespam"))

In the example above, the match function did not match the pattern, as it looks at the beginning of the string.
The search function found a match in the string.

The function re.finditer does the same thing as re.findall, except it returns an iterator, rather than a list.

The regex search returns an object with several methods that give details about it.
These methods include group which returns the string matched, start and end which return the start and ending positions of the match, and span which returns the start and end positions as a tuple.

import re
pattern = r"pam"
match = re.search(pattern, "eggspamsausage")
if match:
   print(match.group())
   print(match.start())
   print(match.end())
   print(match.span())

Search & Replace
One of the most important re methods that use regular expressions is sub.
Syntax:

re.sub(pattern, repl, string, max=0)

This method replaces all occurrences of the pattern in string with repl, substituting all occurrences, unless max provided. This method returns the modified string.
Example:

import re
str = "My name is David. Hi David."
pattern = r"David"
newstr = re.sub(pattern, "Amy", str)
print(newstr)

Metacharacters:

Metacharacters are what make regular expressions more powerful than normal string methods.
They allow you to create regular expressions to represent concepts like “one or more repetitions of a vowel”.

The existence of metacharacters poses a problem if you want to create a regular expression (orregex) that matches a literal metacharacter, such as “$”. You can do this by escaping the metacharacters by putting a backslash in front of them.
However, this can cause problems, since backslashes also have an escaping function in normal Python strings. This can mean putting three or four backslashes in a row to do all the escaping.

To avoid this, you can use a raw string, which is a normal string with an “r” in front of it. We saw usage of raw strings in the previous lesson.

The first metacharacter we will look at is . (dot). This matches any character, other than a new line.
Example:

import re
pattern = r"gr.y"
if re.match(pattern, "grey"):
   print("Match 1")
if re.match(pattern, "gray"):
   print("Match 2")
if re.match(pattern, "blue"):
   print("Match 3")

The next two metacharacters are ^ and $. These match the start and end of a string, respectively.
Example:

import re
pattern = r"^gr.y$"
if re.match(pattern, "grey"):
   print("Match 1")
if re.match(pattern, "gray"):
   print("Match 2")
if re.match(pattern, "stingray"):
   print("Match 3")

The pattern “^gr.y$” means that the string should start with gr, then follow with any character, except a newline, and end with y.

Character Classes:

Character classes provide a way to match only one of a specific set of characters.
A character class is created by putting the characters it matches inside square brackets.
Example:

import re
pattern = r"[aeiou]"
if re.search(pattern, "grey"):
   print("Match 1")
if re.search(pattern, "qwertyuiop"):
   print("Match 2")
if re.search(pattern, "rhythm myths"):
   print("Match 3")

The pattern [aeiou] in the search function matches all strings that contain any one of the characters defined.

Character classes can also match ranges of characters.
Some examples:
The class [a-z] matches any lowercase alphabetic character.
The class [G-P] matches any uppercase character from G to P.
The class [0-9] matches any digit.
Multiple ranges can be included in one class. For example, [A-Za-z] matches a letter of any case.

Example:

import re
pattern = r"[A-Z][A-Z][0-9]"
if re.search(pattern, "LS8"):
   print("Match 1")
if re.search(pattern, "E3"):
   print("Match 2")
if re.search(pattern, "1ab"):
   print("Match 3")

The pattern in the example above matches strings that contain two alphabetic uppercase letters followed by a digit.

Place a ^ at the start of a character class to invert it.
This causes it to match any character other than the ones included.
Other metacharacters such as $ and ., have no meaning within character classes.
The metacharacter ^ has no meaning unless it is the first character in a class.

Example:

import re
pattern = r"[^A-Z]"
if re.search(pattern, "this is all quiet"):
   print("Match 1")
if re.search(pattern, "AbCdEfG123"):
   print("Match 2")
if re.search(pattern, "THISISALLSHOUTING"):
   print("Match 3")

The pattern [^A-Z] excludes uppercase strings.
Note, that the ^ should be inside the brackets to invert the character class.

Courtesy: sololearn

Python’s magic method or special methods or dunder

Magic lamp from the story of Aladdin with Genie appearing in blue smoke concept for wishing, luck and magic
Magic lamp from the story of Aladdin with Genie appearing in blue smoke concept for wishing, luck and magic

Magic methods are special methods which have double underscores at the beginning and end of their names.
They are also known as dunders.
The only one we will encounter is __init__, but there are several others.
They are used to create functionality that can’t be represented as a normal method.

One common use of them is operator overloading.
This means defining operators for custom classes that allow operators such as + and * to be used on them.
An example magic method is __add__ for +.

class Vector2D:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def __add__(self, other):
        return Vector2D(self.x + other.x, self.y + other.y)

first = Vector2D(5, 7)
second = Vector2D(3, 9)
result = first + second
print(result.x)
print(result.y)

The __add__ method allows for the definition of a custom behavior for the + operator in our class.
As you can see, it adds the corresponding attributes of the objects and returns a new object, containing the result.
Once it’s defined, we can add two objects of the class together.

More magic methods for common operators:
__sub__ for –
__mul__ for *
__truediv__ for /
__floordiv__ for //
__mod__ for %
__pow__ for **
__and__ for &
__xor__ for ^
__or__ for |

The expression x + y is translated into x.__add__(y).
However, if x hasn’t implemented __add__, and x and y are of different types, then y.__radd__(x) is called.
There are equivalent r methods for all magic methods just mentioned.
Example:

class SpecialString:
  def __init__(self, cont):
    self.cont = cont

  def __truediv__(self, other):
    line = "=" * len(other.cont)
    return "\n".join([self.cont, line, other.cont])

spam = SpecialString("spam")
hello = SpecialString("Hello world!")
print(spam / hello)

In the example above, we defined the division operation for our class SpecialString.

 

There are several magic methods for making classes act like containers.
__len__ for len()
__getitem__ for indexing
__setitem__ for assigning to indexed values
__delitem__ for deleting indexed values
__iter__ for iteration over objects (e.g., in for loops)
__contains__ for in

There are many other magic methods that we won’t cover here, such as __call__ for calling objects as functions, and __int__, __str__, and the like, for converting objects to built-in types.
Example:

import random

class VagueList:
  def __init__(self, cont):
    self.cont = cont

  def __getitem__(self, index):
    return self.cont[index + random.randint(-1, 1)]

  def __len__(self):
    return random.randint(0, len(self.cont)*2)

vague_list = VagueList(["A", "B", "C", "D", "E"])
print(len(vague_list))
print(len(vague_list))
print(vague_list[2])
print(vague_list[2])

We have overridden the len() function for the class VagueList to return a random number.
The indexing function also returns a random item in a range from the list, based on the expression.

Courtesy: sololearn

Python – When to use list vs. tuple vs. dictionary vs. set (Theory)

img-20151124-wa0011

Data Structures

Python supports the following data structures: lists, dictionaries, tuples, sets.

When to use a dictionary:
– When you need a logical association between a key:value pair.
– When you need fast lookup for your data, based on a custom key.
– When your data is being constantly modified. Remember, dictionaries are mutable.

When to use the other types:
– Use lists if you have a collection of data that does not need random access. Try to choose lists when you need a simple, iterable collection that is modified frequently.
– Use a set if you need uniqueness for the elements.
– Use tuples when your data cannot change.

Many times, a tuple is used in combination with a dictionary, for example, a tuple might represent a key, because it’s immutable.

Hadoop – Stand out from the crowd!

We all know the History and Evolution of Hadoop. Now I will try to explain some key features of Hadoop that made Haddop to stand out from the crowd.

Scalability:

Let me talk about Hadoop Scalability first. Hadoop is linearly scalable. When I said Hadoop is linearly scalable , let me give an example to explain that.

Untitled
Lets say I have two cars, one is Black and another is Red. These two cars are giving 15 kilometers mileage for One liter of Octane. But I want 30 kilometers mileage with the same fuel. So in order to achieve that 30 KM , I have increased the configuration of these two cars. I have exactly doubled the configuration of those cars. But I found that the Black one did not achieve 30 KM, not even 20 KM 😦
From the example the Black car can be compared with RDBMS and Red one with Hadoop.

Affordability:

Hadoop is something which uses distributed file system which distributes the work among different file system. So rather than using a single file system we are using distributed file system and distributing the work among them. So in this case I am increasing my resources rather than having a single resource I am having multiple resources. So a questing might popup , is that not a problem with the budget? Increase of resources will increase the budget which can be a burden to my client.

The nodes in Hadoop clusters are made-up of commodity hardware.

What does Commodity Hardware mean?

Commodity hardware is a term for affordable devices that are generally compatible with other such devices. In a process called commodity computing or commodity cluster computing, these devices are often networked to provide more processing power when those who own them cannot afford to purchase more elaborate supercomputers, or want to maximize savings in IT design.

23

When I will compare enterprise hardware with commodity hardware , it will be around 90% cheaper 🙂

 

 

 

 

Reliability:

How can I trust on Hadoop which stores lots of confidential and critical data on a cheaper hardware’s ?

The answer is YES!!! You can fully rely on Hadoop because it can take over of Auto Fail over of nodes. Hadoop architecture takes care of Auto Fail-over of your nodes. Lets say I have spitted my works among 10 people if One person has fall and sick. What I should do then ? I will route ask someone to do that task.

Should I assign that to anyone ?

No, I need to identify who has less work load to handle the additional task.

Hadoop architecture also do the same thing.

Flexibility:

There are many reasons why Hadoop is flexible. Let me give two examples:
Firstly, as I said in the definition of Hadoop is a framework written in JAVA. And as we all know that, Java is the most powerful, portable, high ways across any operating system. So Hadoop should also be portable across any operating system.

The next thing is, Hadoop is written in Java but it’s not that all it’s programming models to be written in Java. You can write your programming model in Python, C, CPP or whatever you programming language you like. So Hadoop has lot of flexibility so that you can work your programming models in Hadoop.

 

Distributed And Fast:

And finally, let’s talk about the distributed behaviors of Hadoop. The distribution of work among different systems is the main feature of Hadoop and for which Hadoop got prominent. If you’re dealing with large volumes of unstructured data, Hadoop is able to efficiently process terabytes of data in just minutes, and petabytes in hours.

Hadoop_Basic_Arch_2
So, now we all know all significant features of Hadoop.

Happy learning 🙂

 

History and Evolution of Hadoop

Lets talk about the Evolution of Hadoop. Doug Cutting the creator of Hadoop(Yahoo!) and Chief Architect of Cloudera.
Doug CuttingIn the Year of 2002-2004, Doug Cutting was working with Apache in a project called Apache Lucene and Nutch, a distributed search engine that suppose to index 1 billion pages. Lucene is a search indexer and Nutch is a spider or crawler.

What does that mean ? What are the basic things of a general search engine?

A search engine basically contains of three things :

  • A spider or crawler : downloads data whenever you search something over the search engine.
  • Indexer : indexes to the frequently used pages. If the people are using any web site for more number of time. Indexer will point to that.
  • Mapper : maps actual content to the screen.

In December 2004, Google Labs published a paper on the MapReduce(also called MR) algorithm. Doug Cutting found that the project he is working on is not scaling according to expectation. Then he decided to use the concept of MR for building Nutch distributed file system.

In 2006, Doug Cutting had joined Yahoo! And Yahoo had provided some dedicated team to work on a Project called Hadoop!

Checkout the story behind the name :-).

100762110-hadoop.530x298           Source: Doug Cutting, Doug Cutting and Hadoop the elephant

During 2006 – 2008, Hadoop was born out of Nutch as a Large Scale Distributed Computing platform!Which would scale upto multiple number of machines.

By the end of year 2008, Yahoo declared it had 910 node clusters. And by using those it was able to sort One Terabyte of data  within 3.5 minutes. Previously it was taking at least a day to do that work.

So , we can say that Hadoop has got prominent by the Year 2008 !!!

Data Scientist and Data Engineer in the ideal world!

Though it is too early to differentiate between the two roles and responsibilities but still it is nice to have a little understanding of them. Most importantly, both of these roles are important in a well data science world!

Some how this is a common thing which many people get confused with. So in the ideal world Data Scientists are generally people who understand various statistical model and can find out how a problem can be solved using the data around. On the other hand, Data Engineers are the people who implement the ideas of the Data Scientist to create the technical architecture which would be a technical implementation of the solutions.

So now it would be clear that, skills required for Data Scientists are strong Mathematical knowledge and very good understanding of Statistical modeling with problem solving capabilities. Additionally a little skills of programming is also required to become an eligible member for this position.

On the contrary, skills expected from a Data Engineer would be a strong technical knowledge and programming skills and ability to formulate technical solutions. A little statistical knowledge would come in handy. Although in the real world there is a lot of overlap between the two roles. But what is to be understood is that, you do not grow from a Data Engineer to a Data Scientist or Data Scientist are more important. Data Scientist and Data Engineers have different roles and responsibilities and skill sets. So learning hadoop or any other similar tools and technologies doesn’t mean that you will be a Data Scientist but having a good exposure to other mathematical skills and knowledge would be a bigger strength in order to become a Data Scientist.

101.datascience.community presented the difference by using an excellent Venn Diagram.
data-scientist-vs-data-engineer

Difference

So when choosing career or hiring someone for this roles, please choose wisely and understand that they are different roles and responsibilities.