Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Why I Use an Anonymous Web Proxy Service

I've been using an anonymous web proxy service for some time now and I don't think I would ever go back to ordinary surfing.    It's not that I am some sort of shadowy cyber criminal with some secret online life, no I pretty much surf the same web sites as everyone else, I certainly don't go to illegal or criminal sites.

If you want to know why, it's simply because I value my privacy and I know what data is stored in the average organisation and how it is protected.   Take your ISP for example, they logs pretty much everything that passes through them - every single request, every web page, every email and even every image is logged.   The vast majority of this traffic is in cleartext so the majority of it is instantly readable and can be matched to your address. 

It's amazing what these logs can tell about people you can build up a lot of information about someone from what they do on the web.  So do you know who has access to these logs at your ISP, what controls are in place, how is this information stored ?

If you don't know your not alone but after 25 years working in IT, I'll pretty much bet you wouldn't like to know the answer anyway.  This is the problem with much internet communication, it's so completely open - occasionally you'll obviously use a secure site to type in your credit card details.   Yet up to the point all your information is flying across the internet ether completely unprotected and accessible to anyone who has the will and the knowledge to intercept it. 

Believe me there are plenty of people who do have an interest in your information

Why do you think a European Directive was recently passed - Directive on Mandatory Retention of Communications Traffic Data.   It's  a bit of a mouthful but what it's  actually doing  is forcing your ISP to keep a record of every email sent, every internet session and web site visited for two years.   So think back at every web site you've ever visited for the last two years and it's on record and can be matched to your own little electronic profile.

How does that make you feel?

Even if you've done nothing wrong it's bound to make you feel a little uneasy after all why do Governments want all this information about us?   If you want to feel a little more uneasy, many Governments are looking at the UK Governments idea who are going to store all this information in a central database accessible to certain organisations.     Fear not US citizens you'll not be left out - the FBI are pushing hard for similar data retention facilities.

Storing data is dangerous

If you store data you have to be prepared to take good care of it.   All the privacy erosion concerns aside, the more personal data is stored about us, the more people will have access to it.  How thorough do you think is the vetting procedure for an ISP Technical engineer ?  Who potentially has access to huge amounts of this data.    Even if implicitly trust our govenrments with this tremendous invasion of our privacy - fraudsters, identity thieves and others can create mayhem with this sort of information. 

These logs are gold dust to anyone trying to steal our identity for example - they can match up users, with web sites, pick up password and all sorts of personal information.  Even  though a small part of this information is encrypted when you're actually using a secure web site (with the padlock below) there's plenty of other personal information available to compromise your usernames, password etc.

An identity thief can pick up huge amounts of data about the average person all ready, an hour or two picking up all your web traffic and they'll also have where you bank, what web sites you visit and logon to and a whole heap more.  This unfortunately is the tip of the iceberg and I suspect you'll see many, many people protecting  their privacy online soon from identity thieves, hackers and even our friendly snooping governments.

Everything I do online is private, I use a fast professional service that anonymises my connection and encrypts all the data.  My web logs sit next to yours in a server room at my ISP however mine are all completely encrypted and unreadable by anyone, whereas yours are in clear text.

One thing I should warn people about is the use of free anonymous web proxy services which you see all over the web.   Be very, very careful about these and think why are they supplying a very expensive and resource hungry service for free?   Remember also that by using one of these free proxies you are redirecting your traffic via this server whose owner can capture and log your data himself.   Many of these are owned and run by Eastern European Hacking groups as an easy way to gain peoples traffic, so be careful.

Source:http://ezinearticles.com/?Why-I-Use-an-Anonymous-Web-Proxy-Service&id=1679289

Using Bulk SMS Services how the stock advisory companies increasing their sales volume?

BNI's philosophy is "Givers Gain" - members are expected to focus on giving referrals to other members to build relationships and receive referrals in return. Why I made this point here is the BNI is one of the World’s largest referral organization which works via referral networks. Even Stock Advisory firms are also working on the similar kind of model. What they do? They send the list of their targeted audience or prospective customers stating that i.e.) Buy “ABC Ltd Share” BSE Code: XXXX @ Rs. 142-145 for 1 Month Target 380-460 Huge Govt. Orders received buy 2000 shared for 2Lacs Profit in 1 Month. 10 Days Free Trail also Available.

One More Sample – How much did you earn in Stock Market?? Today our client earned 20,500 INR. To join and earn daily in Stock Market.

When the person read the above messages they are tempted to add few more savings to their bank accounts so they show much interest what’s is this and how to invest in a smaller way etc… so the stock advisory firms to ask them to sign up for 10 days free trail where they will get lot of messages from the stock advisory companies like what’s the Buy && Sell and Profit, etc… Especially every day EOD, you will get the report like what’s the profit made by the major customer, what’s the investment, profit, etc… You will also be amazing! With the small amount of money, how the customers are earning more money. You will be showing much interest to have few more earnings. You are start welcoming lot of messages like this i.e.)

Note: Of course the stock advisory company whatever gives you the statement like X is making so much of profit that’s 100% correct. There is one thing you forget to realize, the X is spending his dedicated day ONLY for STOCK’s business. NOT as Part time he is earning HUGE PROFIT.

Here it’s NOT the question of earning the profit or loss. It’s one of the Marketing Strategy how the Stock Advisory Companies are converting the business very nicely with their targeted customer. Most of the time, they are ONLY using Bulk SMS Services as Marketing Strategy to promote their business. They do use Promotional & Transactional Bulk SMS Services. To convert anyone into business they do periodic Promotional Bulk SMS to them regarding the service and daily gain by different customer. Once if the customer is signed for FREE TRAIL then start using Transactional Bulk SMS Services to send the market updates, market call, staring, mid day, closing status, exit rate, etc…

Source:http://blogs.siliconindia.com/bulksmscompany/Business/Using-Bulk-SMS-Services-how-the-stock-advisory-companies-increasing-their-sales-volume-bid-fP9gkbI282792976.html

Internet marking is the finest method of increment of popularity in commerce

In today’s world key to any business is right kind of marketing strategy that needs to be implemented for the success of any entrepreneurial project. One of the fastest growing ways is to get the listing of your company in top search engines. SEO is one of the ways which can help in increasing or diverting the traffic on any website. People should understand the search result of their website is displayed on the World Wide Web. Different search engines work in different ways by providing different ranking to websites according to the search of the people. This is why same website has different ranks on different search engines.

Basically, website owner provides the keywords to the SEO providers which are appropriate to their business. Business SEO providers make sure the owners are charged low and also providing correct information to search engines. Providing better SEO in Ottawa is the basic feature that service providers aim at, their main motive is to make these services available to all the common people so that they can benefit there business in a better way. A good SEO mostly uses popular sites like Google, Bing, Yahoo, Yellow pages etc.

These SEO companies make sure that the listing of their clients is upgraded by repeated updating and also there ranking reaches top on search engines. Consultations with SEO service provider can be availed for free by the website owners. SEO companies carefully chooses keyword for their clients because it has to be unique and commonly used to make them reach at the top .They do so by carefully analyzing the owners website and competitors website and then the strategy is made to determine the perfect keyword for their clients. Owner should make sure that the prices charged to him are competitive according to market.

Consultation and Analysis are important because the keyword formation is directly dependent on these two parameters if the SEO service provider fails to submit the proper keyword it would result in failure of whole SEO services for the owner. SEO provider can re create and also modify the keywords for any website. If anyone specifically looking to create a new SEO website for him then the Looking internet marketing company in ottawa specialists can help them in determining a new domain that could be related to the business of the owner. Secondly, good hosting service is provided by them which help a website fast to access by any user. The biggest task for Internet marketing specialist is Link building because it represents more than 80% result of SEO.

For any website owner good SEO link building can be good in this competitive world because it attracts the people. A good SEO developer plays a pivotal role in developing a perfect keyword for the company. Ottawa SEO companies are very famous to deliver their services to the customers at very reasonable prices. SEO services are available online one can visit their websites see the list of services offered by them, read the testimonials of the existing and choose from them pay online and get started.

For further detail about Business seo service in Ottawa please visit the website.

Source:http://blogs.siliconindia.com/businessseo/Business/Internet-marking-is-the-finest-method-of-increment-of-popularity-in-commerce-bid-cf9206bx84461691.html

Collecting Data With Web Scrapers

There is a large amount of data available only through websites. However, as many people have found out, trying to copy data into a usable database or spreadsheet directly out of a website can be a tiring process. Data entry from internet sources can quickly become cost prohibitive as the required hours add up. Clearly, an automated method for collating information from HTML-based sites can offer huge management cost savings.

Web scrapers are programs that are able to aggregate information from the internet. They are capable of navigating the web, assessing the contents of a site, and then pulling data points and placing them into a structured, working database or spreadsheet. Many companies and services will use programs to web scrape, such as comparing prices, performing online research, or tracking changes to online content.

Let's take a look at how web scrapers can aid data collection and management for a variety of purposes.

Improving On Manual Entry Methods

Using a computer's copy and paste function or simply typing text from a site is extremely inefficient and costly. Web scrapers are able to navigate through a series of websites, make decisions on what is important data, and then copy the info into a structured database, spreadsheet, or other program. Software packages include the ability to record macros by having a user perform a routine once and then have the computer remember and automate those actions. Every user can effectively act as their own programmer to expand the capabilities to process websites. These applications can also interface with databases in order to automatically manage information as it is pulled from a website.

Aggregating Information

There are a number of instances where material stored in websites can be manipulated and stored. For example, a clothing company that is looking to bring their line of apparel to retailers can go online for the contact information of retailers in their area and then present that information to sales personnel to generate leads. Many businesses can perform market research on prices and product availability by analyzing online catalogues.

Data Management

Managing figures and numbers is best done through spreadsheets and databases; however, information on a website formatted with HTML is not readily accessible for such purposes. While websites are excellent for displaying facts and figures, they fall short when they need to be analyzed, sorted, or otherwise manipulated. Ultimately, web scrapers are able to take the output that is intended for display to a person and change it to numbers that can be used by a computer. Furthermore, by automating this process with software applications and macros, entry costs are severely reduced.

This type of data management is also effective at merging different information sources. If a company were to purchase research or statistical information, it could be scraped in order to format the information into a database. This is also highly effective at taking a legacy system's contents and incorporating them into today's systems.

Overall, a web scraper is a cost effective user tool for data manipulation and management.

Source:http://ezinearticles.com/?Collecting-Data-With-Web-Scrapers&id=4223877

Internet Marketing - A Beginners Guide

Every new website owner is faced with the problem of getting visitors to their site, and if the website happens to be a commercial venture it needs to happen fast, even non commercial sites need visitors to survive. A quick look at some of the online tools to find expired domains show just how many websites fail every hour of every day. To help prevent your site joining the daily list of failures effective marketing is essential.

Marketing does not necessarily mean spending lots of money, however if fast results are needed then some money will have to be spent. As this article is aimed at beginners, I'll only briefly look at paid marketing and concentrate mainly on the free or low cost options to promote your site.

Paid Marketing:

In many respects online marketing is not dissimilar to offline marketing and many of the tactics used to promote an offline business will work equally well for websites. For example newspaper or magazine adverts although sometimes expensive can produce excellent results for websites, and occasionally magazines will supply cover cd's with links back to your site, an excellent way to get more visitors.

For larger campaigns television or radio ads can work, and with the expansion of satellite TV stations costs for this type of campaign are coming down all the time. For most new site owners however these options will prove to be too expensive and specialist online advertising will be the preferred choice. In the online paid sector directory listings and Pay Per Click (PPC) are probably the most popular. PPC quite simply work as the name describes and you pay a set amount every time a potential customer clicks your link. This has the advantage you only pay when someone visits your site and the obvious disadvantagethat it's open the click fraud where others click the links to increase your bill.

Most of the providers of PPC offer protection against click fraud and block multiple clicks from the same IP address, although for the site owner this is difficult to monitor, and suspicion is always there that the PPC provider has little incentive to police their policies as it would lead to loss of revenue. From experience PPC advertising can become very expensive and although it's quite easy to set up yourself through companies like google, it can sometimes be worth paying a specialist to run your campaign, as they will know how best to target the clicks to make best use of your money.

Free Marketing:

In business like all walks of life the general rule is that you get nothing for nothing, with Internet marketing this rule somewhat goes out the window as there are many free resources for promoting your website. In this section I'm going to take a look at a few of the most popular, this is of coarse only scraping the surface and as you develop your strategy further you'll find new outlets and tools that will get lots of visitors popping by your website.

Search Engines:

Without question the most effective free source of visitors are the many search engines like Google, Yahoo and MSN... the list really is almost endless. Most new website owners believe to get listed in these popular search engines the best thing to do is submit your site to them, this is not the case. All the main search engines these days use crawlers which automatically browse the web and store the contents of the sites they visit.

The search engines then use this content (along with 100's of other criteria ) to rank sites for specific search terms. Search engine optimization (SEO) is a whole different subject, however in brief what you need is links from other website, and some of the marketing tips below, along with attracting visitors to your site will help with your SEO efforts. The truth is that many people use website marketing solely to increase ranking in search engines, in my experience however I've found that if you promote your site for real visitors the search engines will follow.

There are many SEO companies who specialise in getting your site up the rankings, you do need to be careful as many will make great and exaggerated claims, some of which are simply not possible. The best advice I can give here is to research thoroughly and look at, and if possible contact past customers for a reference. Good SEO can be the most cost effective way to promote your site but you do need to work hard at it or employ someone else to do it for you.

Forums:

Internet forums can be very useful for getting visitors to your site, the biggest advantage with forums is that you can target forums related to your area of interest and most will allow members who contribute useful posts and replies to have a link back to their own website in their signature. You should take care not to just post links as most good forums will consider this as spam, and even if the moderators don't delete the link, visitor looking at it will clearly see what's going on and your sites reputation will suffer.

Blogs:

In my opinion every website owner should have a blog, and for SEO purposes it's probably best to have the blog on a different domain. There are lots of free blog sites like blogger.com which offer you easy to use blogs. The biggest advantage with your own blog is you can write articles and provide links to different areas of your site, this provides different entry points and is also very good SEO practice.

Another use of blogs is comments, a great way of getting visitors back to your site is to search for other blogs relevant to your website and leave comments with a link back to your site. As with forums care should be taken not to spam the comments as it's a bad practice and unlikely to help you long term.

Article Submissions:

If you're up to it, writing an article can be a good way to get links back to your website, most good article distribution sites like EzineArticles.com will allow you to have a short bio at the end of the article, which can direct visitors back to your websites. If possible the article should be about a subject related to your business, but as you can see from my bio below it's not essential :-)

Well that's about it, as I said at the beginning this is only an introduction for beginners and as your site and business grows you'll find many new and exciting ways to market your online business.

Source: http://ezinearticles.com/?Internet-Marketing---A-Beginners-Guide&id=1729653

Sample Essay Writing - What Must Be Considered

You may be looking at it or you may be required to write on it.

This is a particular type of writing essay that is often put for view by online research and writing services. As a student, you should not only consider looking at classification essay, you should also consider writing a sample essay that can be viewed as a sample paper by other students.

In most cases, students will turn to these essays because of the factor of time. Most students will keep research and writing till the last minutes before actually beginning the write-up. One of the most important aspects in any academic writing is the issue of time. In everything you do in academia, it is always necessary to make use of an outline. The outline guides you as you write from start to finish. The outline is there to ensure that you start and finish on time. Writing without an outline is what makes you get caught up by deadlines.

When an online research and writing service offers a sample paper for view, it is asking you to consider its services in writing your essay. There is nothing wrong if you rely on it, but make sure you consider the issue of plagiarism seriously. Your essay is supposed to be something ingenious. Fortunately, there are anti-plagiarism tools over that internet that you can use to check for the authenticity of what has been written for you and to also check the references linked to your reflective essay.

Source: http://ezinearticles.com/?Sample-Essay-Writing---What-Must-Be-Considered&id=1194732