Data Scraping using php

Here is my code

    $ip=$_SERVER['REMOTE_ADDR'];

    $url=file_get_contents("http://whatismyipaddress.com/ip/$ip");

    preg_match_all('/<th>(.*?)<\/th><td>(.*?)<\/td>/s',$url,$output,PREG_SET_ORDER);

    $isp=$output[1][2];

    $city=$output[9][2];

    $state=$output[8][2];

    $zipcode=$output[12][2];

    $country=$output[7][2];

    ?>
    <body>
    <table align="center">
    <tr><td>ISP :</td><td><?php echo $isp;?></td></tr>
    <tr><td>City :</td><td><?php echo $city;?></td></tr>
    <tr><td>State :</td><td><?php echo $state;?></td></tr>
    <tr><td>Zipcode :</td><td><?php echo $zipcode;?></td></tr>
    <tr><td>Country :</td><td><?php echo $country;?></td></tr>
    </table>
    </body>

How do I find out the ISP provider of a person viewing a PHP page?

Is it possible to use PHP to track or reveal it?

Error: http://i.imgur.com/LGWI8.png

Curl Scrapping

<?php
$curl_handle=curl_init();
curl_setopt( $curl_handle, CURLOPT_FOLLOWLOCATION, true );
$url='http://www.whatismyipaddress.com/ip/132.123.23.23';
curl_setopt($curl_handle, CURLOPT_URL,$url);
curl_setopt($curl_handle, CURLOPT_HTTPHEADER, Array("User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.15) Gecko/20080623 Firefox/2.0.0.15") );
curl_setopt($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt($curl_handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl_handle, CURLOPT_USERAGENT, 'Your application name');
$query = curl_exec($curl_handle);

curl_close($curl_handle);
preg_match_all('/<th>(.*?)<\/th><td>(.*?)<\/td>/s',$url,$output,PREG_SET_ORDER);
echo $query;
$isp=$output[1][2];

$city=$output[9][2];

$state=$output[8][2];

$zipcode=$output[12][2];

$country=$output[7][2];
?>
<body>
<table align="center">
<tr><td>ISP :</td><td><?php echo $isp;?></td></tr>
<tr><td>City :</td><td><?php echo $city;?></td></tr>
<tr><td>State :</td><td><?php echo $state;?></td></tr>
<tr><td>Zipcode :</td><td><?php echo $zipcode;?></td></tr>
<tr><td>Country :</td><td><?php echo $country;?></td></tr>
</table>
</body>

Error: http://i.imgur.com/FJIq6.png

What's is wrong with my code here? Any alternative code , that i can use here.

I am not able to scrape that data as described here. http://i.imgur.com/FJIq6.png

P.S. Please post full code. It would be easier for me to understand.



Source: http://stackoverflow.com/questions/10461088/data-scraping-using-php

Php Scraping data from a website

I am very new to programming and need a little help with getting data from a website and passing it into my PHP script.

The website is http://www.birthdatabase.com/.

I would like to plug in a name (First and Last) and retrieve the result. I know you can query the site by passing the name in the URL, but I am having problems scraping the results.

http://www.birthdatabase.com/cgi-bin/query.pl?textfield=FIRST&textfield2=LAST&age=&affid=

I am using the file_get_contents($URL) function to get the page but need help after that. Specifically, I would like to scrape only the results from a certain state if there are multiple results for that name.



You need the awesome simple_html_dom class.

With this class you can query the webpage's DOM in a similar way to jQuery.

First include the class in your page, then get the page content with this snippet:

$html = file_get_html('http://www.birthdatabase.com/cgi-bin/query.pl?textfield=' . $first . '&textfield2=' . $last . '&age=&affid=');

Then you can use CSS selections to scrape your data (something like this):

$n = 0;
foreach($html->find('table tbody tr td div font b table tbody') as $element) {
    @$row[$n]['tr']  = $element->find('tr')->text;
    $n++;
}

// output your data
print_r($row);



Source: http://stackoverflow.com/questions/15601584/php-scraping-data-from-a-website

Obtaining reddit data

I am interested in obtaining data from different reddit subreddits. Does anyone know

if there is a reddit/other api similar like twitter does to crawl all the pages?


Yes, reddit has an API that can be used for a variety of purposes such as data

collection, automatic commenting bots, or even to assist in subreddit moderation.

There are a few places to discover information on reddit's API:

    github reddit wiki -- provides the overview and rules for using reddit's API

(follow the rules)
    automatically generated API docs -- provides information on the requests needed to

access most of the API endpoints
    /r/redditdev -- the reddit community dedicated to answering questions both about

reddit's source code and about reddit's API

If there is a particular programming language you are already familiar with, you

should check out the existing set of API wrappers for various languages. Despite my

bias (I am the package maintainer) I am quite certain PRAW, for python, has support

for the largest number of reddit API features.



Source: http://stackoverflow.com/questions/14322834/obtaining-reddit-data

Scrape Data Point Using Python

I am looking to scrape a data point using Python off of the url http://www.cavirtex.com/orderbook .

The data point I am looking to scrape is the lowest bid offer, which at the current moment looks like this:

<tr>
 <td><b>Jan. 19, 2014, 2:37 a.m.</b></td>
 <td><b>0.0775/0.1146</b></td>
 <td><b>860.00000</b></td>
 <td><b>66.65 CAD</b></td>
</tr>

The relevant point being the 860.00 . I am looking to build this into a script which can send me an email to alert me of certain price differentials compared to other exchanges.

I'm quite noobie so if in your explanations you could offer your thought process on why you've done certain things it would be very much appreciated.

Thank you in advance!

Edit: This is what I have so far which will return me the name of the title correctly, I'm having trouble grabbing the table data though.

import urllib2, sys
from bs4 import BeautifulSoup

site= "http://cavirtex.com/orderbook"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
print soup.title



Here is the code for scraping the lowest bid from the 'Buying BTC' table:

from selenium import webdriver

fp = webdriver.FirefoxProfile()
browser = webdriver.Firefox(firefox_profile=fp)
browser.get('http://www.cavirtex.com/orderbook')

lowest_bid = float('inf')
elements = browser.find_elements_by_xpath('//div[@id="orderbook_buy"]/table/tbody/tr/td')

for element in elements:
    text = element.get_attribute('innerHTML').strip('<b>|</b>')
    try:
        bid = float(text)
        if lowest_bid > bid:
            lowest_bid = bid
    except:
        pass

browser.quit()
print lowest_bid

In order to install Selenium for Python on your Windows-PC, run from a command line:

pip install selenium (or pip install selenium --upgrade if you already have it).

If you want the 'Selling BTC' table instead, then change "orderbook_buy" to "orderbook_sell".

If you want the 'Last Trades' table instead, then change "orderbook_buy" to "orderbook_trades".

Note:

If you consider performance critical, then you can implement the data-scraping via URL-Connection instead of Selenium, and have your program running much faster. However, your code will probably end up being a lot "messier", due to the tedious XML parsing that you'll be obliged to apply...

Here is the code for sending the previous output in an email from yourself to yourself:

import smtplib,ssl

def SendMail(username,password,contents):
    server = Connect(username)
    try:
        server.login(username,password)
        server.sendmail(username,username,contents)
    except smtplib.SMTPException,error:
        Print(error)
    Disconnect(server)

def Connect(username):
    serverName = username[username.index("@")+1:username.index(".")]
    while True:
        try:
            server = smtplib.SMTP(serverDict[serverName])
        except smtplib.SMTPException,error:
            Print(error)
            continue
        try:
            server.ehlo()
            if server.has_extn("starttls"):
                server.starttls()
                server.ehlo()
        except (smtplib.SMTPException,ssl.SSLError),error:
            Print(error)
            Disconnect(server)
            continue
        break
    return server

def Disconnect(server):
    try:
        server.quit()
    except smtplib.SMTPException,error:
        Print(error)

serverDict = {
    "gmail"  :"smtp.gmail.com",
    "hotmail":"smtp.live.com",
    "yahoo"  :"smtp.mail.yahoo.com"
}

SendMail("your_username@your_provider.com","your_password",str(lowest_bid))

The above code should work if your email provider is either gmail or hotmail or yahoo.

Please note that depending on your firewall configuration, it may ask your permission upon the first time you try it...



Source: http://stackoverflow.com/questions/21217034/scrape-data-point-using-python

Why I Use an Anonymous Web Proxy Service

I've been using an anonymous web proxy service for some time now and I don't think I would ever go back to ordinary surfing.    It's not that I am some sort of shadowy cyber criminal with some secret online life, no I pretty much surf the same web sites as everyone else, I certainly don't go to illegal or criminal sites.

If you want to know why, it's simply because I value my privacy and I know what data is stored in the average organisation and how it is protected.   Take your ISP for example, they logs pretty much everything that passes through them - every single request, every web page, every email and even every image is logged.   The vast majority of this traffic is in cleartext so the majority of it is instantly readable and can be matched to your address. 

It's amazing what these logs can tell about people you can build up a lot of information about someone from what they do on the web.  So do you know who has access to these logs at your ISP, what controls are in place, how is this information stored ?

If you don't know your not alone but after 25 years working in IT, I'll pretty much bet you wouldn't like to know the answer anyway.  This is the problem with much internet communication, it's so completely open - occasionally you'll obviously use a secure site to type in your credit card details.   Yet up to the point all your information is flying across the internet ether completely unprotected and accessible to anyone who has the will and the knowledge to intercept it. 

Believe me there are plenty of people who do have an interest in your information

Why do you think a European Directive was recently passed - Directive on Mandatory Retention of Communications Traffic Data.   It's  a bit of a mouthful but what it's  actually doing  is forcing your ISP to keep a record of every email sent, every internet session and web site visited for two years.   So think back at every web site you've ever visited for the last two years and it's on record and can be matched to your own little electronic profile.

How does that make you feel?

Even if you've done nothing wrong it's bound to make you feel a little uneasy after all why do Governments want all this information about us?   If you want to feel a little more uneasy, many Governments are looking at the UK Governments idea who are going to store all this information in a central database accessible to certain organisations.     Fear not US citizens you'll not be left out - the FBI are pushing hard for similar data retention facilities.

Storing data is dangerous

If you store data you have to be prepared to take good care of it.   All the privacy erosion concerns aside, the more personal data is stored about us, the more people will have access to it.  How thorough do you think is the vetting procedure for an ISP Technical engineer ?  Who potentially has access to huge amounts of this data.    Even if implicitly trust our govenrments with this tremendous invasion of our privacy - fraudsters, identity thieves and others can create mayhem with this sort of information. 

These logs are gold dust to anyone trying to steal our identity for example - they can match up users, with web sites, pick up password and all sorts of personal information.  Even  though a small part of this information is encrypted when you're actually using a secure web site (with the padlock below) there's plenty of other personal information available to compromise your usernames, password etc.

An identity thief can pick up huge amounts of data about the average person all ready, an hour or two picking up all your web traffic and they'll also have where you bank, what web sites you visit and logon to and a whole heap more.  This unfortunately is the tip of the iceberg and I suspect you'll see many, many people protecting  their privacy online soon from identity thieves, hackers and even our friendly snooping governments.

Everything I do online is private, I use a fast professional service that anonymises my connection and encrypts all the data.  My web logs sit next to yours in a server room at my ISP however mine are all completely encrypted and unreadable by anyone, whereas yours are in clear text.

One thing I should warn people about is the use of free anonymous web proxy services which you see all over the web.   Be very, very careful about these and think why are they supplying a very expensive and resource hungry service for free?   Remember also that by using one of these free proxies you are redirecting your traffic via this server whose owner can capture and log your data himself.   Many of these are owned and run by Eastern European Hacking groups as an easy way to gain peoples traffic, so be careful.

Source:http://ezinearticles.com/?Why-I-Use-an-Anonymous-Web-Proxy-Service&id=1679289

Using Bulk SMS Services how the stock advisory companies increasing their sales volume?

BNI's philosophy is "Givers Gain" - members are expected to focus on giving referrals to other members to build relationships and receive referrals in return. Why I made this point here is the BNI is one of the World’s largest referral organization which works via referral networks. Even Stock Advisory firms are also working on the similar kind of model. What they do? They send the list of their targeted audience or prospective customers stating that i.e.) Buy “ABC Ltd Share” BSE Code: XXXX @ Rs. 142-145 for 1 Month Target 380-460 Huge Govt. Orders received buy 2000 shared for 2Lacs Profit in 1 Month. 10 Days Free Trail also Available.

One More Sample – How much did you earn in Stock Market?? Today our client earned 20,500 INR. To join and earn daily in Stock Market.

When the person read the above messages they are tempted to add few more savings to their bank accounts so they show much interest what’s is this and how to invest in a smaller way etc… so the stock advisory firms to ask them to sign up for 10 days free trail where they will get lot of messages from the stock advisory companies like what’s the Buy && Sell and Profit, etc… Especially every day EOD, you will get the report like what’s the profit made by the major customer, what’s the investment, profit, etc… You will also be amazing! With the small amount of money, how the customers are earning more money. You will be showing much interest to have few more earnings. You are start welcoming lot of messages like this i.e.)

Note: Of course the stock advisory company whatever gives you the statement like X is making so much of profit that’s 100% correct. There is one thing you forget to realize, the X is spending his dedicated day ONLY for STOCK’s business. NOT as Part time he is earning HUGE PROFIT.

Here it’s NOT the question of earning the profit or loss. It’s one of the Marketing Strategy how the Stock Advisory Companies are converting the business very nicely with their targeted customer. Most of the time, they are ONLY using Bulk SMS Services as Marketing Strategy to promote their business. They do use Promotional & Transactional Bulk SMS Services. To convert anyone into business they do periodic Promotional Bulk SMS to them regarding the service and daily gain by different customer. Once if the customer is signed for FREE TRAIL then start using Transactional Bulk SMS Services to send the market updates, market call, staring, mid day, closing status, exit rate, etc…

Source:http://blogs.siliconindia.com/bulksmscompany/Business/Using-Bulk-SMS-Services-how-the-stock-advisory-companies-increasing-their-sales-volume-bid-fP9gkbI282792976.html

Internet marking is the finest method of increment of popularity in commerce

In today’s world key to any business is right kind of marketing strategy that needs to be implemented for the success of any entrepreneurial project. One of the fastest growing ways is to get the listing of your company in top search engines. SEO is one of the ways which can help in increasing or diverting the traffic on any website. People should understand the search result of their website is displayed on the World Wide Web. Different search engines work in different ways by providing different ranking to websites according to the search of the people. This is why same website has different ranks on different search engines.

Basically, website owner provides the keywords to the SEO providers which are appropriate to their business. Business SEO providers make sure the owners are charged low and also providing correct information to search engines. Providing better SEO in Ottawa is the basic feature that service providers aim at, their main motive is to make these services available to all the common people so that they can benefit there business in a better way. A good SEO mostly uses popular sites like Google, Bing, Yahoo, Yellow pages etc.

These SEO companies make sure that the listing of their clients is upgraded by repeated updating and also there ranking reaches top on search engines. Consultations with SEO service provider can be availed for free by the website owners. SEO companies carefully chooses keyword for their clients because it has to be unique and commonly used to make them reach at the top .They do so by carefully analyzing the owners website and competitors website and then the strategy is made to determine the perfect keyword for their clients. Owner should make sure that the prices charged to him are competitive according to market.

Consultation and Analysis are important because the keyword formation is directly dependent on these two parameters if the SEO service provider fails to submit the proper keyword it would result in failure of whole SEO services for the owner. SEO provider can re create and also modify the keywords for any website. If anyone specifically looking to create a new SEO website for him then the Looking internet marketing company in ottawa specialists can help them in determining a new domain that could be related to the business of the owner. Secondly, good hosting service is provided by them which help a website fast to access by any user. The biggest task for Internet marketing specialist is Link building because it represents more than 80% result of SEO.

For any website owner good SEO link building can be good in this competitive world because it attracts the people. A good SEO developer plays a pivotal role in developing a perfect keyword for the company. Ottawa SEO companies are very famous to deliver their services to the customers at very reasonable prices. SEO services are available online one can visit their websites see the list of services offered by them, read the testimonials of the existing and choose from them pay online and get started.

For further detail about Business seo service in Ottawa please visit the website.

Source:http://blogs.siliconindia.com/businessseo/Business/Internet-marking-is-the-finest-method-of-increment-of-popularity-in-commerce-bid-cf9206bx84461691.html