Achieving Sustainability in Mining

There's so much that our planet gives us for our consumption. These things come in different shapes and sizes, and some of the most abundant of them are minerals. Minerals are essential for living in these modern times, and when it comes to extracting them, mining is still the primary method used.

One of the biggest issues that any industry faces is sustainability, and the mining sector is certainly no exception to it. Some of the things that serve to constrain sustainability in this industry are the ever-increasing demand minerals, the consumption of resources that are needed to extract and process metals, as well as the pollution caused by the process of extracting them.

Increasing Demand for Minerals

There's no question that there's growth in the extraction of construction minerals. As more and more countries become more industrialized, the demand for such minerals is almost directly proportional to the growth in the construction industry. In the 20th century, we saw a growth in the extraction of construction materials. Demand for ores and industrial minerals also increased.

Impacts

Aside from the obvious impact mining has on the environment, it can also have a negative social impact. In order to keep up with the demand for mined resources, there's also a subsequent increase in mining activities to meet such demand. During the course of conducting such activities, there can be times when certain things are overlooked, including the short, medium and even long-term effects of mining activities in the community where they are done. This is then where there arises a need to balance the economic benefits of mining versus its potential harmful effects on the environment.

Sustainability and Maximizing Mining Benefits

There are ways to maximize the benefits we can get from mining as we improve sustainability both on the environmental and social fronts. This was specifically addressed in the Plan of Implementation of the World Summit on Sustainable Development. It identified three priority areas:

a. Support efforts to address the environmental, economic, health and social impacts and benefits of mining, minerals and metals throughout their life cycle;

b. Enhance the participation of stakeholders, including local and indigenous communities and women, to play an active role in minerals, metals and mining development throughout the life cycles of mining operations; and

c. Foster sustainable mining practices through the provision of financial, technical and capacity-building support to developing countries and countries with economies in transition for the mining and processing of minerals.

As long as efforts are made for mining to be environmentally, economically, and socially sustainable, we can enjoy the many benefits of mining without worrying about and suffering the potentially harmful effects mining can have on people and nature.

Source:http://ezinearticles.com/?Achieving-Sustainability-in-Mining&id=8108499

Data Extraction Services - A Helpful Hand For Large Organization

The data extraction is the way to extract and to structure data from not structured and semi-structured electronic documents, as found on the web and in various data warehouses. Data extraction is extremely useful for the huge organizations which deal with considerable amounts of data, daily, which must be transformed into significant information and be stored for the use this later on.

Your company with tons of data but it is difficult to control and convert the data into useful information. Without right information at the right time and based on half of accurate information, decision makers with a company waste time by making wrong strategic decisions. In high competing world of businesses, the essential statistics such as information customer, the operational figures of the competitor and the sales figures inter-members play a big role in the manufacture of the strategic decisions. It can help you to take strategic business decisions that can shape your business' goals..

Outsourcing companies provide custom made services to the client's requirements. A few of the areas where it can be used to generate better sales leads, extract and harvest product pricing data, capture financial data, acquire real estate data, conduct market research , survey and analysis, conduct product research and analysis and duplicate an online database..

The different types of Data Extraction Services:

•    Database Extraction:

•    Reorganized data from multiple databases such as statistics about competitor's products, pricing and latest offers and customer opinion and reviews can be extracted and stored as per the requirement of company.

•    Web Data Extraction:

•    Web Data Extraction is also known as data Extraction which is usually referred to the practice of extract or reading text data from a targeted website.

Businesses have now realized about the huge benefits they can get by outsourcing their services. Then outsourcing is profitable option for business. Since all projects are custom based to suit the exact needs of the customer, huge savings in terms of time, money and infrastructure are among the many advantages that outsourcing brings.

Advantages of Outsourcing Data Extraction Services:

•    Improved technology scalability
•    Skilled and qualified technical staff who are proficient in English
•    Advanced infrastructure resources
•    Quick turnaround time
•    Cost-effective prices
•    Secure Network systems to ensure data safety
•    Increased market coverage

By outsourcing, you can definitely increase your competitive advantages. Outsourcing of services helps businesses to manage their data effectively, which in turn would enable them to experience an increase in profits.

Outsourcing Web Research offer complete Data Extraction Services and Solutions to quickly collective data and information from multiple Internet sources for your Business needs in a cost efficient manner.

Source: http://ezinearticles.com/?Data-Extraction-Services---A-Helpful-Hand-For-Large-Organization&id=2477589

Coal Mining: Timeless Black Gems

Coal is an abundant sedimentary rock and fossil fuel used primarily as an energy source for electricity and other industrial uses such as smelting and alloy production. Coal is seldom confused with charcoal, which is primarily of wooden origin. Coal was previously used as mere household heating commodities but when the industrial revolution began, coal mining started to became large-scale. It then became an important commodity to produce electricity as well as to provide primary energy for industries as well as transportation during the 18th century to the 1950s.

Coal mining can be a very dangerous activity most especially when it involves mining underground. Gases produced can be very toxic or highly flammable, capable of explosions which can instantly kill a team of miners. Fortunately, technology has enabled companies the capacity to effectively protect their workers from the hazards of coal mining. But not only that, they can also do the same or even higher output even with significantly less number or workers.

Coal mining can involve mining underground by shaft mining or, for a more accessible and easier way, open pit mining the rock strata coal beds or coal seams. However, there are several other ways in coal mining.

Coals near the surface can be extracted by using open cut mining methods. Explosives are first used to break through the surface of the mining area and after which it is removed by draglines or by shovel and a truck. With the coal seam exposed, drills are utilized to fracture and thoroughly mine it in strips. Area mining involves drilling holes against the surface of the mining area and then planting the drill holes with explosives. When the surface is exposed, there will be a coal seam exposed. This can be extracted, mined and transported with trucks immediately. If it is still hard enough, this can also be drilled and blasted with explosives. The coal can then be collected until there is none left in the strip - then the process can be repeated to create a new mining strip. This coal mining method is most ideal for flat terrain.

One particular coal mining method is controversial. This is the mountaintop removal mining - and just as its name says, it's literally removing the mountain top, making the ridges and hill tops look like flattened plateaus. It is controversial because it drastically alters the topography as well as disturbing the ecosystem. Valleys will be filled the extracted prize and streams will be covered. The objective to coal mining was to extract these valuable energy sources, but is it really worth the damaging the environment or even risk worse consequences?

Source: http://ezinearticles.com/?Coal-Mining:-Timeless-Black-Gems&id=6333094

The Coal Mining Industry And Investing In It

The History Of Coal Usage

Coal was initially used as a domestic fuel, until the industrial revolution, when coal became an integral part of manufacturing for creating electricity, transportation, heating and molding purposes. The large scale mining aspect of coal was introduced around the 18th century, and Britain was the first nation to successfully use advanced coal mining techniques, which involved underground excavation and mining.

Initially coal was scraped off the surface by different processes like drift and shaft mining. This has been done for centuries, and since the demand was quite low, these mining processes were more than enough to accommodate the demand in the market.

However, when the practical uses of using coal as fuel sparked industrial revolution, the demand for coal rose abruptly, leading to severe shortage of the coal output, gradually paving the way for new ways to extract coal from under the ground.

Coal became a popular fuel for all purposes, even to this day, due to their abundance and their ability to produce more energy per mass than other conventional solid fuels like wood. This was important as far as transportation, creating electricity and manufacturing processes are concerned, which allowed industries to use up less space and increase productivity. The usage of coal started to dwindle once alternate energies such as oil and gas began to be used in almost all processes, however, coal is still a primary fuel source for manufacturing processes to this day.

The Process Of Coal Mining

Extracting coal is a difficult and complex process. Coal is a natural resource, a fossil fuel that is a result of millions of years of decay of plants and living organisms under the ground. Some can be found on the surface, while other coal deposits are found deep underground.

Coal mining or extraction comes broadly in two different processes, surface mining, and deep excavation. The method of excavation depends on a number of different factors, such as the depth of the coal deposit below the ground, geological factors such as soil composition, topography, climate, available local resources, etc.

Surface mining is used to scrape off coal that is available on the surface, or just a few feet underground. This can even include mountains of coal deposit, which is extracted by using explosives and blowing up the mountains, later collecting the fragmented coal and process them.

Deep underground mining makes use of underground tunnels, which is built, or dug through, to reach the center of the coal deposit, from where the coal is dug out and brought to the surface by coal workers. This is perhaps the most dangerous excavation procedure, where the lives of all the miners are constantly at a risk.

Investing In Coal

Investing in coal is a safe bet. There are still large reserves of coal deposits around the world, and due to the popularity, coal will be continued to be used as fuel for manufacturing process. Every piece of investment you make in any sort of industry or a manufacturing process ultimately depends on the amount of output the industry can deliver, which is dependent on the usage of any form of fuel, and in most cases, coal.

One might argue that coal usage leads to pollution and lower standards of hygiene for coal workers. This was arguably true in former years; however, newer coal mining companies are taking steps to assure that the environmental aspects of coal mining and usage are kept minimized, all the while providing better working environment and benefits package for their workers. If you can find a mining company that promises all these, and the one that also works within the law, you can be assured safety for your investments in coal.

Source: http://ezinearticles.com/?The-Coal-Mining-Industry-And-Investing-In-It&id=5871879

The Trouble With Bots, Spiders and Scrapers

With the Q4 State of the Internet - Security Report due out later this month, we continue to preview sections of it.

Earlier this week we told you about a DDoS attack from a group claiming to be Lizard Squad. Today we look at how
third-party content bots and scrapers are becoming more prevalent as developers seek to gather, store, sort and present
a wealth of information available from other websites.

These meta searches typically use APIs to access data, but many now use screen-scraping to collect information.

As the use of bots and scrapers continues to surge, there's an increased burden on webservers. While bot behavior is
mainly harmless, poorly-coded bots can hurt site performance and resemble DDoS attacks. Or, they may be part of a rival's competitive intelligence program.

Understanding the different categories of third-party content bots, how they affect a website, and how to mitigate their impact is an important part of building a secure web presence.

Specifically, Akamai has seen bots and scrapers used for such purposes as:

•    Setting up fraudulent sites
•    Reuse of consumer price indices
•    Analysis of corporate financial statements
•    Metasearch engines
•    Search engines
•    Data mashups
•    Analysis of stock portfolios
•    Competitive intelligence
•    Location tracking

During 2014 Akamai observed a substantial increase in the number of bots and scrapers hitting the travel, hotel and hospitality sectors. The growth in scrapers targeting these sectors is likely driven by the rise of rapidly developed mobile apps that use scrapers as the fastest and easiest way to collect information from disparate websites.

Scrapers target room rate pages for hotels, pricing and schedules for airlines. In many cases that Akamai investigated, scrapers and bots made several thousand requests per second, far in excess of what can be expected by a human using a web browser.

An interesting development in the use of headless browsers is the advent of companies that offer scraping as a service, such as PhantomJs Cloud. These sites make it easy for users to scrape content and have it delivered, lowering the bar to entry and making it easier for unskilled individuals to scrape content while hiding behind a service.

For each type of bot, there is a corresponding mitigation strategy.

The key to mitigating aggressive, undesirable bots is to reduce their efficiency. In most cases, highly aggressive bots are only helpful to their controllers if they can scrape a lot of content very quickly. By reducing the efficiency of the bot through rate controls, tar pits or spider traps, bot-herders can be driven elsewhere for the data they need.

Aggressive but desirable bots are a slightly different problem. These bots adversely impact operations, but they bring a benefit to the organization. Therefore, it is impractical to block them fully. Rate controls with a high threshold, or a user-prioritization application (UPA) product, are a good way to minimize the impact of a bot. This permits the bot access to the site until the number of requests reaches a set threshold, at which point the bot is blocked or sent to a waiting room. In the meantime, legitimate users are able to access the site normally.

Source: https://blogs.akamai.com/2015/01/performance-mitigation-bots-spiders-and-scrapers.html

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

If you’ve ever needed to pull data from a third party website, chances are you started by checking to see if they had an official API. But did you know that there’s a source of structured data that virtually every website on the internet supports automatically, by default?
scraper toolThat’s right, we’re talking about pulling our data straight out of HTML — otherwise known as web scraping. Here’s why web scraping is awesome:

Any content that can be viewed on a webpage can be scraped. Period.

If a website provides a way for a visitor’s browser to download content and render that content in a structured way, then almost by definition, that content can be accessed programmatically. In this article, I’ll show you how.

Over the past few years, I’ve scraped dozens of websites — from music blogs and fashion retailers to the USPTO and undocumented JSON endpoints I found by inspecting network traffic in my browser.

There are some tricks that site owners will use to thwart this type of access — which we’ll dive into later — but they almost all have simple work-arounds.

Why You Should Scrape

But first we’ll start with some great reasons why you should consider web scraping first, before you start looking for APIs or RSS feeds or other, more traditional forms of structured data.

Websites are More Important Than APIs

The biggest one is that site owners generally care way more about maintaining their public-facing visitor website than they do about their structured data feeds.

We’ve seen it very publicly with Twitter clamping down on their developer ecosystem, and I’ve seen it multiple times in my projects where APIs change or feeds move without warning.

Sometimes it’s deliberate, but most of the time these sorts of problems happen because no one at the organization really cares or maintains the structured data. If it goes offline or gets horribly mangled, no one really notices.

Whereas if the website goes down or is having issues, that’s a more of an in-your-face, drop-everything-until-this-is-fixed kind of problem, and gets dealt with quickly.

No Rate-Limiting

Another thing to think about is that the concept of rate-limiting is virtually non-existent for public websites.

Aside from the occasional captchas on sign up pages, most businesses generally don’t build a lot of defenses against automated access. I’ve scraped a single site for over 4 hours at a time and not seen any issues.

Unless you’re making concurrent requests, you probably won’t be viewed as a DDOS attack, you’ll just show up as a super-avid visitor in the logs, in case anyone’s looking.

Anonymous Access

There are also fewer ways for the website’s administrators to track your behavior, which can be useful if you want gather data more privately.

With APIs, you often have to register to get a key and then send along that key with every request. But with simple HTTP requests, you’re basically anonymous besides your IP address and cookies, which can be easily spoofed.

The Data’s Already in Your Face

Web scraping is also universally available, as I mentioned earlier. You don’t have to wait for a site to open up an API or even contact anyone at the organization. Just spend some time browsing the site until you find the data you need and figure out some basic access patterns — which we’ll talk about next.

Let’s Get to Scraping

So you’ve decided you want to dive in and start grabbing data like a true hacker. Awesome.

Just like reading API docs, it takes a bit of work up front to figure out how the data is structured and how you can access it. Unlike APIs however, there’s really no documentation so you have to be a little clever about it.

I’ll share some of the tips I’ve learned along the way.

Fetching the Data

So the first thing you’re going to need to do is fetch the data. You’ll need to start by finding your “endpoints” — the URL or URLs that return the data you need.

If you know you need your information organized in a certain way — or only need a specific subset of it — you can browse through the site using their navigation. Pay attention to the URLs and how they change as you click between sections and drill down into sub-sections.

The other option for getting started is to go straight to the site’s search functionality. Try typing in a few different terms and again, pay attention to the URL and how it changes depending on what you search for. You’ll probably see a GET parameter like q= that always changes based on you search term.

Try removing other unnecessary GET parameters from the URL, until you’re left with only the ones you need to load your data. Make sure that there’s always a beginning ? to start the query string and a & between each key/value pair.

Dealing with Pagination

At this point, you should be starting to see the data you want access to, but there’s usually some sort of pagination issue keeping you from seeing all of it at once. Most regular APIs do this as well, to keep single requests from slamming the database.

Usually, clicking to page 2 adds some sort of offset= parameter to the URL, which is usually either the page number or else the number of items displayed on the page. Try changing this to some really high number and see what response you get when you “fall off the end” of the data.

With this information, you can now iterate over every page of results, incrementing the offset parameter as necessary, until you hit that “end of data” condition.

The other thing you can try doing is changing the “Display X Per Page” which most pagination UIs now have. Again, look for a new GET parameter to be appended to the URL which indicates how many items are on the page.

Try setting this to some arbitrarily large number to see if the server will return all the information you need in a single request. Sometimes there’ll be some limits enforced server-side that you can’t get around by tampering with this, but it’s still worth a shot since it can cut down on the number of pages you must paginate through to get all the data you need.

AJAX Isn’t That Bad!

Sometimes people see web pages with URL fragments # and AJAX content loading and think a site can’t be scraped. On the contrary! If a site is using AJAX to load the data, that probably makes it even easier to pull the information you need.

The AJAX response is probably coming back in some nicely-structured way (probably JSON!) in order to be rendered on the page with Javscript.

All you have to do is pull up the network tab in Web Inspector or Firebug and look through the XHR requests for the ones that seem to be pulling in your data.

Once you find it, you can leave the crufty HTML behind and focus instead on this endpoint, which is essentially an undocumented API.

(Un)structured Data?

Now that you’ve figured out how to get the data you need from the server, the somewhat tricky part is getting the data you need out of the page’s markup.

Use CSS Hooks

In my experience, this is usually straightforward since most web designers litter the markup with tons of classes and ids to provide hooks for their CSS.

You can piggyback on these to jump to the parts of the markup that contain the data you need.

Just right click on a section of information you need and pull up the Web Inspector or Firebug to look at it. Zoom up and down through the DOM tree until you find the outermost <div> around the item you want.

This <div> should be the outer wrapper around a single item you want access to. It probably has some class attribute which you can use to easily pull out all of the other wrapper elements on the page. You can then iterate over these just as you would iterate over the items returned by an API response.

A note here though: the DOM tree that is presented by the inspector isn’t always the same as the DOM tree represented by the HTML sent back by the website. It’s possible that the DOM you see in the inspector has been modified by Javascript — or sometime even the browser, if it’s in quirks mode.

Once you find the right node in the DOM tree, you should always view the source of the page (“right click” > “View Source”) to make sure the elements you need are actually showing up in the raw HTML.

This issue has caused me a number of head-scratchers.

Get a Good HTML Parsing Library

It is probably a horrible idea to try parsing the HTML of the page as a long string (although there are times I’ve needed to fall back on that). Spend some time doing research for a good HTML parsing library in your language of choice.

Most of the code I write is in Python, and I love BeautifulSoup for its error handling and super-simple API. I also love its motto:

    You didn’t write that awful page. You’re just trying to get some data out of it. Beautiful Soup is here to help. :)

You’re going to have a bad time if you try to use an XML parser since most websites out there don’t actually validate as properly formed XML (sorry XHTML!) and will give you a ton of errors.

A good library will read in the HTML that you pull in using some HTTP library (hat tip to the Requests library if you’re writing Python) and turn it into an object that you can traverse and iterate over to your heart’s content, similar to a JSON object.

Some Traps To Know About

I should mention that some websites explicitly prohibit the use of automated scraping, so it’s a good idea to read your target site’s Terms of Use to see if you’re going to make anyone upset by scraping.

For two-thirds of the website I’ve scraped, the above steps are all you need. Just fire off a request to your “endpoint” and parse the returned data.

But sometimes, you’ll find that the response you get when scraping isn’t what you saw when you visited the site yourself.

When In Doubt, Spoof Headers

Some websites require that your User Agent string is set to something they allow, or you need to set certain cookies or other headers in order to get a proper response.

Depending on the HTTP library you’re using to make requests, this is usually pretty straightforward. I just browse the site in my web browser and then grab all of the headers that my browser is automatically sending. Then I put those in a dictionary and send them along with my request.

Note that this might mean grabbing some login or other session cookie, which might identify you and make your scraping less anonymous. It’s up to you how serious of a risk that is.

Content Behind A Login

Sometimes you might need to create an account and login to access the information you need. If you have a good HTTP library that handles logins and automatically sending session cookies (did I mention how awesome Requests is?), then you just need your scraper login before it gets to work.

Note that this obviously makes you totally non-anonymous to the third party website so all of your scraping behavior is probably pretty easy to trace back to you if anyone on their side cared to look.

Rate Limiting

I’ve never actually run into this issue myself, although I did have to plan for it one time. I was using a web service that had a strict rate limit that I knew I’d exceed fairly quickly.

Since the third party service conducted rate-limiting based on IP address (stated in their docs), my solution was to put the code that hit their service into some client-side Javascript, and then send the results back to my server from each of the clients.

This way, the requests would appear to come from thousands of different places, since each client would presumably have their own unique IP address, and none of them would individually be going over the rate limit.

Depending on your application, this could work for you.

Poorly Formed Markup

Sadly, this is the one condition that there really is no cure for. If the markup doesn’t come close to validating, then the site is not only keeping you out, but also serving a degraded browsing experience to all of their visitors.

It’s worth digging into your HTML parsing library to see if there’s any setting for error tolerance. Sometimes this can help.

If not, you can always try falling back on treating the entire HTML document as a long string and do all of your parsing as string splitting or — God forbid — a giant regex.

—

Well there’s 2000 words to get you started on web scraping. Hopefully I’ve convinced you that it’s actually a legitimate way of collecting data.

It’s a real hacker challenge to read through some HTML soup and look for patterns and structure in the markup in order to pull out the data you need. It usually doesn’t take much longer than reading some API docs and getting up to speed with a client. Plus it’s way more fun!

Source: https://blog.hartleybrody.com/web-scraping/

How to Take Advantage of Content Scrapers

This is our approach of dealing with content scrapers, and it turns out quite well. It helps our SEO as well as help us make extra bucks. Majority of the scrapers use your RSS Feed to steal your content. So these are some of the things that you can do:

•    Internal Linking – You need to interlink the CRAP out of your posts. With the Internal Linking Feature in WordPress 3.1, it is now easier than ever. When you have internal links in your article, it helps you increase pageviews and reduce bounce rate on your own site. Secondly, it gets you backlinks from the people who are stealing your content. Lastly, it allows you to steal their audience. If you are a talented blogger, then you understand the art of internal linking. You have to place your links on interesting keywords. Make it tempting for the user to click it. If you do that, then the scraper’s audience will too click on it. Just like that, you took a visitor from their site and brought them back to where they should have been in the first place.

•    Auto Link Keywords with Affiliate Links – There are few plugins like Ninja Affiliate and SEO Smart Links that will automatically replace assigned keywords with affiliate links. For example: HostGator, StudioPress, MaxCDN, Gravity Forms << These all will be auto-replaced with affiliate links when this post goes live.

•    Get Creative with RSS Footer – You can either use the RSS Footer or WordPress SEO by Yoast Plugin to add custom items to your RSS Footer. You can add just about anything you want here. We know some people who like to promote their own products to their RSS readers. So they will add banners. Guess what, now those banners will appear on these scraper’s website as well. In our case, we always add a little disclaimer at the bottom of our posts in our RSS feeds. It simply reads like “How to Put Your WordPress Site in Read Only State for Site Migrations and Maintenance is a post from: WPBeginner which is not allowed to be copied on other sites.” By doing this, we get a backlink to the original article from scraper’s site which lets google and other search engines know we are authority. It also lets their users know that the site is stealing our content. If you are good with codes, then you can totally get nuts. Such as adding related posts just for your RSS readers, and bunch of other stuff. Check out our guide t
o completely manipulating your WordPress RSS feed.

Source:http://www.wpbeginner.com/beginners-guide/beginners-guide-to-preventing-blog-content-scraping-in-wordpress/